ADTF and DDL

Table of Contents

• Description
• Usage of Stream Meta Type "adtf/default"
• Methods to create valid DDL and "adtf/default"
  • Creating a DDL Description file and use it
  • MD Generator Tool
    • Detailed description of MD Generator Tool
    • MD Generator Tool from Header
    • Detailed description of MD Generator Tool from Header
    • Read the DDL file and setup Stream Type
  • Define DDL Description in code and use it
    • Use structure class
    • Use structure<> template
    • Define a huge string for DDL Description
• Serialization of "adtf/default" Samples
• DDL Access to the Sample Content

Description

The Data Definiton Language (DDL) is to describe memory content in a formal way.
So, it is possible to provide functionality, executables and utilities those are able to access, process and display data in a generic way.

The DDL will have some advantages over other ways to describe data like IDL (Interface Description Language), Protocol Buffers or FlatBuffers:

• It describes only the layout of data in a common formal way! Each other description language format for describing data can be created.
• This kind of data description makes it usable on and between cross-platforms, compiler-, architecure- and memory-layout independent machines.
• The DDL does not force any specific method of serialization and deserialisation.
• Describing data in DDL does not force you to use any code generation or toolings if you do not want that.
• Reading data memory and the values via DDL Access methods does not mean, that the creator of that data must have created it with DDL writing methods.
• You can dynamically describe your data in runtime.
• All kinds of basic datatypes existing in C/C++ are supported. No restriction on that.
• Direct access to structured content via C++ code, the DDL will only describe that content.

There are also some disadvantages on DDL:

• DDL is old fashioned XML, but it makes no sense to change to famous JSON, break the users code and confuse them.
• No optimization while compile time, access methods will dynamically address values only in runtime.

Usage of Stream Meta Type "adtf/default"

The most important stream type instance within ADTF uses the default stream meta type adtf/default (see also stream type for a common definition).

This stream meta type is define in adtf::mediadescription::stream_meta_type_default. It will use 3 important properties:

• md_struct: adtf::mediadescription::stream_meta_type_default::strMDStructProperty
• md_definitions: adtf::mediadescription::stream_meta_type_default::strMDDefinitionsProperty
• md_data_serialized: adtf::mediadescription::stream_meta_type_default::strMDDataSerialized

Note

It is very important to know, that many dynamic DDL functionality in ADTF does not require the default stream type itself but uses the properties md_struct and md_definitions. > See i.e. the compatibiliy functionality: adtf/default

In ADTF 2 the DDL file (.dat.description) was an optional side-by-side file description next to the ADTF DAT File (.dat), it described the content of the ADTF DAT file.

Since the ADTFDAT file (.adtfdat) in ADTF 3 contains the complete DDL description if "adtf/default" stream meta type is used, only the units, datatypes, enums and structs tags are relevant.
Within ADTF 3 we do not use the streammetatypes or streams tags anymore which are mentioned within DDL Specification.

Methods to create valid DDL and "adtf/default"

Creating a DDL Description file and use it

It is possible to create a DDL Description file (.description) before you define any structured data within your code.
We recommend using the DDL Editor from DDL Utility standalone or redeployed within ADTF Configuration Editor packages.

MD Generator Tool

ADTF provides a tool that creates C++ headers containing structure definitions and access classes from a given DDL Description file (.description).

If you are using the adtf_add_filter, adtf_add_streaming_service, adtf_add_system_service CMake macros, you only need to specify your .description file within the sources of your target:

find_package(adtf_sdk COMPONENTS filtersdk)
 
adtf_add_filter(myfilter
    myfilter.cpp
    mytypes.description)

This will generate a header file (.h) named after your .description file, that you can include in your source files.
In the above case:

#include <mytypes.h>

You may use following code within your filter if the DDL file contains any valid complex type (struct type) named tMyType:

using namespace adtf::streaming;
using namespace adtf::mediadescription;
adtf::ucom::object_ptr<IStreamType> pMyDDLType = adtf::ucom::make_object_ptr<stream_type_default<tMyType>>();

If you want the structure definitions to be generated in a namespace use the "MD_NAMESPACE" argument:

adtf_add_filter(myfilter
    myfilter.cpp
    mytypes.description
    MD_NAMESPACE a::b::c)

If for some reasons you cannot use these macros, there is the adtf_md_generate_cpp CMake macro provided that will add header generation to existing targets.

The executable of the tool is accessible via the imported adtf::mdgen CMake target.
A non target based CMake macro is provided via adtf_md_generate_from_description.

An example of the usage of the generation facilities can be found at Example Demo Code Generation.

Note

The generated header file will be directly accessible after the filter, streaming service or service has been built for the first time, e.g Qt moc-files.

Detailed description of MD Generator Tool

To describe the MD Generator Tool of the above chapter in more detail we need to tell, that it will create exactly two header files and one cpp-file.

• mytypes_description.h: Containing the generated structs, enums out of the description file
• mytypes.h: Containing the md_sample_data template specializations to access the generated string description and the data of one sample via comfortable access functions
• mytypes.cpp: Implementation of the access functions

It is possible to use additional MD_ARGUMENTS within the CMake macro.

Instead of using the stdtypes (bool, int8_t, uint8_t, ... ) in generated header files this will lead to a usage of adtftypes (tBool, tInt8, tUInt8, ...):

adtf_add_filter(myfilter
    myfilter.cpp
    mytypes.description
    MD_ARGUMENTS --use-adtftypes)

Instead of using enum class for enumerations in generated header files this will lead to a usage of enum only:

adtf_add_filter(myfilter
    myfilter.cpp
    mytypes.description
    MD_ARGUMENTS --no-scoped-enum)

Instead of using the global namepace qualifier for each type usage this will drop the leading :::

adtf_add_filter(myfilter
    myfilter.cpp
    mytypes.description
    MD_ARGUMENTS --no-global-namespace)

Instead of using alignment for positioning the elements, this will use padding bytes only:

adtf_add_filter(myfilter
    myfilter.cpp
    mytypes.description
    MD_ARGUMENTS --padded-only)

MD Generator Tool from Header

ADTF provides a additional toolset that creates a description out of struct definitions within a simple C++ header and the necessary access classes.

To do so there are 3 tools provided:

• A castxml tool that is able to parse a C++ header file and write them into a xml-based temporary castxml file
• A header2ddl tool using this castxml file and generates the .description file
• The adtf_mdgen that generates the access classes for the given .description file

If you are using the adtf_add_filter, adtf_add_streaming_service, adtf_add_system_service CMake macros, you only need to specify your header files to generate descriptions for within the MD_HEADERS of your target:

find_package(adtf_sdk COMPONENTS filtersdk)
 
adtf_add_filter(myfilter
    myfilter.cpp
    MD_HEADERS mytypes.h)

This will generate the temporary castxml and .description file within your build folder.
The adtf_mdgen will generate the necessary access template classes within a header, that you can include in your source files. In the above case:

#include <mytypes_mdgen.h>  //<== generated header with access classes will also include <mytypes.h>

Now you can use following code within your filter like the the example under MD Generator Tool with a complex type named tMyType:

using namespace adtf::streaming;
using namespace adtf::mediadescription;
adtf::ucom::object_ptr<IStreamType> pMyDDLType = adtf::ucom::make_object_ptr<stream_type_default<tMyType>>();

If for some reasons you cannot use these macros, there is the adtf_md_generate_cpp CMake macro provided that will add header generation to existing targets.
A non target based CMake macro is provided via adtf_md_generate_from_header.

An example of the usage of the generation facilities can be found at Example Demo Description Generation.

Note

The generated header file will be directly accessible after the filter, streaming service or service has been built for the first time, e.g Qt moc-files.

Detailed description of MD Generator Tool from Header

To describe the MD Generator Tool of the above chapter in more detail we need to tell, that it will create exactly two header files and one cpp-file.

• mytypes.h: MD_HEADERS source file containing the "hand written" structure definitions to generate the description for
• mytypes.castxml: Generated, temporary castxml type information based on clang parser
• mytypes.description: Generated, temporary DDL media description file with the struct definitions out of mytypes.h
• mytypes_mdgen.h: Generated file containing the md sample access functions for the structs, enums out of mytypes.h
• mytypes_mdgen.cpp: Implementation of the access functions

It is possible to use MD_HEADERS_ARGUMENTS also combined with the MD_ARGUMENTS like described above within the CMake macro.

Instead generating all struct types only a description for the type tMyType is generated:

adtf_add_filter(myfilter
    myfilter.cpp
    MD_HEADERS mytypes.h
    MD_HEADERS_ARGUMENTS --types tMyType)

adtf_add_filter(myfilter
    myfilter.cpp
    MD_HEADERS mytypes.h
    MD_HEADERS_TYPES tMyType)

Instead of using the adtftypes (tBool, tInt8, tUInt8, ...) in generated description files this will lead to a forced usage of stdtypes (bool, int8_t, uint8_t, ... ):

adtf_add_filter(myfilter
    myfilter.cpp
    MD_HEADERS mytypes.h
    MD_HEADERS_ARGUMENTS --force-datatypes stdtypes)

Instead of using the current platform dependent byte order with the generated description, use the given one: LE or BE

adtf_add_filter(myfilter
    myfilter.cpp
    MD_HEADERS mytypes.h
    MD_HEADERS_ARGUMENTS --byteorder BE)

Read the DDL file and setup Stream Type

It is also possible to read a file in code and create the description while runtime dynamically. To do so, please use the ddl::DDFile functionality.

#include <adtffiltersdk/adtf_filtersdk.h>
//...

    using namespace adtf::ucom;
    using namespace adtf::streaming;
    using namespace adtf::mediadescription;
 
    std::string oDDLFileFromConfig = "./my_type.description"; 
 
    //********************************
    //new code to import a DDL file
    //********************************
    auto oDDLFile = ddl::DDFile::fromXMLFile(oDDLFileFromConfig);
    
    //resolve the struct to a small description with the DDString functionality 
    //and create the type from imported file 
    object_ptr<IStreamType> pDDLType = make_object_ptr<stream_type_default<>>("tMyTypeFromFile", ddl::DDString::toXMLString("tMyTypeFromFile", oDDLFile).c_str());
 

Define DDL Description in code and use it

Use structure class

It is possible to use the structure API provided via this ddl package. It will create the necessary DDL description while runtime and compile-time:

#include <adtffiltersdk/adtf_filtersdk.h>
//...

    using namespace adtf::ucom;
    using namespace adtf::streaming;
    using namespace adtf::mediadescription;

    //we create a structure description and its default DDL type
    struct tMyType
    {
        int32_t m_nValue1;
        double m_fValue2;
        bool m_bArray[2];
    };
    auto oDescriptionForMyType = structure<>("tMyType")
                                 .Add<int32_t>("m_nValue1")
                                 .Add<double>("m_fValue2")
                                 .Add<tBool>("m_bArray", 2);

    object_ptr<IStreamType> pDDLType = make_object_ptr<stream_type_default<>>(oDescriptionForMyType);
 

Use structure<> template

It is possible to use the type-reflection API provided via DDL package. It will create the necessary DDL description while runtime and compile-time:

#include <adtffiltersdk/adtf_filtersdk.h>
//...

    using namespace adtf::ucom;
    using namespace adtf::streaming;
    using namespace adtf::mediadescription;
 
    //we create a structure description and its default DDL type
    struct tMyType
    {
        int32_t   m_nValue1;
        double m_fValue2;
        bool m_bArray[2];
    };
    auto oDescriptionForMyType = structure<tMyType>("tMyType")
                                .Add("m_nValue1", &tMyType::m_nValue1)
                                .Add("m_fValue2", &tMyType::m_fValue2)
                                .Add("m_bArray", &tMyType::m_bArray);
 
    object_ptr<IStreamType> pDDLType = make_object_ptr<stream_type_default<>>(oDescriptionForMyType);
 

Define a huge string for DDL Description

This is a very old fashioned way to create your own DDL Decription. We do not recommend to use that, but for completeness it is specified here:

#include <adtffiltersdk/adtf_filtersdk.h>
//...

    using namespace adtf::ucom;
    using namespace adtf::streaming;
    using namespace adtf::mediadescription;
 
    //we create a structure description and its default DDL type
    std::string strMyTypeString;
    strMyTypeString += "<struct name=\"tMyType\" version=\"1\" arraysize=\"1\" alignment=\"4\">";
    strMyTypeString += "   <element type=\"tInt32\" name=\"m_nValue1\">";
    strMyTypeString += "       <deserialized alignment=\"4\"/>"; 
    strMyTypeString += "       <serialized bytepos=\"0\" byteorder=\"LE\"/>";
    strMyTypeString += "   </element>";
    strMyTypeString += "   <element type=\"tFloat64\" name=\"m_fValue2\">";
    strMyTypeString += "       <deserialized alignment=\"4\"/>";
    strMyTypeString += "       <serialized bytepos=\"4\" byteorder=\"LE\"/>";
    strMyTypeString += "   </element>";
    strMyTypeString += "   <element type=\"tBool\" name=\"m_bArray\" arraysize=\"2\">";
    strMyTypeString += "       <deserialized alignment=\"1\"/>";
    strMyTypeString += "       <serialized bytepos=\"8\" byteorder=\"LE\"/>";
    strMyTypeString += "   </element>";
    strMyTypeString += "</struct>";
 
    object_ptr<IStreamType> pDDLType = make_object_ptr<stream_type_default<>>("tMyType", strMyTypeString.c_str());
 

Serialization of "adtf/default" Samples

The DDL describes the data layout in two separate representations:

• Deserialized Memory Respresentation
  • The memory representation of data is the definition of value positions within RAM.
  • To calculate the positions of the values within the memory only alignment, type information and arraysize are relevant attributes.
    • Keep in mind: The byteorder attribute is absolutely not relevant here, because the values exists always in the same byteorder as the platform and the used compiler for that binary.
• Serialized Respresentation
  • The serialized representation of data is the definition of values in a file or network.
  • Within ADTF context this is the case storaging data in a ADTFDAT File or while sharing data over IPC.
  • To access the positions of that values and transform from one to the other one the bytepos, byteorder, type and arraysize are relevant attributes only.

We separate these representations to access the values in a fast way in memory, when deserialized and the deserialized representation will assure that data are described in such a formal way to use it on every single platform.
It enables exchanging data between different architectures (big endian, little endian) and processors (Intel, ARM) vie network and file.

DDL Access to the Sample Content

Reading access via md_sample_data:

Please note, that you are encouraged to use the adtf::mediadescription::md_sample_data template to access sample data via Media Descriptions (DDL), instead of using a decoder directly.

Reading access via DDLDecoder:

To read data from a sample you only need to use the adtf::mediadescription::cStaticSampleDecoder or adtf::mediadescription::cSampleDecoder.
See also ADTF Codecs and Sample Codecs.

Writing access via output_sample_data:

If you're generating output samples, please use the adtf::streaming::output_sample_data template with your structure definition.
There is no need to fill samples via a codec.

Writing access via DDLCodec:

To write data to a sample you can use the adtf::mediadescription::cStaticSampleCodec or adtf::mediadescription::cSampleCodec. See also ADTF Codecs and Sample Codecs.