Table of Contents

Search

  1. Preface
  2. Introduction to Data Engineering Streaming
  3. Data Engineering Streaming Administration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Monitoring REST API Reference
  10. Appendix C: Sample Files

Dynamic Mappings

Dynamic Mappings

A dynamic mapping is a mapping that can accommodate changes to sources, targets, and transformation logic at run time. Use a dynamic mapping to manage frequent schema or metadata changes or to reuse the mapping logic for data sources with different schemas. Configure rules, parameters, and general transformation properties to create a dynamic mapping.
To create a dynamic streaming mapping, enable the refresh schema at the source and target, or enable the mapping flow at the target.

Mapping Flow

You can use the dynamic mapping for a streaming mapping by enabling refresh schema for Confluent Kafka AVRO schema source data objects, if mapping flow is enabled. The mapping flow is enabled for Kafka, Confluent Kafka, and Complex File targets.

Dynamic Port

You can create dynamic ports in a transformation to receive new or changed columns from an upstream transformation. A dynamic port receives one or more columns and generates ports based on input rules. The input rules determine the columns that a dynamic port receives and generates.

Port Mapping

Dynamic port creates a generated port category from an Expression transformation. A static port created in an Expression transformation precedes over generated ports. If a static port and generated port have the same name, the generated port gets renamed. The same is applicable for the header ports. By default, during the runtime, static ports are considered if both static and header port have the same name.
When header and static port names are same, ensure that you link the correct port to the target mapping.

Dynamic Mapping Example

Every week, you receive customer data from different departments that you need to join and aggregate. The departments might periodically change the source schema to include additional columns for departmental analysis. To accommodate the changes to the data source, you create a dynamic mapping. You configure the Read transformation to get data object columns at read time. Create an input rule to include columns that you need and to exclude all other columns.
When your organization wants to manage frequent metadata changes in the streaming data sources, develop a dynamic streaming mapping that can get the metadata changes directly from the data sources at run time.