You can use the dynamic mapping for a streaming mapping by enabling refresh schema for Confluent Kafka AVRO schema source data objects, if mapping flow is enabled. The mapping flow is enabled for Kafka, Confluent Kafka, and Complex File targets.

You can create dynamic ports in a transformation to receive new or changed columns from an upstream transformation. A dynamic port receives one or more columns and generates ports based on input rules. The input rules determine the columns that a dynamic port receives and generates.

Dynamic port creates a generated port category from an Expression transformation. A static port created in an Expression transformation precedes over generated ports. If a static port and generated port have the same name, the generated port gets renamed. The same is applicable for the header ports. By default, during the runtime, static ports are considered if both static and header port have the same name.

Every week, you receive customer data from different departments that you need to join and aggregate. The departments might periodically change the source schema to include additional columns for departmental analysis. To accommodate the changes to the data source, you create a dynamic mapping. You configure the Read transformation to get data object columns at read time. Create an input rule to include columns that you need and to exclude all other columns.

When your organization wants to manage frequent metadata changes in the streaming data sources, develop a dynamic streaming mapping that can get the metadata changes directly from the data sources at run time.