Table of Contents

Search

  1. Preface
  2. Introduction to Big Data Streaming
  3. Big Data Streaming Configuration
  4. Sources in a Streaming Mapping
  5. Targets in a Streaming Mapping
  6. Streaming Mappings
  7. Window Transformation
  8. Appendix A: Connections
  9. Appendix B: Data Type Reference
  10. Appendix C: Sample Files

Big Data Streaming User Guide

Big Data Streaming User Guide

Advanced Properties

Advanced Properties

The following table describes the advanced properties that you configure for Hive targets in streaming mappings:
Property
Description
Tracing level
Controls the amount of detail in the mapping log file.
Create or replace table at run time
The Data Integration Service drops the target table at run time and replaces it with a table based on a target table that you identify.
When you write to a JDBC target and the target does not have a table, Spark creates a table even if you do not select this option. Spark creates columns in the table with appropriate target data type and default precision for the data type irrespective of the the precision you specify in the target properties.
DDL Query
The DDL query based on which the Data Integration Service creates or replace the target table at run time.
You cannot create a custom DDL query that creates or replaces a Hive table at run time in streaming mappings.
Truncate target table
Truncates the target before loading data.
If the mapping target is a Hive partition table, you can choose to truncate the target table only with Hive version 0.11.
Truncate Hive Target Partition
Truncates the external table before loading data.
Streaming mappings do not support this property. The property is ignored.
PreSQL
The SQL command the Data Integration Service runs against the target database before it reads the source.
The Developer tool does not validate the SQL.
PostSQL
The SQL command that the Data Integration Service runs against the target database after it writes to the target.
The Developer tool does not validate the SQL.
Streaming mappings do not support Post-SQL queries. Post-SQL queries in streaming mappings are ignored.
Maintain Row Order
Maintain row order while writing data to the target. Select this option if the Data Integration Service should not perform any optimization that can change the row order.
When the Data Integration Service performs optimizations, it might lose the row order that was established earlier in the mapping. You can establish row order in a mapping with a sorted flat file source, a sorted relational source, or a Sorter transformation. When you configure a target to maintain row order, the Data Integration Service does not perform optimizations for the target.