Rules and Guidelines for Data Preview on the Spark Engine
Rules and Guidelines for Data Preview on the Spark Engine
Consider the following rules and guidelines when you work with data preview on the Spark engine:
You cannot preview data in mappings with dynamic complex data types, such as dynamic arrays, dynamic maps, and dynamic structs.
If a map data type source with primitive keys includes duplicate keys, the Data Viewer displays only one instance of the duplicate key-value pair. If a map data type source with complex keys includes duplicate keys, the Data Viewer displays all key-value pairs.
You cannot preview data for a mapping that reads hierarchical data from a Hive source.
You can run up to 10 concurrent data preview jobs on the Spark engine.
Previewing data on the Spark engine is memory intensive. Increase the Heap memory size when you run concurrent preview jobs.
For high volume data preview jobs that use Spark Jobserver, configure the following Spark advanced properties in the Hadoop connection to increase driver and executor memory:
Effective in version 10.4.0, previewing hierarchical data when the Data Integration Service runs on a grid is available for technical preview.
Technical preview functionality is supported for evaluation purposes but is unwarranted and is not supported in production environments or any environment that you plan to push to production. Informatica intends to include the preview functionality in an upcoming release for production use, but might choose not to in accordance with changing market or technical circumstances. For more information, contact Informatica Global Customer Support.