Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Data Engineering Integration
  3. Mappings
  4. Mapping Optimization
  5. Sources
  6. Targets
  7. Transformations
  8. Python Transformation
  9. Data Preview
  10. Cluster Workflows
  11. Profiles
  12. Monitoring
  13. Hierarchical Data Processing
  14. Hierarchical Data Processing Configuration
  15. Hierarchical Data Processing with Schema Changes
  16. Intelligent Structure Models
  17. Blockchain
  18. Stateful Computing
  19. Appendix A: Connections Reference
  20. Appendix B: Data Type Reference
  21. Appendix C: Function Reference

Rules and Guidelines for Data Preview on the Spark Engine

Rules and Guidelines for Data Preview on the Spark Engine

Consider the following rules and guidelines when you work with data preview on the Spark engine:
  • You cannot preview data in mappings with dynamic complex data types, such as dynamic arrays, dynamic maps, and dynamic structs.
  • If a map data type source with primitive keys includes duplicate keys, the Data Viewer displays only one instance of the duplicate key-value pair. If a map data type source with complex keys includes duplicate keys, the Data Viewer displays all key-value pairs.
  • You can run up to 10 concurrent data preview jobs on the Spark engine.
  • Previewing data on the Spark engine is memory intensive. Increase the Heap memory size when you run concurrent preview jobs.
  • For high volume data preview jobs that use Spark Jobserver, configure the following Spark advanced properties in the Hadoop connection to increase driver and executor memory:

      spark.driver.cores

      spark.driver.memory

      spark.executor.cores

      spark.executor.memory

    For more information, see Spark Advanced Properties.
  • Effective in version 10.4.0, previewing hierarchical data when the Data Integration Service runs on a grid is available for technical preview.
Technical preview functionality is supported for evaluation purposes but is unwarranted and is not production-ready. Informatica recommends that you use in non-production environments only. Informatica intends to include the preview functionality in an upcoming release for production use, but might choose not to in accordance with changing market or technical circumstances. For more information, contact Informatica Global Customer Support.

0 COMMENTS

We’d like to hear from you!