Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Data Engineering Integration
  3. Mappings
  4. Mapping Optimization
  5. Sources
  6. Targets
  7. Transformations
  8. Python Transformation
  9. Data Preview
  10. Cluster Workflows
  11. Profiles
  12. Monitoring
  13. Hierarchical Data Processing
  14. Hierarchical Data Processing Configuration
  15. Hierarchical Data Processing with Schema Changes
  16. Intelligent Structure Models
  17. Blockchain
  18. Stateful Computing
  19. Appendix A: Connections Reference
  20. Appendix B: Data Type Reference
  21. Appendix C: Function Reference

Spark Engine Logs

Spark Engine Logs

The Spark engine logs appear in the LDTM log. The LDTM logs the results of the Spark engine execution plan run for the mapping. You can view the LDTM log from the Developer tool or the Monitoring tool for a mapping job.
The log for the Spark engine shows the following:
  • Step to translate the mapping to an internal format
  • Steps to optimize the mapping
  • Steps to render the mapping to Spark code
  • Steps to submit the code to the Spark executor
  • Scala code that the Logical Data Translation Generator creates from the mapping logic
  • Total number of cluster nodes used to execute the mapping
When you run Sqoop mappings on the Spark engine, the Data Integration Service prints the Sqoop log events in the mapping log.


Updated September 28, 2020