Table of Contents

Search

  1. Preface
  2. Introduction to Informatica Data Engineering Integration
  3. Mappings
  4. Mapping Optimization
  5. Sources
  6. Targets
  7. Transformations
  8. Python Transformation
  9. Data Preview
  10. Cluster Workflows
  11. Profiles
  12. Monitoring
  13. Hierarchical Data Processing
  14. Hierarchical Data Processing Configuration
  15. Hierarchical Data Processing with Schema Changes
  16. Intelligent Structure Models
  17. Blockchain
  18. Stateful Computing
  19. Appendix A: Connections Reference
  20. Appendix B: Data Type Reference
  21. Appendix C: Function Reference

How to Create a Mapping with an Intelligent Structure Model

How to Create a Mapping with an Intelligent Structure Model

You can create a mapping with a data object that incorporates an intelligent structure model to parse data. You run the mapping on the Spark engine or the Databricks Spark engine to process the data.
The following high-level tasks describe how to develop and run a mapping to read and process data in files of any type that an intelligent structure model can process, and then write the data to a target.
The tasks and the order in which you perform the tasks to develop the mapping depend on the mapping scenario.
Optionally, create an intelligent structure model.
You can create an
intelligent structure model
in Cloud
Data Integration
. Create the model using a sample file in
Data Integration
and export the model to the relevant file storage system, following the instructions in Create an Intelligent Structure Model in Cloud Data Integration.
Create a connection.
Create a connection to access data in files that are stored in that system. You can create the following types of connections that work with the data objects that can incorporate an intelligent structure model:
  • Amazon S3
  • Hadoop Distributed File System
  • Microsoft Azure Blob
  • Microsoft Azure Data Lake Store
Create a read data object.
  1. Create a data object with an intelligent structure model to represent the files stored as sources. You can create the following types of data objects with an intelligent structure model:
    • Complex file
    • Amazon S3
    • Microsoft Azure Blob
    • Microsoft Azure Data Lake Store
  2. Configure the data object properties. Note the following guidelines:
    • In
      Resource Format
      , select
      Intelligent Structure Model or Sample File
      .
    • If you created an
      intelligent structure model
      in
      Data Integration
      , select that file.
      Alternatively, select a sample file to base the model on. When you select an XML, JSON, ORC, Avro, or Parquet sample file,
      Intelligent Structure Discovery
      creates an
      intelligent structure model
      based on the sample file that you select.
      To import an intelligent structure model as a data object, you must have the relevant
      Informatica Intelligent Cloud Services
      license. For more information about prerequisites for creating an
      intelligent structure model
      , see Before You Begin.
  3. In the data object read operation, configure columns to project hierarchical data as a complex data type. Enable the Project Column as Complex Data Type property in the Column Projection properties.
Create a write data object.
  1. Create a data object to write the data to target storage.
  2. Configure the data object properties.
Do not associate an intelligent structure model with a data object write operation. If you use a write operation that is associated with an intelligent structure model in a mapping, the mapping is not valid.
Create a mapping.
  1. Create a mapping.
  2. Add a Read transformation based on the data object with the intelligent structure model.
  3. Based on the mapping logic, add other transformations that are supported on the run-time engine. Link the ports and configure the transformation properties based on the mapping logic.
  4. Add a Write transformation based on the data object that passes the data to the target storage or output. Link the ports and configure the transformation properties based on the mapping logic.
Configure the mapping.
  1. Choose a validation environment and a run-time engine.
  2. Choose an execution environment and select a connection.
Run the mapping.
  1. Validate the mapping and fix any errors.
  2. Optionally, view the execution plan to debug the logic.
  3. Run the mapping.