When you run a mapping on the Spark engine, you can read data from and write data to Avro, ORC, and Parquet files organized based on directories.
You can configure an Amazon S3-compatible storage to access and manage the data that is stored over an S3 compliant interface. You can use the Scality RING S3-compatible storage. Use the Amazon S3 connection to connect to Scality RING and perform the read and write operations similar to Amazon S3.
You can configure Federated Single Sign-On (SSO) authentication to securely access the Amazon S3 resources.
You can configure a cached lookup operation to cache the lookup data when you run a mapping on the Spark engine.
You can read and write flat files with and without headers.
You can configure row delimiter and define qualifier scope when you read and write flat files.
You can perform audits for read operations in Amazon S3 mappings that run in the native environment or on the Spark engine.