Amazon Redshift Connector Best Practices

Amazon Redshift Connector Best Practices

ETL and ELT Overview

ETL and ELT Overview

ETL (Extract, Transform, and Load) and ELT (Extract, Load, and Transform) are methods used to transfer data from a source to a data warehouse.
The ETL approach is ideal for transferring external data from either a single source or combined heterogeneous sources into Amazon Redshift cluster. Amazon Redshift recommends that you load the data to Amazon Redshift through Amazon S3 staging location as this approach is the fastest data ingestion option into Amazon Redshift. The ETL approach reads and writes data at high speeds. This allows faster data processing and performing complex calculations in less time to read from or write to disk.
In the ETL approach, memory space of the staging location is the only limiting factor.
However, after loading the data to the Amazon Redshift cluster, you must use a different data transformation approach to transform. Then, move the data across staging location, intermediate, and analysis Amazon Redshift tables in the same Amazon Redshift cluster. Once the data is in the Amazon Redshift cluster, the ETL pattern of moving data in and out of Amazon Redshift tables within the same Amazon Redshift cluster through Amazon S3 is often sub-optimal. Use the ELT approach of pushing the data transformation operations to execute as SQL operations by the Amazon Redshift layer to optimize the process. The ELT approach optimises the process because the data is already available in the data warehouse and the data transformation operations such as filter, join, aggregation, or sorting are supported at the SQL layer enabling faster data processing.
In Informatica, the ELT approach is referred to as SQL Pushdown and Amazon Redshift Full SQL Pushdown is supported in the Informatica Cloud platforms. Informatica supports both ETL as well as ELT approach and provides the option to the users to choose based on the scenario.
The following image illustrates the ETL and ELT approaches:

0 COMMENTS

We’d like to hear from you!