You can use the following Hadoop connections: Hive and Hadoop Distributed File System (HDFS). You can import Hive and HDFS connections in Test Data Manager. In a Hadoop plan, you can select the Hadoop connections as source, target, or both.
In Hive database schema, there might be some temporary junk tables that are created while running a mapping. The following sample formats are the junk tables in a Hive database schema:
When you import data sources from a Hive database and if there are some temporary junk tables present in the schema, you must ensure that you do not select those tables.
You can create a Hadoop plan to move data from Hadoop sources, flat files, or relational databases such as Oracle, DB2, ODBC-Sybase, and ODBC-Microsoft SQL Server into Hive or HDFS targets. You can also create a Hadoop plan when you want to move data between Hadoop sources and targets. If the source is HDFS, you can move data to a Hive or an HDFS target. If the source is Hive, you can move data to a Hive or an HDFS target.
To run a Hadoop plan, TDM uses Data Integration Service that supports the Hadoop pushdown optimization feature. When you generate and run the Hadoop plan, TDM generates the mappings and submits the mappings to the Data Integration Service. The Data Integration Service applies the pushdown optimization and runs the mappings on the Hadoop cluster to improve the performance.
You cannot perform data subset or data generation operations for Hadoop sources and targets.