Required if the Hadoop cluster uses Kerberos authentication. Hadoop impersonation user. The user name that the Data Integration Service impersonates to run mappings in the Hadoop environment.

Data Engineering Integration supports operating system profiles on all Hadoop distributions. In the Hadoop run-time environment, the Data Integration Service pushes the processing to the Hadoop cluster and the run-time engines run mappings with the operating system profile specified Hadoop impersonation properties.

Hadoop compression library for a compression codec class name.

Codec class name that enables data compression and improves performance on temporary staging tables.

Namespace for Hive staging tables. Use the name

default

for tables that do not have a specified database name.

If you do not configure a namespace, the Data Integration Service uses the Hive database name in the Hive target connection to create staging tables.

When you run a mapping in the native environment to write data to Hive, you must configure the Hive staging database name in the Hive connection. The Data Integration Service ignores the value you configure in the Hadoop connection.

SQL commands to set the Hadoop environment. The Data Integration Service executes the environment SQL at the beginning of each Hive script generated by a HiveServer2 job.

The following rules and guidelines apply to the usage of environment SQL:

You can use environment SQL to define Hadoop or Hive parameters that you want to use in the PreSQL commands or in custom queries.
If you use multiple values for the Environment SQL property, ensure that there is no space between the values.

The Data Integration Service uses HiveServer2 to process portions of some jobs by running HiveServer2 tasks on the Spark engine. When you import the cluster configuration through the admin tool, you can choose to create connections. The engine type property is populated by default based on the distribution.

When you manually create a connection, you must configure the engine type. Use the following engine type, depending on the Hadoop distribution:

Amazon EMR - Tez
Azure HDI - Tez
Cloudera CDH - MRv2
Cloudera CDP - Tez
Hortonworks HDP - Tez
Mapr - MRv2

List of advanced properties that are unique to the Hadoop environment. The properties are common to the Blaze and Spark engines. The advanced properties include a list of default properties.

You can configure run-time properties for the Hadoop environment in the Data Integration Service, the Hadoop connection, and in the mapping. You can override a property configured at a high level by setting the value at a lower level. For example, if you configure a property in the Data Integration Service custom properties, you can override it in the Hadoop connection or in the mapping. The Data Integration Service processes property overrides based on the following priorities:

Mapping custom properties set using
infacmd ms runMapping
with the
-cp
option
Mapping run-time properties for the Hadoop environment
Hadoop connection advanced properties for run-time engines
Hadoop connection advanced general properties, environment variables, and classpaths
Data Integration Service custom properties