Mass Ingestion Guide

10.2.1

Back Next

Transformation Override for Hive Targets

When you configure a transformation override for a Hive target, you can edit the mass transformations that are applied to specific source tables when they are ingested to the Hive target. You can additionally edit the Hive target table properties for each source table that you ingest.

The following image shows the transformation override that you can configure for a Hive target:

This screenshot shows the Transformation Override page of the mass ingestion specification. Each row defines one table in the specification and each column defines a different parameter that you can configure. The cells define the specific parameter that you configure for a specific table. Hive target properties for each table are also available in the HIVE column. You edit the parameters by clicking an empty box or by clicking the pencil icon. In the top-right corner, you have the option Save to save the specification, or the button X to discard the specification.

You can edit the following properties for each target table:

Filter By. Filter the columns in the source tables.

Drop Columns. Drop columns from the source tables.

Replace Columns. Replace values in the source table columns.

Trim. Trim spaces from the beginning and end of values in source table columns.

Convert to Uppercase. Convert values in source table columns to uppercase.

Convert to Lowercase. Convert values in source table columns to lowercase.

Hive. Edit the Hive properties for the target table.

The following image shows the dialog box that appears when you edit the properties for the Hive target table:

This image shows a dialog box to configure Hive options. Hive Options is selected.

The following table describes the Hive target table properties:

Property	Description
Hive Options	Select this option to configure the Hive target location.
DDL Query	Select this option to configure a custom DDL query that defines how data from the source tables is loaded to the target tables.
Storage Format	Required. The storage format of the target tables. You can select Text, Avro, Parquet, or ORC. Default is Text.
External Location	The external location of the Hive target. Enter an external location to specify a location other than the default Hive warehouse directory. A sub-directory is created under the specified external location for each source that is ingested. For example, you can enter /temp . A source table named PRODUCT is ingested to the external location /temp/PRODUCT/
Partition Key	The partition key for the target Hive table.
Clustered By	The cluster key for the target Hive table.
Number of buckets	Required if you specify a cluster key. The number of buckets to divide the target Hive table.

The following image shows the dialog box that appears when you edit the DDL query in the properties for the Hive target table:

This image shows a dialog box to configure Hive options. DDL Query is selected.

To edit the DDL query, you can use SQL statements and the following placeholders:

INFA_TABLE_NAME: Fetches the target table name at run time.
INFA_COLUMN_LIST: Fetches a list of columns in the target table at run time.

Rename Saved Search

Table of Contents

Mass Ingestion Guide

Mass Ingestion Guide

Transformation Override for Hive Targets

Transformation Override for Hive Targets