Table of Contents

Search

  1. Preface
  2. Introduction to Mass Ingestion
  3. Prepare
  4. Create
  5. Deploy
  6. Run
  7. Monitor
  8. infacmd mi Command Reference

Mass Ingestion Guide

Mass Ingestion Guide

Mass Transformations

Mass Transformations

Configure mass transformations to define any parameters that you want to apply across all source tables that you ingest from a relational database.
To set parameters for specific tables, configure the transformation override.
The following image shows the Mass Transformations page of the mass ingestion specification:
This screenshot shows the Mass Transformations page of the mass ingestion specification. On the Mass Transformations page, you can configure the following parameters: Filter By, Drop Columns, Trim, Conver to Upper Case, Convert to Lower Case, Replace Columns, and Replace Criteria. In the top-right corner, you have the option Next to go to the next page, or the button X to discard the specification.
The following table describes the parameters that you can configure:
Property
Description
Filter By
Filters rows in the target table based on criteria for a column in the table. Enter a filter clause to determine the criteria, such as
STATE='California'
. You can use any transformation language functions in the filter clause. The filter clause must evaluate to TRUE or FALSE.
To set criteria for multiple columns, use the operators AND and OR. For example, use the operator AND to specify a filter clause such as
STATE='California' AND STATUS='Single'
.
Drop Columns
Drops columns in the target table. Enter each column as a string and separate column names with a comma, or specify a regular expression.
For example, if you enter
COL1
, the column COL1 will be dropped in the target table.
If you enter a regular expression such as
.*SSN.*
, columns that have SSN in the column name will be dropped.
Trim
Trims column values in the target table to remove spaces before and after the values. Enter each column as a string and separate column names with a comma, or specify a regular expression.
For example, if you enter
COL1
, values in the column COL1 will be trimmed in the target table.
If you enter a regular expression such as
.*SSN.*
, values in columns that have SSN in the column name will be trimmed.
Convert to Uppercase
Converts column values in the target table to uppercase. Enter each column as a string and separate column names with a comma, or specify a regular expression.
For example, if you enter
COL1
, values in the column COL1 will be converted to upper case in the target table.
If you enter a regular expression such as
.*SSN.*
, values in columns that have SSN in the column name will be converted to upper case.
Convert to Lowercase
Converts column values in the target table to lowercase. Enter each column as a string and separate column names with a comma, or specify a regular expression.
For example, if you enter
COL1
, values in the column COL1 will be converted to lower case in the target table.
If you enter a regular expression such as
.*SSN.*
, values in columns that have SSN in the column name will be converted to lower case.
Replace Columns
Replaces column values in the target table. Enter each column as a string and separate column names with a comma, or specify a regular expression.
For example, if you enter
COL1
, values in the column COL1 will be replaced in the target table.
If you enter a regular expression such as
.*SSN.*
, values in columns that have SSN in the column name will be replaced.
If you specify columns to replace, you must specify the replace criteria.
Replace Criteria
Required if you specify columns to replace. Determines how to replace column values in the target table. You can select
Pattern
or
Entire String
.
If you select
Entire String
, all values in the columns that you specify are replaced by the value that you configure.
If you select
Pattern
, enter the pattern to be replaced. Then enter the value to replace the pattern.
Pattern
Required if you configure the replace criteria to be a pattern. Determines the pattern to replace. The pattern must be a regular expression. For example, if you want to replace the values in columns that contain Social Security numbers, you can enter the pattern:
^\d(3)-?\d(2)-?\d(4)$
The pattern replaces all values that correspond to the pattern. In the column that contains Social Security numbers, the pattern replaces the entire Social Security number.
To mask only the first five digits of the Social Security number, you can enter the pattern:
^\d(3)-?\d(2)$
Value
Required if you configure columns to replace. Replaces the pattern or the entire string according to the configured criteria.
If you select
Entire String
for the replace criteria, the value that you enter replaces the values in all of the columns that you specify to replace. For example, if you replace COL1 and COL2 and enter the value
XXX
, all values in columns COL1 and COL2 are replaced with the value XXX.
If you select
Pattern
for the replace criteria, the value that you enter replaces all values that correspond to the pattern.
For example, you want to mask the entire Social Security number according to the pattern:
^\d(3)-?\d(2)-?\d(4)$
Enter the value
XXX-XX-XXXX
. All values in the SSN column will appear as XXX-XX-XXXX.
You might also want to mask only the first five digits of the Social Security number according to the pattern
^\d(3)-?\d(2)$
Enter the value
XXX-XX
. The first five digits of every Social Security number will appear as XXX-XX. For example, if the original Social Security number is 123-45-6789, the replaced value is XXX-XX-6789.
If you configure parameters for a table column that does not exist in all of the tables, the tables where the column does not exist will fail to be ingested. You must reconfigure the parameters for each table where the column does not exist when you configure the transformation override.

0 COMMENTS

We’d like to hear from you!