Mapping Validation

Optimization

Set the optimizer level to none or minimal if a mapping validates but fails to run. If you set the optimizer level to use cost-based or semi-join optimization methods, the Data Integration Service ignores this at run-time and uses the default.
The run-time engine does not honor the early projection optimization method in all cases. If the Data Integration Service removes the links between unused ports, the run-time engine might reconnect the ports.
When you use the auto optimizer level, the early selection optimization method is enabled if the mapping contains any data source that supports pushing filters to the source on Spark or Databricks Spark engines. For more information about optimizer levels, see the
Informatica Developer Mapping Guide
.
When the Spark engine runs a mapping, it processes jobs on the cluster using HiveServer2 in the following cases:
The mapping writes to a target that is a Hive table bucketed on fields of type char or varchar.
The mapping reads from or writes to Hive transaction-enabled tables.
The mapping reads from or writes to Hive tables where column-level security is enabled.
The mapping writes to a Hive target and is configured to create or replace the table at run time.

High Precision

Overflow Values

Null and Invalid Values and Rejected Rows

The Spark run-time engine drops rejected rows even if you configure the mapping to forward rejected rows. The rejected rows are not written to the session log file.
If an expression results in numerical errors, such as division by zero or SQRT of a negative number, the Spark engine returns null. In the native environment, the same expression results in a row error.
The Hadoop environment treats "/n" values as null values. If an aggregate function contains empty or NULL values, the Hadoop environment includes these values while performing an aggregate calculation.
The Spark engine writes null values for rows when invalid values are passed in the following situations:
The
terms
argument in PV, FV, PMT, and RATE finance functions passes a 0 value. The value of
terms
must be an integer greater than 0.
The
month
argument in the MAKE_DATE_TIME function passes an invalid value. The value of
month
must be from 1 to 12.
In the native environment, the Data Integration Service rejects the row and does not write it to the target.
If data overflow occurs, the Spark engine returns null. For example, if you use the expression
TO_DECIMAL(12.34, 2)
in a port that has a precision of 3 and a scale of 2, the return value is null. The null value will be propagated through the mapping. The mapping might overwrite it using a default value, detect it using the function IS_NULL, and write the null value to the target.

Data Conversions

The results of arithmetic operations on floating point types, such as Decimal, can vary between the native environment and a Hadoop environment. The difference between the results can increase across multiple operations.
When the number of fractional digits in a double or decimal value exceeds the scale that is configured in a decimal port, the Spark engine trims trailing digits, rounding the value if necessary.
If you use Hive 2.0 or higher, the Spark engine guarantees scale values. For example, when the Spark engine processes the decimal
1.1234567
with scale 9 using Hive 2.0, the output is
1.123456700
.
The Spark engine cannot process dates to the nanosecond. It can return a precision for date/time data up to the microsecond.

Scale

Function

Avoid including single and nested functions in an Aggregator transformation. The Data Integration Service fails the mapping in the native environment. It can push the processing to the Hadoop environment, but you might get unexpected results. Informatica recommends creating multiple transformations to perform the aggregation.
The Spark METAPHONE function uses phonetic encoders from the
org.apache.commons.codec.language
library. When the Spark engine runs a mapping, the METAPHONE function can produce an output that is different from the output in the native environment. The following table shows some examples:
String	Data Integration Service	Spark Engine
Might	MFT	MT
High	HF	H
If you use the TO_DATE function on the Spark engine to process a string written in ISO standard format, you must add
*T*
to the date string and
*”T”*
to the format string. The following expression shows an example that uses the TO_DATE function to convert a string written in the ISO standard format YYYY-MM-DDTHH24:MI:SS:
TO_DATE(‘2017-11-03*T*12:45:00’,’YYYY-MM-DD*”T”*HH24:MI:SS’)
The following table shows how the function converts the string:
ISO Standard Format	RETURN VALUE
2017-11-03T12:45:00	Nov 03 2017 12:45:00
The UUID4 function is supported only when used as an argument in UUID_UNPARSE or ENC_BASE64.
The UUID_UNPARSE function is supported only when the argument is UUID4( ).

Other Rules and Guidelines