Table of Contents

Search

  1. Preface
  2. Introduction to Microsoft Azure Synapse SQL Connector
  3. Connections for Microsoft Azure Synapse SQL
  4. Mappings for Microsoft Azure Synapse SQL
  5. Upgrading to Microsoft Azure Synapse SQL Connector
  6. Migrating a mapping
  7. SQL ELT optimization
  8. Data type reference
  9. Troubleshooting

Microsoft Azure Synapse SQL Connector

Microsoft Azure Synapse SQL Connector

Rules and guidelines for SQL ELT optimization

Rules and guidelines for
SQL ELT optimization

Certain rules and guidelines apply for
SQL ELT optimization
to a Microsoft Azure Synapse SQL database.
General rules and guidelines
Consider the following rules and guidelines when you enable a mapping for
SQL ELT optimization
:
  • When you use Rank transformation, the data is not stored as per the Rankindex order in the database.
  • When you read varchar or nvarchar fields and use the Rank transfomation, the target data is sorted as per collation of the database.
  • When you use Rank transformation, the session sort order that you specify in the advanced session property is not considered in a
    Microsoft Azure Synapse SQL
    mapping.
  • When you select multiple objects as the source type, you cannot read more that two objects.
  • When you use a custom query as the source type, the first three letters of the table name alias is always
    INF
    .
  • To write date/time fields to a Microsoft Azure Synapse SQL target created at the runtime, edit the metadata for the target fields and change the native data type of all the date/time fields to datetime2. Else, the mapping fails.
  • When you use a custom query as the source type, ensure that the table name or column name that you specify in the query does not contain unicode characters. Else, the mapping fails.
  • When you use a Filter transformation, ensure that the filter value does not contain special characters. Else, the mapping does not write any rows to the target.
  • You can use the update and delete operations only for an existing target.
  • When you enable the
    Create New at Runtime
    option in a
    Microsoft Azure Synapse SQL
    target to write data from file such as Delimited, ORC, or Parquet, delete the
    FileName
    field.
  • The reject threshold for a
    Microsoft Azure Synapse SQL
    target is considered for the whole data set, not for a batch of data.
  • Do not use the Aggregator transformation with update operation.
  • Do not configure a nested aggregate function in an Aggregator transformation.
  • Do not configure parallel transformations in a mapping.
  • Ensure that the Expression transformation does not contain a variable field.
  • Use the ISNULL function within the IIF clause in Aggregator and Expression transformations.
  • Use the IN function in a Filter transformation or within the IIF clause in Aggregator and Expression transformations.
  • When you use STDDEV, VARIANCE, and TRUNC functions in an Aggregator transformation, data corruption is possible if the table has a defined decimal column with precision more than 28, but the table contains data with precision less than 28.
  • When you use the
    lastruntime
    variable in an Expression transformation in full
    SQL ELT optimization
    , you must use the
    yyyy-mm-dd hh:mi:ss
    date format.
    For example,
    to_char($lastruntime,'yyyy-mm-dd hh:mi:ss')
  • You cannot use the
    ISNULL(column name1)<>ISNULL(column name2)
    syntax.
    Use the following syntax for Filter and Router transformations:
    (ISNULL(column name1) AND ISNULL(column name2)) OR ( NOT ISNULL(column name1) AND NOT ISNULL(column name2))
    Use the following syntax for Expression and Aggregator transformations:
    IIF((NOT ISNULL(column name1) AND ISNULL(column name 2)) OR (ISNULL(column name1) AND NOT ISNULL(column name 2)), 1, 0)
  • When you use a Filter or Router transformation and specify a string value in the filter condition, the mapping appends the character N before the condition.
    For example,
    Select*from [Azure].[employee] where (10125 <= "INF1"."EMPLOYEE_ID_NEW") AND ("INF1"."JOB_ID_NEW" =
    N
    'FI_ACCOUNT');
  • When you pass columns with Null values in a Normalizer transformation, Null values are not written to the target.
  • When you push the TO_DATE() function to process in Microsoft Azure Synapse SQL, ensure that the value matches the supported format. Prefix 0 if the value contains a single digit.
  • When you push the TO_DATE() function to the Microsoft Azure Synapse SQL database and the argument contains null data, the mapping runs without
    SQL ELT optimization
    .
  • When you push the MD5() function to process in Microsoft Azure Synapse SQL, you must specify a value of the String data type. Else, the mapping runs without
    SQL ELT optimization
    .
  • When you push the MD5() function to process in Microsoft Azure Synapse SQL, the function returns different values for non-English characters in a mapping and in a mapping enabled with
    SQL ELT optimization
    .
  • You cannot push the MD5() function to Microsoft Azure Synapse SQL if the table or column names contain unicode or special characters.
  • When you push the REG_MATCH() function to process in Microsoft Azure Synapse SQL, it is recommended that you specify all the characters instead of a range of characters in the regular expression in a case-sensitive database. For example, use
    [ABCDEF]
    instead of
    [A-F
    ].
  • When you push the TO_BIGINT() function to convert Varchar data to Bigint in Microsoft Azure Synapse SQL, use the format TO_BIGINT (numeric expression [, 1]) for the function to return the correct values. To round values stored in the Varchar column, convert values to any numeric format, and then convert to Bigint.
  • When you run a mapping enabled with
    SQL ELT optimization
    and push functions to process in Microsoft Azure Synapse SQL, the function considers the collation rules that are set in the database. This causes the functions to return different values in a mapping compared to a mapping enabled with
    SQL ELT optimization
    .
  • When you read from a Microsoft Azure Synapse SQL source and write to multiple targets from a mapping enabled for SQL ELT optimization in a single pipeline, and the Source transformation contains pre-SQL and post-SQL queries, the mapping runs these queries for each of the targets. This might cause data discrepancy. To avoid this, define the pre-SQL and post-SQL queries only in the Target transformation.
Mapping with
Microsoft Azure Synapse SQL
source and target
Use the following rules and guidelines when you configure
SQL ELT optimization
in a mapping that reads from and writes to
Microsoft Azure Synapse SQL
:
  • The source and target table must be in the same database.
  • When you configure filters in a Source transformation, consider the following guidelines:
    • If a mapping contains a Filter transformation and also a filter in the Source transformation, the mapping consolidates the filter conditions from both these transformations to filter the records. However, it is recommended that you use only one of these filters at a time in a mapping.
    • The column names in the filter condition must not contain unicode characters.
    • You cannot use system variables in filters.
    • You cannot use filters when the source object type is query.
  • Ensure that each key column for the update operation is mapped, else the query generation fails and the mapping runs without
    SQL ELT optimization
    .
  • When you use a custom query as the source type, ensure that there is no semicolon at the beginning of or within the query.
  • When you use a custom query as the source type, ensure that the table name or column name that you specify in the query does not contain special characters.
  • You cannot completely parameterize a multi-line custom query using a parameter file.
  • If the custom query starts with the WITH clause, you must enable the
    Create Temporary View
    option to run the mapping with
    SQL ELT optimization
    .
  • You cannot use the ORDER BY clause in a source custom query unless you also specify a TOP, OFFSET, or FOR XML clause in the query.
  • You cannot use the OPTION clause in a custom query or SQL override.
  • When you run a mapping to write data to new target created at runtime and you stop the job using clean stop, the target table created is not dropped.
  • Source
    SQL ELT optimization
    • To run a mapping with
      SQL ELT optimization
      , the mapping must have at least one transformation between the source and target.
    • When you select source
      SQL ELT optimization
      , the task pushes the transformation logic for all the supported transformations downstream in the mapping.
    • When a function within a transformation is not supported, the query generation fails and the mapping runs without
      SQL ELT optimization
      .
    • You can connect the Source transformation only to a single downstream transformation.
Mapping with Microsoft Azure Data Lake Storage Gen2 source and write to a
Microsoft Azure Synapse SQL
target
Use the following rules and guidelines when you configure
SQL ELT optimization
in a mapping that reads from a Microsoft Azure Data Lake Storage Gen2 source and write to a
Microsoft Azure Synapse SQL
target:
  • The Microsoft Azure Data Lake Storage Gen2 account and
    Microsoft Azure Synapse SQL
    account must be associated with the same Azure Active Directory tenant.
  • The Azure Government endpoints for the source and target must be associated with the same Azure Active Directory tenant.
  • You cannot configure a private endpoint or a virtual network to connect to Microsoft Azure Data Lake Storage Gen2. You can use managed identity authentication instead to securely connect to Microsoft Azure Data Lake Storage Gen2.
  • You cannot read objects stored in subdirectories in Microsoft Azure Data Lake Storage Gen2.
  • Ensure that the Microsoft Azure Data Lake Storage Gen2 source object is not parameterized.
  • When you read data from Microsoft Azure Data Lake Storage Gen2, do not use escape characters. Else, it might lead to incorrect results.
  • When you push down a mapping that reads from a Microsoft Azure Data Lake Storage Gen2 source and writes to a Microsoft Azure Synapse SQL target, the mapping task fails to generate the push down query if the task has all the following configurations:
    • The Microsoft Azure Synapse target connection is parameterized.
    • The target operation and Treat Source Rows As property in the mapping is set to Insert.
    • The Treat Source Rows As property is changed to Update in the mapping task.
Mappings in advanced mode
Use the following rules and guidelines when you configure
SQL ELT optimization
in a mapping in advanced mode:
  • When you configure
    SQL ELT optimization
    to push the transformation logic to Microsoft Azure Synapse SQL, the queries generated for a mapping in advanced mode differs from the queries generated for a mapping.
  • When you push functions to process in Microsoft Azure Synapse SQL, ensure that you specify valid parameters and return types for the functions. Otherwise, the mapping fails.
  • When you configure
    SQL ELT optimization
    , ensure that the transformation does not contain a variable port.
  • When you create a new target at runtime to write data to Microsoft Azure Synapse SQL, map all the incoming fields to the target fields.
  • When a mapping enabled with
    SQL ELT optimization
    cannot push certain transformations to Microsoft Azure Synapse SQL, the mapping runs without
    SQL ELT optimization
    and the session log does not log an error.

0 COMMENTS

We’d like to hear from you!