Data Integration Connectors
- Data Integration Connectors
- All Products
Property
| Description
|
---|---|
Connection
| Name of the lookup connection.
|
Source Type
| Type of the source object. Select Single Object or Parameter.
|
Lookup Object
| Name of the Google BigQuery lookup object for the mapping.
|
Multiple Matches
| Behavior when the lookup condition returns multiple matches. You can return all rows, any row, the first row, the last row, or an error.
You can select from the following options in the lookup object properties to determine the behavior:
Only return any row, return all rows, and report error is applicable for mappings in advanced mode.
|
Property
| Description
|
---|---|
Source Dataset ID
| Optional. Overrides the Google BigQuery dataset name that you specified in the connection.
|
Source Table Name
| Optional. Overrides the Google BigQuery table name that you specified in the Lookup transformation.
|
Number of Rows to Read
| Specifies the number of rows to read from the Google BigQuery source table.
|
Allow Large Results¹
| Determines whether
Google BigQuery V2 Connector creates arbitrarily large result tables to query large source tables.
If you select this option, you must specify a destination table to store the query results.
|
Query Results Table Name¹
| Required if you select the
Allow Large Results option.
Specifies the destination table name to store the query results. If the table is not available in the dataset,
Google BigQuery V2 Connector creates the destination table with the name that you specify.
|
Job Poll Interval In Seconds¹
| The number of seconds after which
Google BigQuery V2 Connector polls the status of the read job operation.
Default is 10.
|
Read Mode
| Specifies the read mode to read data from the Google BigQuery source.
You can select one the following read modes:
Default is Direct mode.
|
Number of Threads for Downloading Staging Files¹
| Specifies the number of files that
Google BigQuery V2 Connector downloads at a time to enable parallel download.
This property applies to staging mode.
|
Data format of the staging file¹
| Specifies the data format of the staging file. You can select one of the following data formats:
This property applies to staging mode.
|
Local Stage File Directory¹
| Specifies the directory on your local machine where
Google BigQuery V2 Connector stores the Google BigQuery source data temporarily before it reads the data.
This property applies to staging mode.
|
Staging File Name¹
| Name of the staging file where data from the Google BigQuery source table is exported to Google Cloud Storage.
This property applies to staging mode.
|
Enable Staging File Compression¹
| Indicates whether to compress the size of the staging file in Google Cloud Storage before
Google BigQuery V2 Connector reads data from the staging file.
You can enable staging file compression to reduce cost and transfer time.
This property applies to staging mode.
|
Persist Destination Table¹
| Indicates whether
Google BigQuery V2 Connector must persist the query results table after it reads data from the query results table.
By default,
Google BigQuery V2 Connector deletes the query results table.
|
pre SQL¹
| SQL statement that you want to run before reading data from the source.
For example, if you want to select records in the database before you read the records from the table, specify the following pre SQL statement:
SELECT * FROM [api-project-80697026669:EMPLOYEE.DEPARTMENT] LIMIT 1000;
|
post SQL¹
| SQL statement that you want to run after reading data from the source.
For example, if you want to update records in a table after you read the records from a source table, specify the following post SQL statement:
UPDATE [api-project-80697026669.EMPLOYEE.PERSONS_TGT_DEL]
SET phoneNumber.number=1000011, phoneNumber.areaCode=100 where fullname='John Doe'
|
pre SQL Configuration¹
| Specify a pre SQL configuration.
For example,
DestinationTable:PRESQL_SRC,DestinationDataset:EMPLOYEE,
FlattenResults:False,WriteDisposition:WRITE_TRUNCATE,UseLegacySql:False
|
post SQL Configuration¹
| Specify a post SQL configuration.
For example,
DestinationTable:POSTSQL_SRC,DestinationDataset:EMPLOYEE,
FlattenResults:True,WriteDisposition:WRITE_TRUNCATE,UseLegacySql:False
|
SQL Override Query¹
| Overrides the default SQL query used to read data from the Google BigQuery source.
When you specify SQL override query, you must specify a dataset name in the Source Dataset ID advanced source property.
Ensure that the list of selected columns, data types, and the order of the columns that appear in the query matches the columns, data types, and order in which they appear in the source object.
Ensure that you only map all the columns in the SQL override query to the target.
Does not apply when you enable partitioning.
|
Use Legacy SQL For SQL Override¹
| Indicates that the SQL Override query is specified in legacy SQL.
Use the following format to specify a legacy SQL query for the
SQL Override Query property:
SELECT <Col1, Col2, Col3> FROM [projectID:datasetID.tableName]
Clear this option to define a standard SQL override query.
Use the following format to specify a standard SQL query for the
SQL Override Query property:
SELECT * FROM 'projectID.datasetID.tableName'
|
Label¹
| You can assign a label for the transformation to organize and filter the associated jobs in the Google Cloud Platform Log Explorer.
For more information about labels and their usage requirements, see
Assign a label to the transformations.
|
Billing Project ID¹
| The project ID for the Google Cloud project that is linked to an active Google Cloud Billing account where the Secure Agent runs query and extract jobs.
If you omit the project ID here, the Secure Agent runs query and extract jobs in the Google Cloud project corresponding to the
Project ID value specified in the Google BigQuery V2 connection.
|
Retry Options¹
| Comma-separated list to specify the following retry options:
Use the following format to specify the retry options:
RetryCount:5,RetryInterval:1,RetryExceptions:java.net.ConnectException|java.io.IOException
The retry options are available for preview. Preview functionality is supported for evaluation purposes but is unwarranted and is not production-ready. Informatica recommends that you use in non-production environments only. Informatica intends to include the preview functionality in an upcoming release for production use, but might choose not to in accordance with changing market or technical circumstances. For more information, contact Informatica Global Customer Support. To use the functionality, your organization must have the appropriate licenses.
|
Number of Spark Partitions²
| Specifies the maximum number of partitions that the Spark engine splits the data into.
Default is 1.
|
Lookup Data Filter ¹
| Limits the number of lookups that the mapping performs on the cache of the lookup source table based on the value you specify in the filter condition.
This property is applicable when you select object as the source type and enable lookup cache on the
Advanced tab for the Lookup transformation.
Maximum length is 32768 characters.
For more information about this property, see
Transformations in the Data Integration documentation.
|
¹ Doesn't apply to mappings in advanced mode.
² Applies only to mappings in advanced mode.
|