Amazon S3 V2 Connector

Back Next

Troubleshooting for Amazon S3 V2 Connector

Java heap size configuration

This section describes the errors that you might encounter if the JVM options in the Secure Agent is not configured accordingly to read a large number of files.

"ERROR java.lang.OutOfMemoryError: GC overhead limit exceeded" occurs when you run a Mapping task to write large number of records.: To resolve this issue, perform the following tasks to configure the JVM options in the Secure Agent to increase the memory for the Java heap size:
Select
Administrator
Runtime Environments
.
On the
Runtime Environments
page, select the Secure Agent for which you want to increase memory from the list of available Secure Agents.
In the upper-right corner, click
Edit
.
In the
System Configuration Details
section, select the
Type
as
DTM
for the Data Integration Server.
Edit the
JVMOption1
as
-Xms2048m
.
Specify the maximum and minimum heap size based on the data you want to process.

Restart the Secure Agent manually.

"[ERROR] java.lang.OutOfMemoryError: Java heap space" occurs when you run a mapping to write a file of size 1.4 GB or higher and select Informatica Encryption as the encryption type.: To resolve this issue, perform the following tasks to configure the JVM options in the Secure Agent to increase the memory for the Java heap size:
Select
Administrator
Runtime Environments
.
On the
Runtime Environments
page, select the Secure Agent for which you want to increase memory from the list of available Secure Agents.
In the upper-right corner, click
Edit
.
In the
System Configuration Details
section, select the
Type
as
DTM
for the Data Integration Server.
Edit the
JVMOption1
as
-Xmx8046m
.
Restart the Secure Agent manually.

FFParserRetainNullString custom property

When you read from a .csv file that contains a string named

null

, the task does not write any data to the target. To resolve this issue, perform the following tasks and configure the custom property

FFParserRetainNullString

Select

Administrator

Runtime Environments

On the

Runtime Environments

page, select the Secure Agent.

In the upper-right corner, click

Edit

In the

Custom Configuration Details

section, select the

Type

Tomcat JRE

for the Data Integration Server.

Enter the

Name

FFParserRetainNullString

and the

Value

true

Click

Save

RedirectToSessionLog custom property

When you select a source

or a target

Amazon S3 V2 file and configure the Parquet file format option, a large number of log files might be generated. To resolve this issue and disable the logs, perform the following tasks and configure the custom property

RedirectToSessionLog

Select

Administrator

Runtime Environments

On the

Runtime Environments

page, select the Secure Agent.

In the upper-right corner, click

Edit

In the

Custom Configuration Details

section, select the

Type

DTM

for the Data Integration Server.

Enter the

Name

RedirectToSessionLog

and the

Value

Click

Save

Empty struct data in JSON file

If a JSON file has a field with an empty struct data, the Secure Agent ignores the field and reads the remaining fields during metadata read.

For example, if the JSON file has the following data in the first row:

{"id":123,"address":{}}

, the

address

field is ignored and does not appear in the

Fields

tab. If the JSON file has values for the

address

field in the consecutive row, you can use the

Data elements to sample

property to fetch this field.

Amazon S3 bucket does not exist or the user does not have permission to access the bucket

Do not modify the time on the machine that hosts the Secure Agent. The time on the Secure Agent must be correct as per the time zone. Otherwise, the mapping fails with an exception.

Connection timeout error

Network performance and reliability issues when connecting to Amazon S3 can cause slow response times, connection failures, or request errors.

These issues might occur due to suboptimal configuration of client-side connection management parameters such as maximum concurrent connections, connection timeout, socket timeout, and retry settings.

To resolve these issues, configure the following properties for the Secure Agent:

-DAWSS3MaxConnectionPool. The maximum number of simultaneous HTTP connections that the Amazon S3 client can maintain.

-DAWSS3ConnectionTimeOut. The timeout duration in milliseconds to establish a connection to Amazon S3.

-DAWSS3SocketTimeOut. The timeout duration in milliseconds to read data from an established connection.

Perform the following tasks to configure the JVM options in the Secure Agent:

Click

Administrator

Runtime Environments

On the Runtime Environments
page, select the Secure Agent.

Click Edit
.

In the System Configuration Details
section, select

Service

Data Integration Server

and

Type

DTM

Edit and configure the following JVM options:

JVM option	Value
JVMOption1	-DAWSS3MaxConnectionPool=<number of connections>
JVMOption2	-DAWSS3ConnectionTimeOut=<duration in milliseconds>
JVMOption3	-DAWSS3SocketTimeOut=<duration in milliseconds>