Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Ask INFA.

Table of Contents

Search

  1. Preface
  2. Introduction to Salesforce Connector
  3. Connections for Salesforce
  4. Synchronization tasks with Salesforce
  5. Mappings and mapping tasks with Salesforce
  6. Replication tasks with Salesforce
  7. Masking tasks with Salesforce
  8. Common configuration
  9. Troubleshooting
  10. Appendix A: Data type reference

Salesforce Connector

Salesforce Connector

Salesforce bulk API limits

Salesforce bulk API limits

Salesforce limits the amount of data that you can read or write through the Salesforce Bulk API.
To improve the performance and reduce the number of API requests for large data sets, you use Salesforce Bulk API.

Batch Limit

You can submit up to 5,000 batches in every 24 hours. You can perform query, write, and delete operations.

Bulk API Writer

Default size of the Bulk API Writer is 10,000 rows for a batch. The task can process up to 5000*10000 which is 50 million records a day. Edit the Secure Agent properties to configure the batch size of the Bulk API Writer.

Bulk API Reader

With Salesforce Bulk API Reader, you can retrieve up to 15 GB data from a query. If the query exceeds 15 GB, the
masking
task fails. You can manually calculate whether the amount of data that you want to process exceeds the Salesforce limitations.
To calculate the amount of data you can query through the Salesforce Bulk API Reader, you can use the following formula:
Sum (the number of bytes in the fields that you want to mask) * (the number of rows in the query results) < 15 GB
In the Salesforce application, view the size of the fields of the objects and add the number of bytes of the fields that you want to mask. Find the total number of rows in the query results for the Salesforce object and apply the formula. If the result is within the 15 GB limit, the task succeeds. If the result is greater than 15 GB, the task fails.

Salesforce Bulk API Limits Example

The Account object contains 100 fields and you want to mask 10 fields of sensitive data. The sum of the size of all the 10 fields that you want to mask is 100 bytes. The total number of rows for the Account object is 500,000.
Use the following formula:
Total query size = 100 * 500,000 = 0.05 GB
The result 0.05 GB is within the 15 GB data limit and the application can process the task successfully.
The following table contains some more examples of total query size calculations for the Salesforce objects:
Objects
Number of Rows in Query Results
Sum of the Size of the Fields to be Masked
Total Query Size
Account
500,000
1,000
0.5 GB
Contact
6,000,000
2,000
12 GB
Lead
8,500,000
2,000
15.8 GB
The total query size for the Account and Contact objects are within the 15 GB data limit and the application processes the task successfully. The total query size for the Lead object is greater than the 15 GB data limit and the application cannot process the data.
To reduce the query for the Lead object, you can use horizontal or vertical partitioning and create multiple
masking
tasks.
In horizontal partitioning, you can split the number of rows in a Salesforce object. Specify a condition in the data filter criteria to reduce the total number of rows in the query results. You can run two
masking
tasks, one with closed Leads and the other with open Leads.
In vertical partitioning, you can split the number of fields in a Salesforce object. For the Lead object, you can create 2
masking
tasks, 1 task to mask 10 fields with 1300 bytes of data, and other task to mask 3 fields with 700 bytes of data.

0 COMMENTS

We’d like to hear from you!