Retrieve Data for Multiple Confidential Fields and Bulk Upload Records
Retrieve Data for Multiple Confidential Fields and Bulk Upload Records
Effective in version 10.5.7, you can configure the Import Catalog Lineage job to retrieve data from the Elastic Search server for multiple fields with confidential values instead of retrieving data for a single field with confidential value.
You can also configure the bulk upload of records into the Data Privacy Management database, when the Import Catalog Lineage job processes extracted calculated lineage data from the Elastic Search server.
To enable the retrieval of data for multiple confidential fields and bulk upload of records in the Import Catalog Lineage job, add the following options in the custom properties of Data Privacy Management:
IgnoreBatchProcessingForESDataFetch
Set the value to false if you want the Import Catalog Lineage job to retrieve data from the Elastic Search server for multiple fields with confidential values. Default is true.
DpmEvaluatedLineageProcessingBatchSize
Number of fields with confidential values for which the Import Catalog Lineage job retrieves data from the Elastic Search server. Default is 10.
IgnoreLineageBatchCommit
Set the value to false if you want to enable the bulk upload of records into the Data Privacy Management database, when the Import Catalog Lineage job processes extracted calculated lineage data from the Elastic Search server.
LineageJobCommitBatchSize
Number of records that the Import Catalog Lineage job processes for bulk upload into the Data Privacy Management database. Default is 100.