Incremental Loading for Cloudera Navigator Resources
Incremental Loading for Cloudera Navigator Resources
The first time that you load a Cloudera Navigator resource, Metadata Manager performs a full metadata load. You can configure Metadata Manager to perform subsequent loads incrementally.
By default, a full load extracts all entities except for filtered Hive and Impala query templates and entities excluded by the search query. Incremental loading causes Metadata Manager to load recent changes to the metadata instead of loading complete metadata.
Incremental loading reduces the amount of time that it takes to load the resource because Metadata Manager extracts fewer entities.
During an incremental load, Metadata Manager extracts only the following entities:
HDFS entities that were created or changed since the previous resource load
Pig tables that were created or changed after the previous resource load
All Hive tables, views, and partitions
Operation executions that were created after the previous resource load
All templates related to the new operation executions
Use the
Enable incremental load
connection property to enable or disable incremental loading for Cloudera Navigator resources.
Even if you enable incremental loading, a complete metadata load occurs in the following circumstances:
You purge the resource.
You cancel the first load of the resource before the ETL task starts.
The first load of the resource fails.
The Cloudera administrator invokes a purge operation in Cloudera Navigator after the last successful metadata load in Metadata Manager.