Table of Contents

Search

  1. Preface
  2. Using Metadata Manager
  3. Configuring Metadata Manager
  4. Application Resources
  5. Business Glossary Resources
  6. Business Intelligence Resources
  7. Custom Resources
  8. Data Integration Resources
  9. Data Modeling Resources
  10. Database Management Resources
  11. Universal Resources
  12. Managing Resources
  13. Loading and Monitoring Resources
  14. Managing Permissions
  15. Resource Migration
  16. Repository Migration
  17. Appendix A: Metadata Manager Login
  18. Appendix B: Metadata Manager Properties Files
  19. Appendix C: Resource Configuration Files
  20. Appendix D: Glossary

Metadata Manager Administrator Guide

Metadata Manager Administrator Guide

Incremental Loading for Cloudera Navigator Resources

Incremental Loading for Cloudera Navigator Resources

The first time that you load a Cloudera Navigator resource, Metadata Manager performs a full metadata load. You can configure Metadata Manager to perform subsequent loads incrementally.
By default, a full load extracts all entities except for filtered Hive and Impala query templates and entities excluded by the search query. Incremental loading causes Metadata Manager to load recent changes to the metadata instead of loading complete metadata.
Incremental loading reduces the amount of time that it takes to load the resource because Metadata Manager extracts fewer entities.
During an incremental load, Metadata Manager extracts only the following entities:
  • HDFS entities that were created or changed since the previous resource load
  • Pig tables that were created or changed after the previous resource load
  • All Hive tables, views, and partitions
  • Operation executions that were created after the previous resource load
  • All templates related to the new operation executions
Use the
Enable incremental load
connection property to enable or disable incremental loading for Cloudera Navigator resources.
Even if you enable incremental loading, a complete metadata load occurs in the following circumstances:
  • You purge the resource.
  • You cancel the first load of the resource before the ETL task starts.
  • The first load of the resource fails.
  • The Cloudera administrator invokes a purge operation in Cloudera Navigator after the last successful metadata load in Metadata Manager.

0 COMMENTS

We’d like to hear from you!