You might want to use an incremental load since a full load can be time- and resource-intensive, especially if the data is largely unchanged between runs of a specification. For a more cost-efficient solution, you can use an incremental load to ingest only the incremental data each time that the specification runs.

Before you begin running incremental loads on a specification, run an initial full load. An initial full load allows the Spark engine to internally persist the original source data and use the persisted values as a basis to fetch incremental data in an incremental load. After the Spark engine persists the original source data, you can ingest incremental data based on the original source data to any target.