task several times and uses machine learning to assess the performance of each run. It uses the information to create a tuning recommendation for the set of Spark properties that optimizes task performance. CLAIRE Tuning considers parameters such as the complexity of the mapping, the size of the data, and the processing capacity on the
advanced cluster
.
You can run initial tuning or enable continuous tuning. When you run initial tuning, you can view the tuning recommendation to see a list of recommended Spark properties and their values. You can apply the recommendation to use the values in the
mapping
task. When you enable continuous tuning, CLAIRE silently monitors the
mapping
task and adjusts the Spark properties over time.
Continuous tuning is more effective if you run initial tuning first. During initial tuning, CLAIRE gets an optimized set of Spark properties that it can use as a baseline to make additional adjustments during continuous tuning.
If you run initial tuning on a
mapping
task that incrementally loads files, tuning runs on all of the source files. The recommended properties and values might not be optimal for future jobs that load and process only modified files.