Release Notes

10.5

Back Next

Status Cache

The memory consumption of the status cache has been improved by up to 50% - depending on the number of data quality results per item. The more DQ statuses attached to a record, the larger the saving.

Multiple load strategies have been implemented which are used in combination to provide the fastest startup time of the cluster. If a server is started, it first will check if it can load the status cache from the local persistence storage. If not, it will check if there are other servers in the cluster already started, and if so, it will load the status cache from them. Only in case the server is the first one or the network load is disabled, the server would have to load the status entries from the database. This logic ensures that the resources on the database are used with care and the most performant way to load the cache content is used at all times.

Load from Database

The status cache initialization will now use multiple threads to load the data quality results from the database. The way to store the data quality results in the database has been enhanced for faster load performance as well. Please note that this will require additional space in the MAIN schema and the very first server start will take some time to optimize the persisted data. Please factor this in when upgrading larger deployments.

New preferences have been added which control the database load:

# Status cache initialization will utilize multiple threads once the threshold has been reached.
# Default: 200000
com.heiler.ppm.status.server/statusCache.parallelTreshold = 200000
  
# The maximum amount of concurrent threads to query the database.
# Default: empty (using "number of DB CPU cores" configured in server.properties)
com.heiler.ppm.status.server/statusCache.parallelDegree =

Load Over Network

In multi-server scenarios, we additionally provide the load of the status cache from already running servers. So, in a multi-server scenario, all but the first server will load the cache from another server. This improves the startup time even further and reduces the overall load on the database during cluster startup (as only one server needs to access the persisted data there). This is especially useful in deployment scenarios in which the IOPS of the database are limited.

# Allows the status cache initialization over network.
# Default: true
com.heiler.ppm.status.server/statusCache.networkEnabled = true
  
# The amount of status cache elements contained in every network request.
# Default: 500000
com.heiler.ppm.status.server/statusCache.networkBatchSize = 500000
  
# The maximum amount of parallel threads to transfer the status cache over the network.
# While there can be only one active socket at a time, the other threads will be preparing their payload.
# In case there are less CPU cores available, all CPU cores except one will be utilized.
# Default: 4
com.heiler.ppm.status.server/statusCache.networkParallelDegree = 4

Load from Local Storage

In case a server is orderly shutdown, it will also store the status cache on the local disk to improve the startup time for the next startup. A corresponding check file will be saved to recognize if the server has been orderly shutdown. The shutdown of the server takes a few minutes extra time now to persist the cache locally depending on the size of the cache.

# Allows the status cache initialization from local storage.
# Default: true
com.heiler.ppm.status.server/statusCache.localStorageEnabled = true
  
# The local file path of the status cache snapshot
# Default: empty (using workspace)
com.heiler.ppm.status.server/statusCache.localStoragePath =
  
# The maximum amount of parallel threads to read the status cache from the local storage.
# This process is I/O bound, i.e. increasing parallel degree could lead to worse performance.
# In case there are less CPU cores available, all CPU cores except one will be utilized.
# Default: 4
com.heiler.ppm.status.server/statusCache.localStorageParallelDegree = 4

Performance

Download Guide

Watch

Comments

Communities

Knowledge Base

Success Portal