Enhanced Compliance Report Ingestion
Enhanced Compliance Reporting has been introduced in the Chef Automate which comes with a few new compliance APIs and modified compliance reporting APIs. The existing APIs can be used to work on a specific date. The new modification to the compliance APIs have been introduced to work on the Date Range.
While specifying the date range, the end date will always be the current date whereas the start date can be any date before 90 days.
The modification and introduction of compliance APIs cover the following functionalities:
- Finding the list of nodes over a date range.
- Finding the list of profiles used over a date range.
- Finding the list of controls across scans over a date range.
- Finding the statistical information of compliance scans across the nodes over a date range.
Asset Compliance Reporting gives you count of how many compliance nodes have been reported between the specified date range. The new APIs cover the below functionalities:
- Finding the number of assets reported over a date range.
- Find the number of assets which have not reported over a date range.
- Find the number of assets which can be termed as unreachable.
The asset compliance report also gives the count of how many nodes have not been reported and have been unreachable in the specified date range.
The above-mentioned APIs are also data range specific, i.e., the end date will always be the current date whereas the start date can be any date before 90 days.
Asset reporting can be differentiated into three types:
Reported Asset: Any compliance node which has been sent or reported compliance between a specified data range time is called as reported assets.
Unreported Asset: Any assets which do not send a report in a specified date range is called as unreported assets.
Unreachable Asset: These assets are based on the predefined configuration. An asset is termed as unreachable if it has not sent any compliance report during the specified duration.
To enable the enhanced and asset compliance report:
patch.tomlif one does not already exist for your Chef Automate installation.
Add the following configuration to the
[compliance.v1.sys.service] enable_enhanced_compliance_reporting = true
enable_enhanced_compliance_reportingattribute is set to true, which enables the enhanced and asset compliance report. To disable it, set the value of
Patch the config by running the following command:
chef-automate config patch patch.toml
Once you have enabled the enhanced compliance reporting (
enable_enhanced_compliance_reporting = true), the data from the current indexes get migrated to the new indexes which are used to support the enhanced compliance reporting and asset compliance reporting.
To check the status of the migration, run the following command:
chef-automate enhance-compliance migrate status
If the migration has completed the following message will be shown:
The Migration of compliance controls and assets have completed.
The migration is dependent on :
- The number of controls it needs to migrate across all the nodes
- The size of every compliance report totalling up the total size of reports
- The resource allocated to OpenSearch depending on the type of deployment
Here are some indications of migration performance:
|OpenSearch Deployment Type||Number of controls each report||Total size of report||Total time taken|
|External (8CPU, 32GB RAM)||2000||6GB||60 minutes|
|Internal (4CPU, 16GB RAM)||2000||6GB||48 hours|
Compliance data ingestion ingests data from remote nodes using data-collector API.
The compliance data gets ingested to the OpenSearch indexes but also asynchronously loads data to the new indexes if
enhanced_compliance_reporting is enabled.
These indexes are needed to support the APIs for
The asynchronous data ingestion in to the new indexes are done using workers which are running at the background.
The number of asynchronous data populators can be configured by setting the following configuration:
[compliance.v1.sys.service] control_data_populators_count = 2
Here are some performance numbers executed on a machine with 4 vCPUs and 16 GB of RAM. The ingestion performance depends on:
- The number of CPU of the machine
- The size of memory of the machine
- The size of Compliance report
- The number of asynchronous control data populator.
|Size of the Report||Number of Nodes||Concurrency of Ingestion||Number of async Control Data Populators||Max CPU||Max Memory|
The system gets impacted when
enhanced_compliance_reporting is enabled. All the tests are performed on a minimum system requirement:
- 4 vCPUs
- 16 GB of RAM
Please take note of the following performance test numbers before enabling the feature:
The migration of data to the new index is done in asynchronous manner slowly in the background. The ingestion of data may be run during the migration and find out the impact on the resources. The performance benchmark is executed on an embedded Automate installation running on a machine with 4 vCPUs and 16 GB of RAM.
|Enhanced Compliance Reporting enabled||Migration Running||Compliance Report Size||Concurrency||Max CPU Utilisation||Max Memory Utilisation|
The CPU utilization increment is primarily for OpenSearch service utilization.
The disk usage increases when all the data is migrated from the current indexes to the new indexes. The following number depicts the disk size increase after completion of migration:
|Before Migration||After Migration||% of Increase|
|9.6 GB||13 GB||35%|
The following tests are performed on a machine with 4 vCPUs and 16 GB of RAM by running:
- All the Compliance APIs called in parallel at certain duration
|Enhanced Compliance Reporting Enabled||No of Compliance APIs||Max CPU%||Min CPU %||Max Memory%||Min Memory %|
Was this page helpful?