Chef Infra Client Security

The Chef Infra Server API handles all communication between Chef Infra Client or Chef Workstation. The Chef Infra Server API is an authenticated REST API, which means all requests require authentication and authorization. The Chef Infra tools such as knife and chef-server commands use the Chef Infra Server API for you.

Authentication

The authentication process ensures that Chef Infra Server only responds to requests made by trusted users or clients. Chef Infra Server uses public key encryption. You create the public and private keys when you configure Chef Infra Client or setup Chef Workstation.

Chef Infra Server stores the public key
Chef Workstation saves the private key in ~/.chef/
Chef Infra Client saves the private key in /etc/chef

Both Chef Infra Client and Chef Workstation communicate with the Chef Infra Server using the Chef Infra Server API. Each time that Chef Infra Client or Chef Workstation makes a request to Chef Infra Server, they use a special group of HTTP headers and sign the rest with their private key. The Chef Infra Server then uses the public key to verify the headers and the contents.

chef-validator

Every request made by Chef Infra Client to the Chef Infra Server must be an authenticated request using the Chef Infra Server API and a private key. When Chef Infra Client makes a request to the Chef Infra Server, Chef Infra Client authenticates each request using a private key located in /etc/chef/client.pem.

The private key doesn’t yet exist the first time that Chef Infra Client runs from a new node.

During the first Chef Infra Client run:

Chef Infra Client uses the chef-validator private key, located in /etc/chef/validation.pem to register with Chef Infra Server
Chef Infra Server assigns Chef Infra Client a private key for all future authentication requests to the Chef Infra Server
Chef Infra Client saves the private key on the node as /etc/chef/client.pem

If the request to communicate with Chef Infra Server with the chef-validator key fails, then the entire first Chef Infra Client run fails.

After the first completed Chef Infra Client run, delete the chef-validator private key at /etc/chef/validation.pem

SSL Certificates

Warning

The following information applies to on-premises Chef Infra Server and doesn’t apply to Hosted Chef.

Chef Infra Server 12 and later enables SSL verification by default for all requests made to the server, such as those made by knife and Chef Infra Client. The certificate that is generated during the installation of the Chef Infra Server is self-signed, which means the certificate is not signed by a trusted certificate authority (CA) recognized by Chef Infra Client. The certificate generated by the Chef Infra Server must be downloaded to any machine from which knife and/or Chef Infra Client will make requests to the Chef Infra Server.

For example, without downloading the SSL certificate, the following knife command:

knife client list

responds with an error similar to:

ERROR: SSL Validation failure connecting to host: chef-server.example.com ...
ERROR: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 ...

This is by design and will occur until a verifiable certificate is added to the machine from which the request is sent.

trusted_certs Directory

Your organization may use a private Certificate Authority (CA) to generate SSL Certificates or they may create self-signed SSL certificates to use on internal networks or during software development and testing.

The trusted_certs directory on Chef Workstation and in Chef Infra Client works as a trusted certificate store for all communication in the Chef Infra system. Chef Infra trusts all SSL certificates stored in this directory–including certificates that aren’t issued by a trusted Certificate Authority (CA).

Place private and self-signed certificates in the trusted_certs directory to use them within Chef Infra Client and Workstation tools.

Use the chef_client_trusted_certificate Chef Infra Client resource to manage these certificates continuously.

trusted_certs Locations

Chef Workstation

When you install Chef Workstation, it creates a trusted_certs directory located at:.

Windows: C:\.chef\trusted_certs
All other systems: ~/.chef/trusted_certs

Chef Infra Client Nodes

When you bootstrap a node, the Chef Infra Client copies the SSL certificates for the Chef Infra Server onto the node. The trusted_certs directory on the node is located at:

Windows: C:\chef\trusted_certs
All other systems: /etc/chef/trusted_certs

SSL_CERT_FILE

Use the SSL_CERT_FILE environment variable to specify the location for the SSL certificate authority (CA) bundle for Chef Infra Client.

A value for SSL_CERT_FILE isn’t set by default. Unless updated, the locations in which Chef Infra will look for SSL certificates are:

Chef Infra Client: /opt/chef/embedded/ssl/certs/cacert.pem
Chef Workstation: /opt/chef-workstation/embedded/ssl/certs/cacert.pem

To use a custom CA bundle, update the environment variable to specify the path to the custom CA bundle. The first step to troubleshoot a failing SSL certificate is to verify the location of the SSL_CERT_FILE.

client.rb Settings

Use following client.rb settings to manage SSL certificate preferences:

Setting	Description
`local_key_generation`	Whether the Chef Infra Server or Chef Infra Client generates the private/public key pair. When `true`, Chef Infra Client generates the key pair, and then sends the public key to the Chef Infra Server. Default value: `true`.
`ssl_ca_file`	The file for the OpenSSL key. Chef Infra Client generates this setting automatically.
`ssl_ca_path`	The location of the OpenSSL key file. Chef Infra Client generates this setting automatically.
`ssl_client_cert`	The OpenSSL X.509 certificate for mutual certificate validation. Required for mutual certificate validation on the Chef Infra Server. Default value: `nil`.
`ssl_client_key`	The OpenSSL X.509 key used for mutual certificate validation. Required for mutual certificate validation on the Chef Infra Server. Default value: `nil`.
`ssl_verify_mode`	Set the verification mode for HTTPS requests. The recommended setting is`:verify_peer`. Depending on your OpenSSL configuration, you may need to set the `ssl_ca_path`. Default value: `:verify_peer`. Use `:verify_none` to run without validating any SSL certificates. Use `:verify_peer` to validate all SSL certificates, including the Chef Infra Server connections, S3 connections, and any HTTPS remote_file resource URLs used in a Chef Infra Client run.
`verify_api_cert`	Verify the SSL certificate on the Chef Infra Server. Set to `true`, Chef Infra Client always verifies the SSL certificate. Set to `false`, Chef Infra Client uses `ssl_verify_mode` to determine if the SSL certificate requires verification. Default value: `false`.

Knife Subcommands

The Chef Infra Client includes two knife commands for managing SSL certificates:

Use knife ssl check to troubleshoot SSL certificate issues
Use knife ssl fetch to pull down a certificate from the Chef Infra Server to the /.chef/trusted_certs directory on the workstation.

After the workstation has the correct SSL certificate, bootstrap operations from that workstation will use the certificate in the /.chef/trusted_certs directory during the bootstrap operation.

knife ssl check

Run the knife ssl check subcommand to verify the state of the SSL certificate, and then use the response to help troubleshoot issues that may be present.

Verified

If the SSL certificate can be verified, the response to

knife ssl check

is similar to:

Connecting to host chef-server.example.com:443
Successfully verified certificates from 'chef-server.example.com'

Unverified

If the SSL certificate cannot be verified, the response to

knife ssl check

is similar to:

Connecting to host chef-server.example.com:443
ERROR: The SSL certificate of chef-server.example.com could not be verified
Certificate issuer data:
  /C=US/ST=WA/L=S/O=Corp/OU=Ops/CN=chef-server.example.com/emailAddress=you@example.com

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.2u  20 Dec 2019
* Certificate file: /opt/chef-workstation/embedded/ssl/cert.pem
* Certificate directory: /opt/chef-workstation/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/grantmc/Downloads/chef-repo/.chef/trusted_certs"

TO FIX THIS ERROR:

If the server you are connecting to uses a self-signed certificate,
you must configure chef to trust that certificate.

By default, the certificate is stored in the following location on the
host where your Chef Infra Server runs:

  /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently:

  /Users/grantmc/Downloads/chef-repo/.chef/trusted_certs)

using SSH/SCP or some other secure method, then re-run this command to
confirm that the certificate is now trusted.

knife ssl fetch

Run the knife ssl fetch to download the self-signed certificate from the Chef Infra Server to the /.chef/trusted_certs directory on a workstation.

Verify Checksums

The SSL certificate that is downloaded to the /.chef/trusted_certs directory should be verified to ensure that it is, in fact, the same certificate as the one located on the Chef Infra Server. This can be done by comparing the SHA-256 checksums.

View the checksum on the Chef Infra Server:

ssh ubuntu@chef-server.example.com sudo sha256sum /var/opt/opscode/nginx/ca/chef-server.example.com.crt

The response is similar to:

<ABC123checksum>  /var/opt/opscode/nginx/ca/chef-server.example.com.crt

View the checksum on the workstation:

gsha256sum .chef/trusted_certs/chef-server.example.com.crt

The response is similar to:

<ABC123checksum>  .chef/trusted_certs/chef-server.example.com.crt

Verify that the checksum values are identical.