Install Chef Automate

[edit on GitHub]

_images/chef_automate_full.png

A Chef Automate installation consists of a minimum of two servers:

  • Chef server

    Contains the cookbooks and data used to build, test, and deploy your components within Chef Automate and your infrastructure

  • Chef Automate server

    Coordinates the process of moving a change through the workflow pipeline as well as providing insights and visualizations about your Chef Automate cluster

Optional components include:

  • Push jobs server

    Runs push jobs, which is used to create infrastructure nodes for deployment testing and is also needed if you use push jobs-based build nodes as part of your testing and deployment process

  • Runners/build nodes

    Perform the work of running builds, tests, and deployments out of Chef Automate. These are only required when using the workflow capabilities of Chef Automate

  • Chef Compliance server

    Used in conjunction with the audit cookbook to run InSpec profiles as part of a Chef client run

Prerequisites

Chef Automate requires a license from Chef to install. You will receive an email with a license key valid for 30 days after you complete the Chef Automate registration. If you would like to continue using Chef Automate after that period, please contact awesome@chef.io for a permanent key.

Platform Support

Please see the platforms topic for details on supported platforms, architectures, and versions for both Chef Automate servers and runners.

Hardware Requirements

Chef Automate deployments have the following hardware requirements:

Function vCPU RAM Free disk space (in /var)
Chef Automate server 4 16GB* 80GB
Chef server (must be v12). See additional information in note, below. 4 8GB 80GB
Build nodes/Runners (Optional, but required if you use workflow) 2 4GB 60GB

*If you use your own Elasticsearch cluster instead of the single Elasticsearch instance provided with Chef Automate, then the Chef Automate server only requires 8 GB of RAM.

Node Hostnames and Network Access

The automated configuration of Chef Automate and Chef servers use the hostname command to determine the visible fully-qualified domain name (FQDN) of the node. Prior to installation, ensure that hostname and hostname -f on each node resolves to a matching, correct FQDN that is visible to the other nodes in the cluster. If necessary, update the /etc/hosts file on the nodes to ensure that the names resolve.

Disable IPv6 on the host and remove any ip6 settings found in /etc/hosts. The host should also point its name at its own external ip address in /etc/hosts.

The Chef Automate server hostname is also expected to match the hostname that you will use to work with Chef Automate via its web interface. It is not currently possible to use the Chef Automate web interface if the host name used in the URL does not match the one it is configured with.

Chef Automate has the following network and port requirements. At a minimum the following machines must be able to reach each other:

  • Chef Automate server -> Chef server
  • Build node/Runner -> Chef Automate server
  • Build node/Runner -> Chef server
Ports Description Server State
10000-10003 TCP Push Jobs Chef server LISTEN
8989 TCP Delivery Git (SCM) Chef Automate server LISTEN
443 TCP HTTP Secure Chef server, Chef Automate server LISTEN
22 TCP SSH All LISTEN
80 TCP HTTP Chef server, Chef Automate server LISTEN

Note

Any build nodes/runners must be accessible from the Chef Automate server over SSH and they must have a user account configured that has sudo privileges.

Chef Server Installation and Configuration

Chef Automate associates with a Chef server for application/cookbook building and testing as well as proxying cluster data from nodes for visualization purposes. Because of hardware requirements for both Chef server and Chef Automate, it is recommended that Chef server is installed on a separate node from Chef Automate server.

Also, if you have an existing Chef server installation, it is recommended that you have a separate organization for use with Chef Automate. This keeps any existing production organizations separate from the organization used by runners and infrastructure nodes in your Chef Automate cluster.

Instructions for creating a new organization can be found below under the subheading Create a User and Organization to Manage Your Cluster.

If you already have an existing Chef server and wish to manage infrastructure nodes for deployment testing (or want to use push jobs-based build nodes), update it with the push jobs server add-on.

If you don’t have an existing Chef server installed and configured, the steps below will configure a minimal Chef server for use with Chef Automate.

Chef Server Installation

The standalone installation of Chef server creates a working installation on a single server. This installation is also useful when you are installing Chef server in a virtual machine, for proof-of-concept deployments, or as a part of a development or testing loop.

To install Chef server 12:

  1. Download the package from https://downloads.chef.io/chef-server/.

  2. Upload the package to the machine that will run the Chef server, and then record its location on the file system. The rest of these steps assume this location is in the /tmp directory.

  3. As a root user, install the Chef server package on the server, using the name of the package provided by Chef. For Red Hat and CentOS 6:

    $ rpm -Uvh /tmp/chef-server-core-<version>.rpm
    

    For Ubuntu:

    $ dpkg -i /tmp/chef-server-core-<version>.deb
    

    After a few minutes, the Chef server will be installed.

  4. Run the following to start all of the services:

    $ chef-server-ctl reconfigure
    

    Because the Chef server is composed of many different services that work together to create a functioning system, this step may take a few minutes to complete.

Create a User and Organization to Manage Your Cluster

As noted above, it’s a best practice to use a separate organization when managing nodes in a Chef Automate cluster. Perform the following steps to create a new administrator user and a new organization for your Chef Automate cluster:

  1. Create a user named delivery, and specify a first name, last name, email address, and password. A private key will be generated for you, so specify where to save that key using the --filename option with an absolute path to its intended location.

    sudo chef-server-ctl user-create delivery FIRST_NAME LAST_NAME EMAIL_ADDRESS 'PASSWORD' --filename AUTOMATE_CHEF_USER_KEY
    

    The path to the key is referenced as AUTOMATE_CHEF_USER_KEY in step 4 of Chef Automate Server Installation and Configuration.

  2. Create an organization for managing your Chef Automate cluster and associate the Chef Automate delivery user with it.

    sudo chef-server-ctl org-create AUTOMATE_CHEF_ORG 'org description' --filename ~/AUTOMATE_CHEF_ORG-validator.pem -a delivery
    

    The organization name (denoted by the placeholder AUTOMATE_CHEF_ORG above) must begin with a lower-case letter or digit, may only contain lower-case letters, digits, hyphens, and underscores, and must be between 1 and 255 characters. For example: 4thcoffee.

    The full name must begin with a non-white space character and must be between 1 and 1023 characters. For example: 'Fourth Coffee, Inc.'.

    The --association_user (-a) option will associate the delivery user with the admins security group on the Chef server.

    An RSA private key is generated automatically. This is the chef-validator key and should be saved to a safe location. The --filename option will save the RSA private key to the specified absolute path.

Note

The --filename option is used so that the validator key for your organization will not be shown on-screen. The key is not required for this process.

Push Jobs Server Installation (Optional)

Chef Automate, through the delivery-truck cookbook, can use push jobs to spin up infrastructure environments for deployment testing and can also be used to coordinate build jobs across build nodes when using the push jobs-based job dispatch system.

Push jobs server is available as an add-on to Chef server. If you only wish to use push jobs for deployment testing, you can use runners and the new job dispatch system in conjunction with Push jobs server.

Note

Chef Automate is fully compatible with Push jobs server 1.x and 2.x. Please use 2.x for new installations. Information about upgrading from Push jobs server version 1.x to 2.x can be be found here.

Download the appropriate package for your platform from https://downloads.chef.io/push-jobs-server/ and copy it to the Chef server. The location that it’s been saved to is referred to as PATH_TO_DOWNLOADED_PACKAGE.

Run the command below on the Chef server:

sudo chef-server-ctl install opscode-push-jobs-server --path PATH_TO_DOWNLOADED_PACKAGE

After it has been installed, you must reconfigure it to complete your setup of Push jobs server.

sudo opscode-push-jobs-server-ctl reconfigure

Completing Setup

Run the following command on the Chef server node to complete setup and configuration of Chef server.

sudo chef-server-ctl reconfigure

Running this reconfigure command may trigger a brief restart of your Chef server. This will typically fall within the standard retry window for Chef clients, so no significant interruption of service is expected.

Chef Automate Server Installation and Configuration

Now that you have your Chef server set up, install and configure Chef Automate by doing the following:

Note

If you plan to use a private Supermarket with your Chef Automate server, please make sure it is set up correctly by following the steps in Install Private Supermarket.

  1. Download and install the latest stable Chef Automate package for your operating system from https://downloads.chef.io/automate/ on the Chef Automate server machine.

    For Debian:

    dpkg -i PATH_TO_AUTOMATE_SERVER_PACKAGE
    

    For Red Hat or Centos:

    rpm -Uvh PATH_TO_AUTOMATE_SERVER_PACKAGE
    
  2. In Chef Automate 0.6.64, you have the option of running the preflight-check command. This command is optional, but you are encouraged to use it, as it can uncover common environmental problems prior to the actual setup process. For example, there may be required ports that are unavailable, which would have to be rectified prior to setup.

    sudo automate-ctl preflight-check
    

    This triggers a series of validation steps on your system that will be sent to stdout as they are run, along with whether they are passing or failing. The end of the check will include a report of all failures and remediation steps that you can take to fix them.

    Note

    As shown in the example above, this command requires root user privileges.

    Please refer to the troubleshooting section for more information about the error codes and remediation steps.

  3. Ensure that the Chef Automate license file and the delivery user key you created earlier in the Chef Server setup are located on the Chef Automate server.

  4. Run the setup command. This command requires root user privileges. Any unsupplied arguments will be prompted for.

    sudo automate-ctl setup --license AUTOMATE_LICENSE \
                            --key AUTOMATE_CHEF_USER_KEY \
                            --server-url https://CHEF_SERVER_FQDN/organizations/AUTOMATE_CHEF_ORG \
                            --fqdn AUTOMATE_SERVER_FQDN \
                            --enterprise ENTERPRISE_NAME \
                            --supermarket-fqdn SUPERMARKET_FQDN (Optional)
    

    All paths called for here should be supplied as the absolute path to a file, including the filename.

    AUTOMATE_LICENSE is the full path and file name of your Chef Automate license file. For example: /root/automate.license.

    Note

    After your Chef Automate server is successfully setup, this file will be copied into the /var/opt/delivery/license directory as delivery.license.

    AUTOMATE_CHEF_USER_KEY is the delivery user key that you created on your Chef server. For example: /root/delivery.pem.

    The --server-url is the URL of your Chef server, which contains the fully-qualified domain name of the Chef server and the name of the organization you created when you created the delivery user.

    AUTOMATE_SERVER_FQDN is the external fully-qualified domain name of the Chef Automate server. This is just the name of the system, not a URL. For example: host.4thcoffee.co.

    ENTERPRISE_NAME is the name of your enterprise. For example: 4thcoffee_inc.

    Note

    Currently, only one enterprise is allowed in Chef Automate.

    If you are using a private Supermarket, tell the setup command about it by supplying the --supermarket-fqdn command line argument:

    --supermarket-fqdn SUPERMARKET_FQDN
    

    Because the Supermarket FQDN argument is optional, it will not be prompted for when not specified. You must include this option to set up the Chef Automate server to interact with a private Supermarket. The setup command can be re-run as often as necessary.

    Note

    To enable Chef Automate to upload cookbooks to a private Supermarket, you have to manually log into the Supermarket server with the delivery user, and when it prompts you to enable the user for Supermarket, enter yes. Also, you must copy the Supermarket certificate file to /etc/delivery/supermarket.crt on the Chef Automate server.

Once setup of your Chef Automate server completes, you will be prompted to apply the configuration. This will apply the configuration changes and bring services online, or restart them if you’ve previously run setup and applied configuration at that time. You can bypass this prompt by passing in the argument --configure to the setup command, which will run it automatically, or pass in --no-configure to skip it.

Note

Your Chef Automate server will not be available for use until you either agree to apply the configuration, or manually run sudo automate-ctl reconfigure.

If you’ve applied the configuration, you will also be prompted to set up a Chef Automate runner and submit additional information. In addition to installing runners during setup, you can also install push jobs-based build nodes after your Chef Automate setup completes using the command sudo automate-ctl install-build-node. If you need to install additional runners, run sudo automate-ctl install-runner. These commands can be run each time you want to install a new build node or runner. See the next section for installation instructions.

After setup successfully completes and a configuration has been applied, login credentials are reported in the completion output; however, they are also saved to /etc/delivery/ENTERPRISE_NAME-admin-credentials.

And if you don’t have DNS, specify the fully-qualified domain names for your Chef server and Chef Automate server in /etc/hosts:

CHEF_SERVER_IP         CHEF_SERVER_FQDN
AUTOMATE_SERVER_IP     AUTOMATE_SERVER_FQDN

Note

If your environment requires going through a proxy server, please see About Proxies for information on how to configure proxy settings.

If you plan on using the workflow capabilities of Automate, proceed to the next section to setup your build nodes/runners. After they are set up, you can attempt to run an initial application or cookbook change through your Chef Automate server.

For more information about automate-ctl and how to use it, see automate-ctl (executable).

Configure node data collection

After you have set up your Chef Server and Chef Automate server, you must perform some simple configuration steps to visualize node data in Chef Automate. This process, along with more advanced data configuration scenarios, is detailed in Configure Data Collection.

Set up a build node/runner (Optional)

Chef Automate’s workflow engine automatically creates phase jobs as project code is promoted through the phases of a workflow pipeline. These phase jobs are dispatched to special nodes, called runners and build nodes, that automatically execute each job as it is created. The previous job dispatch system using push jobs-based build nodes is still supported; however the new SSH-based system using runners is the default job dispatch system and should be used for any new deployment.

Note

ChefDK 2.0 or later should only be installed on runners that are associated with Chef Automate 1.5 or later. Using ChefDK 2.0 on runners that are associated with an earlier version of Chef Automate will result in an error during deployment. If you are running an older version of Chef Automate, you should either downgrade your runners to use ChefDK 1.x or upgrade to Chef Automate 1.5 or later.

The following steps show how to set up a runner from a Chef Automate server. For instructions on how to set up a push jobs-based build node, see Set up a build node.

  1. If you have an on-premises Supermarket installation, copy the Supermarket certificate file to /etc/delivery/supermarket.crt.

  2. Run the install-runner subcommand.

    Important

    The install-runner command will create a new file called job_runner in the /etc/sudoers.d directory to give the runner the appropriate sudo access. If your runner does not have the #includedir /etc/sudoers.d directive included in its /etc/sudoers file, you must put that directive in before you run the install-runner command.

    Note

    You can optionally download the latest ChefDK from https://downloads.chef.io/chefdk/ to specify a local package via --installer. Doing so is useful if you are in an air-gapped environment. Version 0.15.16 or greater of the ChefDK is required. The download location is referred to below as OPTIONAL_CHEF_DK_PACKAGE_PATH. This option cannot be used with the --chefdk-version as the version of the local package will be used.

    automate-ctl install-runner RUNNER_FQDN \
                                SSH_USERNAME \
                                [--password OPTIONAL_SSH_OR_SUDO_PASSWORD] \
                                [--installer OPTIONAL_CHEF_DK_PACKAGE_PATH] \
                                [--ssh-identity-file OPTIONAL_SSH_IDENTITY_FILE] \
                                [--chefdk-version VERSION] \
                                [--port SSH_PORT]
    

    The SSH_USERNAME provided must have sudo access on the intended runner, and at least one of --password PASSWORD or --ssh-identity-file FILE is required by Chef Automate in order to communicate with it.

    If you require a specific version of the ChefDK to be downloaded and installed on your runners, you can specify it in the --chefdk-version option. This is useful if your cookbooks are not compatible the Chef client that comes with the latest version of the ChefDK.

    For more install-runner usage examples, see install-runner, and for more information on runners and the SSH-based job dispatch system, see Runners.

    Note

    Legacy build nodes created by delivery-cluster can be used with a Chef Automate server. Some visibility features are designed to only work with new build nodes and runners installed through the command line process, but the workflow feature in Chef Automate can use legacy, new, or mixed node pools; however, you cannot upgrade a legacy build node to the new build node or runner models. If you would like to use new build nodes/runners, please use fresh hosts or completely wipe your legacy build nodes before attempting to run automate-ctl install-build-node or automate-ctl install-runner.

    Depending on whether you created runners or build nodes, you can view the logs at either /var/log/delivery-ctl/runner-install_$RUNNER_FDQN.log or /var/log/delivery-ctl/build-node-install_$BUILD_NODE_FDQN.log.

    Any existing nodes with the same name as your runner’s FQDN will be overwritten on the Chef server. This will remove any previous run lists or Chef Server configuration on this node. This is done in case the hostname was previously being used for something else. When calling install-runner, it will give you a warning if you will overwrite a node before installation begins, which you can bypass by passing --yes.

Note

Certain sensitive files are copied over to a temporary directory on the build node/runner. In the event of failure after these files have been copied, the installer will attempt to remove them. If it is unable to do so, it will provide you with instructions for doing so manually.

Note

Setting up a build node or a runner involves a Chef client run on the target node. This requires the target node to be able to reach your installation’s Chef server. Especially in setups that involve proxies, connectivity issues abound and lead to hard-to-spot errors. One indicator of not having interacted with the Chef server is this output in your Chef client run (note the “Server Response” section):

================================================================================
Chef encountered an error attempting to load the node data for "bldr-1.example"
================================================================================

Authorization Error
-------------------
Your client is not authorized to load the node data (HTTP 403).

Server Response:
----------------
Cannot fetch the contents of the response.

About Proxies

If the Chef Automate setup process is happening in an environment that is configured to only allow http/https traffic to go through a proxy server, then some additional steps need to be taken.

The http_proxy, https_proxy and no_proxy environment variables will need to be set appropriately for the setup process to complete successfully. These can be set in the environment directly, or added to a knife.rb file (for example, in /root/.chef/knife.rb).

Any host that needs to make outgoing http or https connections will require these settings as well. For example, the Chef Automate server (which makes knife calls to Chef server) and Chef server (for push jobs) should have these configured. To update the Chef Automate server, update /etc/delivery/delivery.rb on your Chef Automate server with the values specified in Proxy Settings. After you have configured your settings, run sudo automate-ctl reconfigure.

For general information on proxy settings, please see About Proxies.

Compliance

Profiles

Chef Automate contains a compliance profiles asset store that provides several built-in profiles covering baseline security checks through CIS benchmarks across multiple operating systems.

In Chef Automate 0.8.5 or later, the compliance profiles asset store is enabled by default. You can manage your profiles through Chef Automate API as well as through the Chef Automate UI. See An Overview of Compliance in Chef Automate for more information on the new integrated compliance functionality.

In Chef Automate version 0.6, the profiles asset store functionality is available; however, you must enable the service by adding this line:

compliance_profiles['enable'] = true

into /etc/delivery/delivery.rb and running automate-ctl reconfigure. The automate-ctl status subcommand should now list the status of the compliance_profiles service.

Also, the profiles in this asset store are managed using the Chef Automate API and cannot be managed through the UI as with the 0.8.5 release.

Scanning

Allows nodes to execute infrastructure tests or compliance profiles as part of the chef-client runs. For more details, see Perform a Compliance Scan in Chef Automate.

Troubleshooting

If you run into issues during during setup or in the use of Chef Automate, see Troubleshooting Chef Automate for debugging tips and remediations.

Delivery-truck setup

Delivery-truck is Chef Automate’s recommended way of setting up build cookbooks. See About the delivery-truck Cookbook for directions on how to get started.