Chapter 7. Advanced Topics

Throughout this book we have focused on learning the basic concepts of CFEngine, those that will allow you to set up CFEngine and start using it in a productive manner as soon as possible. However, CFEngine is a vastly more complex framework for configuration management. In this last chapter, I would like to briefly introduce you to some of the more advanced features, so that you can get a taste for the sort of things you can achieve. I will not delve into the full details of the topics presented here, but rather give you some simple examples and pointers to the places where you can find more information.

Setting Up Multiple CFEngine Environments

Like any system that can modify the state of a running machine, CFEngine has the potential to seriously damage your systems if any errors are introduced in its configuration. Due to its very purpose, an error in a CFEngine configuration file can spread almost instantly to a very large number of machines, leaving them incorrectly configured or breaking them completely, rendering them inoperational.

For this reason, it is extremely important to thoroughly test CFEngine configuration files before deploying them to your production machines. One of the best ways to do this is to create different environments for development/testing and production. In this section we will go through some techniques that you can use to achieve this goal.

Ideally, testing environments should be as similar to the production environment as possible, so that the configurations are tested as much as possible in the same way they will be exercised in deployment. My suggestion is to choose a set of your machines for testing. These machines will receive any changes you make to your CFEngine policies as soon as you decide to publish them. Once you are satisfied that the changes behave as intended, they can be made available to the production environment.

Which machines to choose for testing? They should be as close as possible to the production machines and be in regular use. On the other hand, of course, they should be non-critical machines, since they will be subject to possible mistakes in your configuration changes. The world should not end if these machines break. The specific answer depends largely on your environment, but here are some suggestions:

• If you are deploying CFEngine in user workstations, you could include your own machine in the test set. This will allow you to detect fairly soon when something breaks. If you are part of an IT support group, your team members are also good candidates, since they should be able to give you direct feedback and even fix the problem themselves when something breaks. Keep in mind, however, that you should have at least one machine you can use that is not included in the test set, so that if something breaks catastrophically, you will have access to your network and systems to fix things. You could also ask for volunteers among your power users (this depends on the type of users you support—it might be easier to get technically-minded or research-minded users to volunteer), as long as they understand the possible consequences, and are able and willing to give you feedback when things don’t go as expected, as well as when they do.

• If you are deploying CFEngine in servers, you could include in your test set, apart from at least one dedicated test machine, some of the less critical servers. This could mean non-critical internal web servers, redundant mirrors of existing servers, and other servers that would not affect critical production work if they go down. Of course, you should be careful to choose servers that are representative of the kinds of servers you are configuring (i.e., web servers, DNS servers, database servers, etc.) so that all your configurations are exercised properly. If you are frequently configuring new servers, you should include some of those new servers in your test set, to fully exercise your policy’s initial-configuration abilities.

Once you have defined your environments, you can use CFEngine itself to differentiate between them. The technique I am about to show you works well in CFEngine Community, and is also recognized by the CFEngine Enterprise Mission Portal, so if you choose at some point to upgrade to the commercial version, your environments will be properly handled by the graphical interface.

The trick is to store the files for each environment in a different directory under /var/cfengine/masterfiles/ on the policy hub (which will be copied under /var/cfengine/inputs/ on the clients), and instructing the main promises.cf file to load the files from the appropriate directory. The directory structure looks like this:

/var/cfengine/masterfiles
  promises.cf
  failsafe.cf
  Possibly other files
  environment_development/
    env_promises.cf
    Possibly other files
  environment_testing/
    env_promises.cf
    Possibly other files
  environment_production/
    env_promises.cf
    Possibly other files
  environment_none/
    env_promises.cf
    Possibly other files

In the main promises.cf file, we need to define which machines belong in each environment, as well as implement a way to tell CFEngine from which directory the files need to be read. We do this through a bundle called environments(), which you need to add to your /var/cfengine/masterfiles/promises.cf file:

bundle common environments
{
  classes:
      "environment_development" or => {  
                                        "hostname1_example_com",
                                        "ipv4_10_1_2_3",
                                      };
      "environment_testing"     or => {
                                        classmatch(".*_test_example_com"), 
                                        "ipv4_10_1_2",
                                        "suse_11",
                                      };
      "environment_production"  or => {
                                        classmatch(".*_example_com"),  
                                        "ipv4_10_1",
                                      };
  vars:
    any::   
      "active" string => "none", policy => "overridable";
    environment_production::
      "active" string => "production", policy => "overridable";
    environment_testing::
      "active" string => "testing", policy => "overridable";
    environment_development::
      "active" string => "development", policy => "overridable";
}

In a nutshell, this bundle is first defining three classes called environment_development, environment_testing, and environment_production, and then assigning the name of the environment to the variable $(active). This is how it works:

We define environment_development to be a very specific set of machines. Because the or expression means that environment_development will be true if any of the two classes listed is true, the set will include only the machine called hostname1.example.com and the one with IP address 10.1.2.3 will be included. I’m assuming these would be the machines on which the development takes place, plus those used for immediate testing of new changes made to the policies.

The environment_testing class is defined with a slightly larger set of machines. In this case, we use the classmatch() function to indicate that any machines in the .test.example.com domain will be included. Also, any machines in the 10.1.2.0/24 IP address range will be included.

Note

Remember that CFEngine automatically defines hard classes based on a number of automatically-discovered characteristics of the system in which it is running. Several of those hard classes are based on the current IP addresses of the system. If the current host has IP address 10.1.2.5, CFEngine will automatically define the hard classes ipv4_10_1_2_5, ipv4_10_1_2, ipv4_10_1 and ipv4_10.

The final line of this definition includes all machines with SuSE 11 in the testing environment (those in which the hard class suse_11 is defined). This could be useful, for example, if we are in the process of rolling out SuSE 11—we would like those new machines to be automatically configured using the latest available version of our CFEngine policy. This example also shows you that we can use any CFEngine class (not only those related to hostname or IP address) to define our environments.

Finally, we define class environment_production to include the rest of the network. In this case, we are defining all of .example.com, and the whole 10.1.0.0/16 network, to be the production environment.

Now it is time to store the name of the current environment in a string variable, so that we can use it later on to control which files are read. We assign values to the $(active) variable depending on the class that is defined. Note that we order the classes from most generic (any) to most specific (environment_development). This way, if a host belongs to more than one of these classes (for example, host 10.1.2.5 would have both environment_production and environment_testing defined), the most-specific one (in this case, environment_testing) will be used (the last matching class sets the value of the variable). The $(active) variable is defined using the policy => "overridable" attribute, which makes it possible for its value to be reassigned by later statements.

After the environments() bundle is evaluated, the variable $(environments.active) will contain the name of the environment to which the current machine belongs, or "none" if it didn’t match any of the environment classes. We can now use this to load the appropriate env_promises.cf file:

body common control
{
        inputs => {
                    "libraries/cfengine_stdlib.cf",
                    "environment_$(environments.active)/env_promises.cf",   
                  };
        bundlesequence => {
                            "environments",   
                            "main",
                            "env_main",   
                          };
        version => "Community Promises.cf 1.0.0";
}

We use the $(environments.active) variable as part of the filename to load. So for example, when this variable has the value "production", the file environment_production/env_promises.cf will be loaded. You can, of course, load additional files from the environment-specific directory, by making further use of the $(environments.active) variable.

The environments() bundle needs to be called before all others, to ensure the $(environments.active) variable is properly set for the rest of the policy.

We call the env_main() bundle, which must be defined in all versions of the env_promises.cf file, so that evaluation does not fail.

The env_main() bundle can be as simple or as complex as needed. It could be just a jumping off point to multiple other bundles, or a simple report like this:

bundle agent env_main
{
  reports:
    cfengine::
      "Environment: $(environments.active)";
}

You can also use the per-file inputs and dynamic bundlesequence techniques described in Dynamic Loading and Execution to load additional files and execute arbitrary combinations of bundles.

Using a Version-Control System to Separate Environments

One of the most natural mechanisms to separate different types of environments is through the use of version control software. VCS provides many advantages, including the following:

• Keeping track of the changes made to the configuration files. If you need to find the state in which CFEngine was configured months ago, a VCS makes it easy to find the exact state of the configuration files at that point in time.

• Allowing rollback to older versions, in case new versions produce unwanted side effects, or you need to recreate the state of your environment at some point in the past. This is particularly powerful when used with CFEngine—in principle, simply installing an old version of the CFEngine configuration on a system should be enough to roll back the system itself to the state in which it was at that point (at least with respect to the items that are controlled by CFEngine, of course).

• Using tags and branches, you can mark significant points in time, or try out drastic modifications without affecting your main copy of the configuration. This is the particular technique that lends itself to managing different types of environments using a VCS.

The choice of a particular VCS depends on many factors, including your personal preferences and experience, and possibly requirements from your organization. I particularly recommend Git, because it makes it very easy to develop things in an agile fashion. In conjunction with Git, it is particularly recommended to use a model known as git-flow and an associated tool to make it easier to implement, created by Vincent Driessen. Git-flow specifies a model for keeping track of development and production branches, as well as “feature branches” for making drastic changes to the code without affecting any of the two main branches.

Depending on the size of your environment, you may want to have separate branches for testing and development. This would allow you to continue making changes in the development branch while having a frozen version in the testing environment, for later transfer into production.

Warning

Remember that in this configuration, CFEngine will use files according to the content of the environment-specific directories as documented in Setting Up Multiple CFEngine Environments, regardless of Git branches. Make sure you copy the files from one directory to the other when merging branches, or when deploying them. It is also possible to implement things in such a way that the development and production branches are automatically copied into the environment-specific directories, although that is outside the scope of our current description.

The CFEngine policy hub must be configured to automatically check out the Git repository under the /var/cfengine/masterfiles/ directory, so that the policy files are automatically distributed to all the clients. This is easily done with an entry in /etc/crontab or could be scheduled through CFEngine. For example, if /var/cfengine/masterfiles/ is already a checked-out version of the Git repository, the following promise can be used to update it every 30 minutes (it should be added to the update() bundle, in /var/cfengine/masterfiles/failsafe.cf in the commands: section)

    am_policy_hub&(Min00_05||Min30_35)::
      "/usr/bin/git pull origin"
        contain => u_in_dir("$(master_location)"),
        comment => "Update $(master_location) from git repository",
        handle  => "update_masterfiles_from_git";
...
body contain u_in_dir(s)
{
        chdir => "$(s)";
}

Flow of Development and Deployment

The development of new CFEngine policies, or changes to the existing ones, can take place anywhere that the Git repository can be cloned. In my setup, I normally maintain the CFEngine files in my own laptop, making new changes first in the environment_development directory and committing them to my local copy of the repository. Once I’m satisfied enough to make them available for testing, I copy them to the environment_testing directory and push the repository to the master branch of the Git server.

When new changes are pushed onto the master branch, the policy hub updates /var/cfengine/masterfiles/ from the Git repository, and from there distributes the files to all its clients. Machines in the test group will use the files under environment_testing. Thus, when new changes are pushed, they will soon be available automatically for test machines.

At this stage, you should wait and observe the effects of the changes on the test machines.

When you are satisfied with the changes, you can copy the changes to the environment_production directory, push the changes to the repository, and wait for them to propagate.

At this point, the new configuration files will be automatically deployed to the production directories of the policy servers, and from there to the production CFEngine clients the next time they request policy updates.

CFEngine Testing

As in any type of development and deployment, testing in configuration management is essential for ensuring correct operation. However, very little has been written about structured testing mechanisms for configuration management policies.

Normally, testing in configuration management systems is done simply by deploying the modified policies in a set of test systems and letting it run, or making some modifications to the system to see if it executes the necessary configuration changes. This is akin to an application developer running his application to try it out and see if there are any unexpected behaviors. While this type of testing is important and necessary, it suffers from the same problems in CM as it does in software development, namely:

• Incompleteness: It is essentially impossible to exercise all possible paths in the code, and in similar ways it is impossible to exercise all possible system states to see if the new policy behaves correctly in all of them.

• Subjectiveness: if the developer tests the software, the testing is necessarily biased by the developer knowing what changes have been made, and (perhaps even unconsciously) testing only behaviors related to that activity. The same problem occurs in CM if the person or team who makes the changes is the same one that tests the changes. This problem can be alleviated by having separate people perform the testing.

What is needed is a more rigorous and disciplined approach to testing configuration management (in this case, CFEngine) policies. I propose you use two types of testing—behavioral and unit testing—and a tool that can help make testing much easier: Vagrant.

Behavioral Testing for CFEngine Policies

In recent years, Behavior-Driven Development (BDD) has emerged as a powerful way of testing software: the expected behavior of the software under different circumstances is described (often in high-level, human-readable language), from which tests are automatically derived. Then the software is developed or modified to pass those tests.

A similar approach can be had with configuration management. The idea is to describe the desired system behavior under different scenarios and inputs, and then develop the configuration-management policies to ensure those behaviors are achieved. This has been described as Behavior-Driven Infrastructure.

Details about these techniques are outside the scope of this book, but I will point you to some resources for further learning:

• Cucumber is a framework for BDD in which desired behaviors are expressed in near-natural language. It was originally developed for Ruby development, but can be extended to perform any kind of testing.

• Cucumber-nagios is a Cucumber extension that operates as a Nagios plugin, and thus can be used to verify statements made in Cucumber format using the Nagios functionality.

• Patrick Debois’ Collection of Test Driven Infrastructure links is an excellent resource for reading more about this topic.

Unit Testing for CFEngine Policies

Unit testing refers to testing individual components of a system to ensure they perform their functions appropriately.

As with software development, unit testing can be extremely beneficial to CFEngine policies. A CFEngine policy consists of multiple bundles and body components that perform different configuration functions. As such, these bundles can be individually tested to ensure they perform the desired tasks. Performing unit testing on these components as they are written or modified can provide enormous cost savings when debugging complete policies.

I would advise writing unit tests for different components as they are developed, so that the tests can be run when desired, ideally as part of the SCM or build process.

Happily, CFEngine comes with a unit-testing framework included in the distribution, under the tests/ directory. The testall script in this directory is used to run the tests, either individually, by directory or all of them. This is the framework used by the CFEngine developers to test the basic language functionality and to run regression tests when changes are made to the code, but you can also use it for testing your own CFEngine bundles and policies.

An individual test must be a completely self-contained CFEngine policy file (except for libraries or other files that it needs to include as part of the test), and it must contain at least three bundles called init(), test(), and check(). The init() bundle is used to set up the test: to create required input files, for example. The test() bundle is the one that actually performs the test, exercising the functionality that you want to verify. Finally, the check() bundle verifies whether the test was successful, and reports the result. These three bundles are automatically executed by the testall script, and their results collected and presented to the user.

As an example, consider the following CFEngine bundle, which replaces a line in a text file if it matches a regular expression. If the regular expression is not found, the line of text is added to the file. This bundle is part of the CFEngine Standard Library:

bundle edit_line replace_or_add(pattern,line)

# Replace a pattern in a file with a single line.
# If the pattern is not found, add the line to the file.
# The pattern must match the whole line (it is automatically
# anchored to the start and end of the line) to avoid
# ambiguity.

{
  vars:
      "cline" string => canonify("$(line)");

  replace_patterns:
      "^(?!$(line)$)$(pattern)$"
        replace_with => value("$(line)"),
        classes => always("replace_done_$(cline)");

  insert_lines:
      "$(line)"
        ifvarclass => "replace_done_$(cline)";
}

To build unit tests for this bundle, we must first consider the different cases that we want to test:

1. $(pattern) appears in the text file, so it will be replaced by $(line)

2. $(pattern) does not appear in the text file, so $(line) will be added to the file.

Let’s now build a test file for this bundle. All tests are stored inside the test/ directory of the CFEngine source distribution, and are organized in directories named xx_category/yy_subcategory/nnn.cf. In our case, we will store this file as 16_stdlib/01_edit_line/001.cf.

Since it must be self-contained, it must include a body common control:

body common control
{
        inputs => { "../../default.cf.sub",   
                    "../../../../masterfiles/libraries/cfengine_stdlib.cf"   
                  };
        bundlesequence  => { default("$(this.promise_filename)") };   
        version => "1.0";
}

The testing framework includes a file called default.cf.sub, which contains a number of useful utility bundles and bodies for initializing common parameters and variables, for checking test results, etc. We will see a couple of these later, but I encourage you to read it in full to learn everything it contains.

For this particular test, we need to include cfengine_stdlib.cf, since the bundle we will test is defined in it. In general, you should not include cfengine_stdlib.cf unless it is necessary, to make each test case as stand-alone as possible. Even in this case, you will notice that replace_or_add() is the only bundle we use from the standard library. All other bundles and bodies are defined in the test file itself to reduce dependencies and ensure predictability of behavior.

The bundlesequence of all tests must call the default() bundle, defined in default.cf.sub, and which takes care of automatically calling the init(), test() and check() bundles. The variable $(this.promise_filename) contains the current filename, and must be passed as argument to default() for reporting purposes.

Now we have to define the init(), test() and check() bundles. First comes init():

bundle agent init
{
  vars:
      "states" slist => { "actual", "expected" };   

      "actual" string =>   
      "BEGIN
line1 plus more text
END";

      "expected" string =>
      "BEGIN
new line 1
END
new line 2";

  files:
      "$(G.testfile).$(states)"   
        create => "true",
        edit_line => init_insert("$(init.$(states))"),
        edit_defaults => init_empty;
}

bundle edit_line init_insert(str)   
{
  insert_lines:
      "$(str)";
}

body edit_defaults init_empty
{
        empty_file_before_editing => "true";
}

We define a list called @(states) with the names of the two variables we will define next. This will help in simplifying the policy by using list expansion to create the test files.

We define two variables called $(actual) and $(expected). The text in $(actual) will be used as the starting point for the test, and the test operations will be applied to it. The text in $(expected) is the expected end result of the test, and will be compared to the final result to check whether the test passed or failed.

Finally, we use a files: promise to write the values of $(actual) and $(expected) to two files, using the values in @(states) to loop over the variables. The variable $(G.testfile) is defined in default.cf.sub as a default base filename for use with the tests, along with some other variables:

  temp_declared::
    "testroot" string => getenv("TEMP", "65535");
    "testdir" string => concat(getenv("TEMP", "65535"), "/TEST.cfengine");
    "testfile" string => concat(getenv("TEMP", "65535"), "/TEST.cfengine");
  !temp_declared::
    "testroot" string => "/tmp";
    "testdir" string => "/tmp/TEST.cfengine";
    "testfile" string => "/tmp/TEST.cfengine";

As you can see, the default value of $(G.testfile) is "/tmp/TEST.cfengine", unless the TEMP environment variable is defined, in which case the value of TEMP is used instead of "/tmp/" to construct the filename. The value of $(G.testfile) is concatenated with the different values in @(states), so that the two files created in this example will be /tmp/TEST.cfengine.actual and /tmp/TEST.cfengine.expected, containing the values of $(actual) and $(expected), respectively.

The init_insert() bundle and the init_empty body are used by the files: promise that creates the files, and are declared here as well. Both of these components have equivalents in the standard library, but as we said before, we avoid using them to reduce external dependencies to a minimum.

After init() runs, the test inputs are ready: the edit operations will be applied on /tmp/TEST.cfengine.actual, and at the end its contents will be compared to /tmp/TEST.cfengine.expected to determine if the test was successful. We are now ready to run the actual test:

bundle agent test
{
  vars:
      "tpat1" string => "line1.*";    
      "tstr1" string => "new line 1";
      "tpat2" string => "line2.*";
      "tstr2" string => "new line 2";

  files:
      "$(G.testfile).actual"   
        create => "false",
        edit_line => replace_or_add("$(test.tpat1)", "$(test.tstr1)");
      "$(G.testfile).actual"
        create => "false",
        edit_line => replace_or_add("$(test.tpat2)", "$(test.tstr2)");
}

We first define some variables for use in the test. In this case, we are going to perform two calls to replace_or_add(), as dictated by the two cases we want to test. For this, we define two pairs of variables containing the pattern and line for each of the two cases.

Finally! We get to the meat of the test. In the files: section, we have two promises on the “actual” file. The first one replaces a line that already exists ("line1.*") with some new text. The second one is called with a pattern that does not appear in the original file ("line2.*"), so a new line should be added to the file.

That’s it. In most cases, the setup is the most complicated part of a test; the actual execution of the test is quite simple. After test() is done, we will have in /tmp/TEST.cfengine.actual the result of the operations on the original file, and in /tmp/TEST.cfengine.expected the expected result. It is time to verify the results of the test:

bundle agent check
{
  methods:
      "any" usebundle => default_check_diff("$(G.testfile).actual",
                                            "$(G.testfile).expected",
                                            "$(this.promise_filename)");
}

In this bundle, we simply have a call to the default_check_diff() bundle, which is also defined in default.cf.sub. It receives as arguments the two files to compare and the test filename. If the files are the same, it simply produces a “pass” result. If the files differ, it produces a “fail” result, plus copious log output (diff result, plus full content and hex dump of both files) to allow you to debug the problem.

When writing your own tests, remember that the output from the tests must contain "<testname> Pass" if the test passes, and "<testname> FAIL" if the test fails. Anything else (in fact, all output, including the Pass/Fail string) is written to the test.log file so you can determine exactly what happened.

Well, let’s now run the test! By default testall executes all the tests (which you should try sometime), but for now we will just run the new test we wrote:

# cd cfengine-3.4.0/tests
# ./testall 16_stdlib/01_edit_line/001.cf
======================================================================
Testsuite started at 2012-01-26 22:52:57
----------------------------------------------------------------------
Total tests: 1

-n ./16_stdlib/01_edit_line/001.cf
Pass

======================================================================
Testsuite finished at 2012-01-26 22:52:57 (0 seconds)

Passed tests: 1
Failed tests: 0
Failed to crash tests: 0
Skipped tests: 0

Now, let’s imagine we made a mistake in the replace_or_add() bundle, and inverted the logic on the test for the insert_lines: promise (note the added ! sign):

insert_lines:
  "$(line)"
    ifvarclass => "!replace_done_$(cline)";

This will produce the wrong output, and it will be caught by the test:

# ./testall 16_stdlib/01_edit_line/001.cf
======================================================================
Testsuite started at 2012-01-26 23:03:34
----------------------------------------------------------------------
Total tests: 1

-n ./16_stdlib/01_edit_line/001.cf
FAIL (UNEXPECTED FAILURE)

======================================================================
Testsuite finished at 2012-01-26 23:03:34 (0 seconds)

Passed tests: 0
Failed tests: 1
Failed to crash tests: 0
Skipped tests: 0

We can now look at test.log to see the details of the failure. Among a lot of other information, we can find the output of the diff command between the expected and actual files:

R: --- /Users/a10022/CFEngine/src/core/tests/acceptance/workdir/...
+++ /Users/a10022/CFEngine/src/core/tests/acceptance/workdir/...
@@ -1,4 +1,5 @@
 BEGIN
 new line 1
 END
+new line 1
 new line 2

This will give you already a hint of what the problem was. The line is being inserted even though it exists already in the file.

Of course, the tests can be arbitrarily complex, since they are full CFEngine bundles. In our case both test cases of the bundle are being tested together—if one of them fails, the whole test fails. Alternatively, we could have the script test the two possible cases separately: first on a file where a line is to be replaced, then on a file where a new line is to be added. This would make it easier to distinguish which part of the bundle is failing. The level of granularity used for the unit tests needs to be decided on a case by case basis depending on the bundle that is being tested and the complexity of its inputs.

In this example I have shown you a very simple example, easy to replicate and test on its own. What do you do for CFEngine bundles that effect more profound changes in a system, and whose input cannot be easily replicated? For example, how do you unit-test a bundle that manages system users? There is no single answer to this. For example, for manipulation of system files, you could set up copies of the real files under /tmp, and run the tests on them. For other system components, you may need to create custom programs to emulate their behavior. For testing package management, the test suite already includes the mock-package-manager program, which you can set to an arbitrary state (installed packages, for example) during the init() bundle, and query or modify during the test() bundle, without having to actually install or modify anything on the system.

A large number of tests are distributed with CFEngine. I encourage you to look at them to learn the capabilities of the framework, and to use it for testing all your bundles.

Using Vagrant with CFEngine

Vagrant is a tool that allows programmatic creation and interaction with virtual machine environments, and thus can be used to automate system creation, configuration, and testing. Vagrant allows you to create arbitrary VM setups (including multi-VM environments) in a consistent, repeatable manner. Together with CFEngine, Vagrant can be used both to configure VMs using CFEngine, and as a powerful development and testing tool for CFEngine policies.

This section shows you how to use Vagrant with CFEngine. The goal is to set up isolated environments—virtual machines—where you can install CFEngine and download policies in a consistent, easy-to-use and repeatable manner.

Setup is extremely simple. Your first step must be to download and install VirtualBox, which is the default virtualization environment used by Vagrant. Then you can download and install the latest Vagrant from http://downloads.vagrantup.com (the latest version as of this writing is 1.2.7). In both cases, make sure you download and install the latest available installer for your host operating system.

Note

Since version 1.1, Vagrant supports multiple virtualization providers, so you can use Vagrant to control VMs on systems other than VirtualBox, including AWS EC2 and VMware. This makes it possible to use Vagrant to deploy production environments in addition to development and testing.

I strongly advise you to read through the Vagrant Getting Started guide to fully understand how it works, but these are the main components that you need to understand for the purposes of our examples:

• A box is a base VM image that Vagrant uses to instantiate VMs. A box typically includes the base operating system install, plus some Vagrant-specific configuration that makes it possible for Vagrant to interact with it.

• The Vagrantfile is the configuration file for a Vagrant project. The Vagrantfile includes the entire specification of what VMs to create, how to configure them, what to install and run on them. It is the complete specification of the desired state of your Vagrant-created systems. A single Vagrantfile can contain multiple VM definitions, so you can use the file to describe an entire environment with a policy hub and multiple clients. This makes Vagrant a very powerful tool for testing CFEngine deployments and policies.

• A Provisioner is a plugin that tells Vagrant how to interact with the VM after it is created and booted, to perform arbitrary actions on it. For example, the Shell provisioner allows you to run shell commands on the VM, and the CFEngine provisioner allows you to install, configure and execute CFEngine on the VM.

Now, let us work through an example of using Vagrant to create a virtual machine managed by CFEngine:

1. First, download the precise32 box, which is one of the standard Vagrant boxes, based on Ubuntu 12.04.

   $ vagrant box add precise32 http://files.vagrantup.com/precise32.box
   Downloading or copying the box...
   Extracting box...
   Successfully added box 'precise32' with provider 'virtualbox'!

   This step may take a long time depending on the speed of your Internet connection, because it downloads the entire base VM image. Happily, it only needs to be done once for each box you want to have on your system. Once a box is downloaded, it can be reused as many times as you want to create multiple VMs based on it.

2. Now, you need to create a Vagrantfile that tells Vagrant how you want your VM to be configured. The easiest way to get started is to create a directory to hold the Vagrantfile (this is not required, but makes things easier to understand), and ask Vagrant to create a basic Vagrantfile for you.

   $ mkdir vagrant_test
   $ cd vagrant_test
   $ vagrant init precise32
   A `Vagrantfile` has been placed in this directory. You are now
   ready to `vagrant up` your first virtual environment! Please read
   the comments in the Vagrantfile as well as documentation on
   `vagrantup.com` for more information on using Vagrant.

   You can now examine the generated Vagrantfile. It’s quite long and with many comments that help you understand all the different options. Stripped of all comments, the basic Vagrantfile is a extremely simple snippet of Ruby:

   Vagrant.configure("2") do |config|
     config.vm.box = "precise32"
   end

   All it does is tell Vagrant that you want a VM based on the precise32 image, with no further customization.

3. We will now tell Vagrant to install CFEngine on our VM. For this we will use the CFEngine Provisioner included with Vagrant. You need to edit the Vagrantfile to include the CFEngine provisioner, and to tell Vagrant to use it on the VM.

   Vagrant.configure("2") do |config|
     config.vm.box = "precise32"
   
     config.vm.provision :cfengine do |cf|   
       cf.am_policy_hub = true   
     end
   end

   A lot is happening in this short configuration file, which modifies the behavior of the default VM that would be created by Vagrant:

   

   We indicate that the VM needs to be configured using the CFEngine provisioner. Within the :cfengine provider block, we can set the parameters for the provisioner, to specify its behavior.

   

   We set the am_policy_hub parameter to true, which indicates that this VM should be configured as a policy hub. By doing this, the CFEngine provisioner knows that it has to bootstrap CFEngine to the machine’s own IP address.

   This is all you need for a single-machine CFEngine setup. There are many other parameters you can provide both to Vagrant and to the CFEngine provisioner, but these will give you a working system.

4. We can now ask Vagrant to start the VM.

   $ vagrant up
   Bringing machine 'default' up with 'virtualbox' provider...
   [default] Importing base box 'precise32'...
   ...
   [default] Booting VM...
   [default] Waiting for VM to boot. This can take a few minutes.
   [default] VM booted and ready for use!
   ...
   [default] Running provisioner: cfengine...
   [default] Installing CFEngine onto machine...
   [default] Detected policy server IP address: 10.0.2.15...
   [default] Bootstrapping CFEngine with policy server: 10.0.2.15...
   ...
   2013-09-05T05:46:45+0000   notice: R: --> I'm a policy hub.

   I have omitted many lines from the output for the sake of space, but you can see that Vagrant creates and boots the VM, and then the CFEngine provisioner takes over, installs CFEngine, and bootstraps it according to the configuration we provided. You can view the status of the VM using the vagrant status command:

   $ vagrant status
   Current machine states:
   
   default                   running (virtualbox)
   
   The VM is running. To stop this VM, you can run `vagrant halt` to
   shut it down forcefully, or you can run `vagrant suspend` to simply
   suspend the virtual machine. In either case, to restart it again,
   simply run `vagrant up`.

5. We can now login to the VM and verify that CFEngine is indeed installed and running.

   $ vagrant ssh
   Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686)
   
    * Documentation:  https://help.ubuntu.com/
   Welcome to your Vagrant-built virtual machine.
   Last login: Thu Jun  7 00:54:26 2012 from 10.0.2.2
   vagrant@precise32:~$ ls /var/cfengine/
   bin                    inputs                 policy_server.dat
   cfagent.precise32.log  lastseen               ppkeys
   cf_classes.tcdb        lib                    promise_summary.log
   cf_classes.tcdb.lock   masterfiles            randseed
   cf-execd.pid           modules                reports
   cf_lastseen.tcdb       outputs                share
   cf_lastseen.tcdb.lock  performance.tcdb       state
   cf-serverd.pid         performance.tcdb.lock
   vagrant@precise32:~$ ps ax | grep cf-
    4452 ?        Ss     0:00 /var/cfengine/bin/cf-execd
    4458 ?        Ss     0:00 /var/cfengine/bin/cf-serverd

   In this basic setup, CFEngine is already running with its default policies. The CFEngine provisioner has many options, including some to specify the address of the policy server to which a client should be bootstrapped (cf.policy_server_address), files that should be copied to /var/cfengine/ in the VM (cf.files_path), etc. Additionally, Vagrant automatically makes the contents of the directory in which the Vagrantfile is stored available to the VM through a shared folder mounted under /vagrant, so you can store things there and access them from within the VM.

   vagrant@precise32:~$ ls /vagrant/
   Vagrantfile

There are many additional options that you can use in the Vagrantfile to provide very fine-grained control over the configuration of your VMs and of CFEngine. For example, you can specify the creation of both a hub and a client with the following Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "precise32"

  config.vm.define :hub do |hub|
    hub.vm.network :private_network, ip: "10.1.1.10"

    hub.vm.provision :cfengine do |cf|
      cf.am_policy_hub = true
      cf.policy_server_address = "10.1.1.10"
    end
  end

  config.vm.define :client do |client|
    client.vm.network :private_network, ip: "10.1.1.11"

    client.vm.provision :cfengine do |cf|
      cf.policy_server_address = "10.1.1.10"
    end
  end
end

This example defines two VMs, identified as hub and client, each one in its own config.vm.define block, and each including its own configuration for the CFEngine provisioner. Note that in this case we are specifying static IP addresses for both VMs, so that we know which address to use for bootstrapping to the policy server. When you run vagrant up with this configuration, Vagrant will automatically create both VMs and configure them appropriately.

Vagrant is an extremely powerful tool, and together with CFEngine makes it possible to create and configure virtual machines in a repeatable, consistent and controlled fashion. I encourage you to read through the Vagrant documentation to learn more about it.

Service Management Using CFEngine

One of the biggest concerns in managing computer infrastructure is making sure that all the necessary services are running on each system, and also that no unnecessary services are. Each service provides access to certain resources, most often to other machines in the network. For this reason, machines need to run services so they can talk to each other or perform their local functions. But each running service also introduces potential security risks, so running unnecessary services opens the possibility of additional security problems. CFEngine is perfectly able to help you in bringing services under control, making sure that all necessary services, and only those, are running on your infrastructure, in addition to making sure they are properly configured.

Basic Concepts of Service Management with CFEngine

In CFEngine, promises of type services: are the main mechanism to declare the desired state of a service. At its most basic, a services: promise looks like this:

  services:
      "ssh"     service_policy => "start";
      "postfix" service_policy => "stop";

By default, CFEngine will invoke a bundle called standard_services() to process the services: promise. This is a regular agent bundle that receives as arguments the name of the service and the value of the service_policy attribute, and can take any necessary actions to ensure that the service is in the desired state. The CFEngine standard library includes a standard_services() bundle that knows how to handle most common services for RedHat, Ubuntu, and SuSE Linux. The bundle has a fairly simple structure:

bundle agent standard_services(service,state)   
{
  vars:
      ...
    linux::

      "startcommand[cfengine3]" string => "/etc/init.d/cfengine3 start";   
      "stopcommand[cfengine3]"  string => "/etc/init.d/cfengine3 stop";
      "pattern[cfengine3]"      string => ".*cf-execd.*";

      Similar assignments for all supported services
      ...

  classes:

      "start" expression => strcmp("start","$(state)"),   
        comment => "Check if to start a service";
      "stop"  expression => strcmp("stop","$(state)"),
        comment => "Check if to stop a service";

  processes:

    start::
      "$(pattern[$(service)])"   
        comment => "Verify that the service appears in the process table",
        restart_class => "restart_$(service)";

    stop::
      "$(pattern[$(service)])"   
        comment => "Verify that the service does not appear in the process",
        process_stop => "$(stopcommand[$(service)])",
        signals => { "term", "kill"};

  commands:
      "$(startcommand[$(service)])"   
        comment => "Execute command to restart the $(service) service",
        ifvarclass => "restart_$(service)";
}

The bundle receives as arguments the name of the service, as provided in the promiser to the services: promise, and the desired state, as provided in the service_policy attribute. The valid values of this attribute are “start”, “stop”, “restart”, “reload”, and “disable”. The semantics of each one of these desired states are established by the behavior of the standard_services() bundle.

This triad of variable assignments is repeated for all supported services, defining the commands used to start and stop the service, and the regular expression that will be used to check whether the service is running.

In the classes: section of the bundle, we define classes according to the desired state for the service.

When the service needs to be started, we use a processes: promise to check whether the corresponding processes are already running. If a process with the corresponding pattern (as defined in the pattern[] array) is not found, the bundle will define the class specified by the restart_class attribute, which can be used later to trigger the appropriate command to start the service. Note that we are using “restart_$(service)" as the value of restart_class, which means that a different class will be created for every service that needs to be started. For example, when the value of $(service) is “cfengine3” and the CFEngine processes are not found, the class restart_cfengine3 will be defined.

When the service needs to be stopped, we also use a processes: promise, but this time with the process_stop and signals attributes. If the processes are running, CFEngine will first execute the command indicated in process_stop, which is meant to attempt a graceful shutdown of the process by running the “stop” command indicated for that particular service. If after this the processes are still running, CFEngine will send them the signals indicated.

If the service needs to be started and the corresponding class has been set (which means the processes are not running), a commands: promise is used to run the appropriate service-start command.

Note

The standard_services() bundle in the standard library only handles the cases for “stop” and “start”, but the service_policy attribute can also take other values, including “disable” and “restart”. Can you extend the bundle to properly handle these additional values?

If you improve the standard_services() bundle in this way, or to handle additional services or operating systems, I would encourage you to contribute your changes back to the community, by submitting a pull request against the CFEngine standard library in the GitHub repository at https://github.com/cfengine/core/.

Defining Your Own Service Management Mechanisms

As you can see from our previous example, service management is ultimately performed by a generic agent bundle, which means that it can perform any arbitrary actions needed to bring the corresponding services into compliance. Of course, you can define and use your own service-management bundles if standard_services() does not fit your needs.

As an example, let us develop a policy to manage the SSH service on Mac OS X. In this system, the canonical way to enable or disable SSH is to enable or disable “Remote login” in the System Preferences. This setting can be modified from the command line using the /usr/sbin/systemsetup command:

# /usr/sbin/systemsetup -f -setremotelogin on

Also, even when SSH is enabled, the sshd daemon is not always running, because the connections are handled by the launchd daemon, which is a generic service handler that listens on the appropriate ports and starts the appropriate processes, much like inetd-style daemons do on other Unix systems. So instead of looking at the process table, we need to query the “Remote login” setting using the same systemsetup command:

# /usr/sbin/systemsetup -getremotelogin
Remote Login: On

Let us write a service handler for this. The first step is to write a new service_method body (we will call it my_services, which defines the bundle that will be called to handle the services:

body service_method my_services
{
    darwin::   
        service_bundle => osx_service("$(this.promiser)",
                                      "$(this.service_policy)");
      # If not on darwin, the default standard_services bundle will be called
}

The service_bundle attribute must call an agent bundle that will take care of performing the necessary operations on services. The variable $(this.promiser) is automatically populated with the value of the promiser from which this body is invoked ("ssh” in this example), and $(this.service_policy) contains the value given for the service_policy attribute ("start” in this example). Note that the assignment to service_bundle is conditioned to the class darwin, which will be defined only when running on a Darwin (Mac OS X) system. If this body is invoked in other platforms, the service_bundle attribute will keep its default value and call standard_services() from the standard library. So we can safely use service_method => my_services on any platform, and CFEngine will do the right thing.

The osx_service() bundle is the one that actually does the work of starting/stopping the services on OS X. For now we will only show the code for managing SSH, but you can extend the technique to any other services.

bundle agent osx_service(service, state)
{
  vars:   
      "startcommand[ssh]"
        string => "/usr/sbin/systemsetup -f -setremotelogin on";
      "stopcommand[ssh]"
        string => "/usr/sbin/systemsetup -f -setremotelogin off";
      # An exit status of zero means the service is running
      "checkcommand[ssh]"
        string => "/usr/sbin/systemsetup -getremotelogin | /usr/bin/grep -q On",
        comment => "Command to check if ssh is running.";
  classes:
      "$(service)_is_running"   
        expression => returnszero("$(checkcommand[$(service)])", "useshell"),
        comment => "Indicate if $(service) is running.";
      "start"   
        expression => strcmp("start","$(state)"),
        comment => "Check if to start a service";
      "stop"
        expression => strcmp("stop","$(state)"),
        comment => "Check if to stop a service";
  commands:
      "$(stopcommand[$(service)])"   
        comment => "Execute command to stop $(service)",
        ifvarclass => "stop.$(service)_is_running";
      "$(startcommand[$(service)])"
        comment => "Execute command to start $(service)",
        ifvarclass => "start.!$(service)_is_running";
  reports:
    verbose::   
      "Stopping $(service) with $(stopcommand[$(service)])."
        ifvarclass => "stop.$(service)_is_running";
      "Starting $(service) with $(startcommand[$(service)])"
        ifvarclass => "start.!$(service)_is_running";
}

Just as in standard_services(), we define the commands for starting and stopping the service, in an array indexed by service name. However, instead of defining a regular expression for matching the appropriate processes, we define a command to check whether the service is running. If the command exits with a status code of 0, we assume the service is running correctly. In this particular example, we use systemsetup and parse its output to check whether the service is on or off.

We run the “check service” command just defined and assign a class based on its exit code. For example, if SSH is running, the ssh_is_running class will be defined. With this promise we determine the current state of the service.

We also define, as in standard_services(), classes named start or stop, according to the desired state for the service.

We execute the appropriate commands, for starting or stopping the service, depending both on its current state (ssh_is_running) and on the desired state (start, stop), so as not to do anything unless it is necessary.

Finally, if the verbose class is set, we print some informative messages.

Once the infrastructure has been laid, we can simply invoke it through a services: promise that makes reference to our new service_method body, and defines the desired state for the service:

bundle agent test
{
  services:
      "ssh"
        service_policy => "start",
        service_method => my_services;
}

Remember that when running this bundle on an OS X machine, the osx_service() bundle will be used, which uses the correct OS X-specific commands to verify and manage the service. But when running on a different platform, the default standard_services() bundle from the CFEngine Standard Library will transparently be used. This is an example of how CFEngine allows you to express the desired state of the system at a high level, without worrying about the lower-level implementation details, unless you need to look at them.

We can see this in action by executing it manually:

# systemsetup -getremotelogin
Remote Login: On
# systemsetup -f -setremotelogin off
# systemsetup -getremotelogin
Remote Login: Off
# cf-agent -Kf ./ssh_osx.cf -Dverbose
2013-10-14T00:59:09-0500   notice:
  R: Starting ssh with /usr/sbin/systemsetup -f -setremotelogin on
# systemsetup -getremotelogin
Remote Login: On

As you can see, CFEngine provides extensive and flexible service management capabilities, that allow you to use arbitrary CFEngine policy to check and manage services. And for the most common cases, you can simply use the default service definitions included in the standard library.

Where to from Here?

In the course of this book we have discussed many concepts and techniques for effectively using CFEngine to manage computer systems. Whether you are managing your own machine, a network of five machines, or a Google-sized datacenter, the same basic principles apply. Still, we have only scratched the surface, even with the more advanced topics we explored in this last chapter. CFEngine is a rich and complex tool, and it has capabilities way beyond what we have covered here. I expect to have given you a running start, but it is up to you to continue experimenting, discovering and evolving. I encourage you to read CFEngine’s abundant documentation to learn many of the more advanced capabilities. I also encourage you to take advantage of the friendly and helpful CFEngine community, in which both users and developers participate to provide a helping hand and to discuss the future of CFEngine.

There are many aspects of CFEngine functionality that we have not touched in this book, including the capabilities of the knowledge management tools included in CFEngine, database integration, and advanced Windows configuration facilities. Many of these advanced capabilities are limited to (or better developed in) the commercial editions of CFEngine. Many others, however, are available in the Community edition. Practice makes perfect, and there is no single way of achieving many tasks in CFEngine. As you learn more, you will develop better and more efficient ways of achieving the same tasks. As you become familiar with the syntax and the constructs available in the language, you will discover more advanced ways of using the tool.

A word of caution: CFEngine is not suited for every single system-management application. While it is great for many configuration tasks, it may lack the power provided by specialized tools for some particular areas. For example, you can use CFEngine to configure a backup utility, but the utility itself is probably much more suited for actually scheduling and performing the backups. Other tasks for which specialized tools may be better suited include network monitoring, intrusion detection, inventory, and user management. The mark of a good sysadmin is the use of the right tools for the right task. CFEngine is now one more tool in your arsenal, a very flexible tool, and as such is able to integrate properly in many different situations and environments. Hopefully, after reading this book, you will be able to see ways in which CFEngine can integrate into your existing environment and make it better, possibly aiding also the configuration and management of some of the specialized tools that you use.

CFEngine is constantly evolving. You can watch this evolution in action, and even contribute to it, by participating in the CFEngine community. Whether you post in the forums, write in your blog, teach CFEngine to your coworkers, or simply spread the word about it, you will be participating in the development of one of the oldest and longest-living configuration management tools in existence. With CFEngine 3, the foundations are laid for a much smoother evolution path, grounded in strong computer science theory and with the mechanisms in place to develop new features with much less friction than in the past. In this situation, the CFEngine developers are now, more than ever before, open to comments, questions and suggestions by users that help them better understand the needs, and if new worthwhile features are suggested, or better ways of achieving certain tasks are proposed, there is a good chance that they will be implemented.

More than anything else, I encourage you to experiment and have fun. There are few things as satisfying to a system administrator as having systems running smoothly and with as little human intervention as possible. Play with CFEngine to explore its capabilities, try new things, break things (in a controlled manner whenever possible! We don’t want you to get fired) and learn new tricks. The basic principle of CFEngine is convergent configuration, and over time your systems will converge to a stable situation. As your CFEngine policies evolve and stabilize, you will be able to free your mind from mundane day-to-day details. And this is the whole purpose of CFEngine: to elevate your thinking about your systems—to enable you to manage them through expressions of intent. CFEngine will implement the details and give you knowledge that you can use to further improve your infrastructure. CFEngine allows you to become much more than a sysadmin—it makes you an infrastructure engineer.

Enjoy the ride.