adrift on a cosmic ocean

Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We have lived in Berlin since 2009, have two kids, and have far too little time to really justify having a blog.

How we use cucumber-puppet

Posted by Oliver on the 16th of May, 2011 in category Tech
Tagged with: bddcucumbercucumber-puppetpuppet

I got a question recently by email in followup to my presentation at Puppet Camp in Amsterdam about how we use cucumber-puppet. I touched on the subject only briefly in my talk but what I did say is that it revolutionised my approach to Puppet in general. Don't get too high an opinion of the tool from that statement! Behaviour-driven development in general was a new thing to me and did change my ways of thinking, but my opinions of it in conjunction with Puppet have changed over the months slightly.

Before I go into too much depth, let's take a look at the tool and how it is used. To be fair, cucumber-puppet is a good tool (as is cucumber itself and cucumber-puppet's cousin, cucumber-nagios). Typically you'll start off by running cucumber-puppet-gen world in your Puppet code repository and let it generate the infrastructure necessary to start writing your own tests. Basically they are grouped into three main categories:

  • modules - where you actually write your high-level language tests
  • steps - the Ruby-language breakdowns to help cucumber-puppet turn natural-language requests into things to test in the Puppet catalog
  • support - cucumber-puppet settings and globals

As you might have noticed, you are actually directing cucumber-puppet with natural-language, which gets translated into native Ruby and applied as tests for various content in the Puppet catalog. It's actually not that much magic. Let's look at an example feature:

Feature: Base repositories
  In order to have a system that can install packages
  As a sysadmin
  I want all of the CentOS repositories to be available

  Scenario: CentOS yum repositories
    Given a node of class "yum::base"
   When I compile the catalog
    Then there should be a yum repository "Base"
    And there should be a yum repository "Updates"
    And there should be a yum repository "Extras"

Pretty cool huh? The most important aspect of this is that it is readable by humans. As you go on though, you'll realise it is somewhat verbose and you are prone to much repetition. Anyway, let's take a look at some of the steps that make this work. You'll notice we said a node of class "yum::base". It's not exactly a real node, we are just directing cucumber-puppet to compile a catalog that has just a single class in it - yum::base and treat it as if that is the entire node.

Given /^a node of class "([^"]*)"$/ do |klass|
  @klass = klass

Then /^there should be a yum repository "([^"]*)"$/ do |name|
  steps %Q{
    Then there should be a resource "Yum::Repo[#{name}]"
    And it should be "enabled"

Then /^it should be "(enabled|disabled)"$/ do |bool|
  if bool == "enabled"
    fail unless @resource["enabled"] == "1"
    fail unless @resource["enabled"] == "0"

Then /^there should be a resource "([^"]*)"$/ do |res|
  @resource = resource(res)
  fail "Resource #{res} was not defined" unless @resource

Those are all the steps necessary to make the previous feature work. They should be fairly clear even if you have no idea about Ruby or cucumber-puppet. Some important items to note:

  • Steps can call other steps, so that you have a useful abstraction mechanism between many different things that test resources in similar ways - e.g. for presence in the catalog.
  • Yes, not all the feature text is actually parsed. A lot of it is human-understandable, but computationally-useless fluff.
  • We have replaced the built-in Yum provider with our own (that is just a managed directory and template files in a defined type) - as the Yum provider famously doesn't support purging.
  • ...@resource["enabled"] == "1" looks wrong, but that's how Yum repositories represent settings so we mirror that here, even if it is not strictly boolean.

You do get a lot of steps for free if you run cucumber-puppet-gen world, like the first and last I've quoted, so you don't have to come up with it all by yourself. This is the general style of testing with cucumber-puppet at least up to version 0.0.6 - very verbose, can duplicate your Puppet code in an almost 1:1 ratio (or even beyond it) but still a very useful tool for refactors etc. Starting with 0.1.0 it was possible to create a catalog policy - think of it as a bit closer to reality as instead of testing arbitrary aspects of your Puppet code, or fake machine definitions (or even real ones, with faked facts) you can now test some real facts from a real machine and be sure that the machine's catalog compiles.

We're not doing this just yet (I've been busy working on other aspects of our Puppet systems, but in theory it is not much work to get going. You do, however need some mechanism for transporting your real machine YAML files from your Puppetmasters to your testing environment for runs through cucumber-puppet. While this is definitely a step forward, it also gets into the territory that perhaps more people are considering - pre-compiling catalogs on your real Puppetmasters and checking for success/failure there. It also gives you the ability to check for differences between catalogs on the same machine when inputs change (i.e. your configuration data changes) - Puppet Faces will give you the ability to do this quite easily.

Another couple of cool things about cucumber-puppet before I sign off. Due to it being based on cucumber, it has the capacity for generating output in many different useful formats. For example, you can output a JUnit report (in XML format). Jenkins supports JUnit reports natively, so you can run your cucumber-puppet tests in a Jenkins job and have the JUnit test results integrated into the build result and history. Very cool.

Finally, since you are testing catalogs with cucumber-puppet you can make tests for just about anything in those catalogs. For example, if you are generating some application configuration using an ERB template and want to check that certain values have been correctly substituted, you can just test what has been generated as the file content:

Scenario: Proxy host and port have sensible defaults
  Given a node of class "mymodule::myapp"
  And we have loaded "test" settings
  And we have unset the fact "proxy_host"
  And we have unset the fact "proxy_port"
  When I compile the catalog
  Then there should be a file "/etc/myapp/"
  And the file should contain "proxy.port=-1"
  And the file should contain /$/


Then /^the file should contain "(.*)"$/ do |text|
  fail "File parameter 'content' was not specified" if @resource["content"].nil?
  fail "Text content [#{text}] was not found" unless @resource["content"].include?(text)

Then /^the file should contain \/([^"].*)\/$/ do |regex|
  fail "File parameter 'content' was not specified" if @resource["content"].nil?
  fail "Text regex [/#{regex}/] did not match" unless @resource["content"] =~ /#{regex}/

The complexity and coverage of your tests are only limited by your inquisitivity with respect to the Puppet catalog, and your Ruby sklils (both of which are easily developed).

© 2010-2018 Oliver Hookins and Angela Collins