Testing puppet manifests part 1 – Local Compilation

Testing puppet manifests

The pipeline approach we use to move our infrastructure changes from one environment to the next gives us the advantage of having some visibility into what will happen in an upstream environment. Still, it would be nice to be able to get some quick feedback on potential issues with out puppet codebase before we even apply the changes. We have come up with 2 mechanisms to do this that provide us with very fast feedback and some assurance that our changes won’t immediately break the first upstream environment. The first, covered in this blog post, is the local node compilation.

Node manifest compilation

In the same way that a developer compiles their code locally prior to checking in, the node manifest compilation step is a verification step that runs through each and every node we have defined in our puppet manifests and compiles the puppet code. This catches errors such as:

  • Syntax errors
  • Missing resource errors – i.e. a file source is defined but not checked in
  • Missing variable errors for templates

The code to do this is pretty simple:

  1. Configure Puppet with the manifest file location (nodes.pp) and the module directory path
  2. Use the puppet parser to evaluate the manifest file and find all available nodes for compilation
  3. For each node found, create a Puppet node object and then call compile on it
  4. Compile all nodes, fail only at end of run if any nodes fail to compile and provide all failed nodes in output
require 'rubygems'
require 'puppet'
require 'colored'
require 'rake/clean'

desc "verifies correctness of node syntax"
task :verify_nodes, [:manifest_path, :module_path, :nodename_filter] do |task, args|
  fail "manifest_path must be specified" unless args[:manifest_path]
  fail "module_path must be specified" unless args[:module_path]

  setup_puppet args[:manifest_path], args[:module_path]
  nodes = collect_puppet_nodes args[:nodename_filter]
  failed_nodes = {}
  puts "Found: #{nodes.length} nodes to evaluate".cyan
  nodes.each do |nodename|
    print "Verifying node #{nodename}: ".cyan
      puts "[ok]".green
    rescue => error
      puts "[FAILED] - #{error.message}".red
      failed_nodes[nodename] = error.message
  puts "The following nodes failed to compile => #{print_hash failed_nodes}".red unless failed_nodes.empty?
  raise "[Compilation Failure] at least one node failed to compile" unless failed_nodes.empty?

def print_hash nodes
  nodes.inject("\n") { |printed_hash, (key,value)| printed_hash << "\t #{key} => #{value} \n" }

def compile_catalog(nodename)
  node = Puppet::Node.new(nodename)
  node.merge('architecture' => 'x86_64',
             'ipaddress' => '',
             'hostname' => nodename,
             'fqdn' => "#{nodename}.localdomain",
             'operatingsystem' => 'redhat',
             'local_run' => 'true',
             'disable_asserts' => 'true')

def collect_puppet_nodes(filter = ".*")
  parser = Puppet::Parser::Parser.new("environment")
  nodes = parser.environment.known_resource_types.nodes.keys
  nodes.select { |node| node =~ /#{filter}/ }

def setup_puppet manifest_path, module_path
  Puppet.settings.handlearg("--config", ".")
  Puppet.settings.handlearg("--manifest", manifest_path)
  Puppet.settings.handlearg("--modulepath", module_path)

Code available here: https://github.com/oldNoakes/puppetTesting

Note that in our production code, we break up our nodes into subsets and then fork a process for each of these to compile in. Currently we run 20 parallel processes for over 400 nodes – typically takes about 45 seconds on a fast machine (i.e. our build server) and up to 120 seconds on a slower one (i.e. the worst developer station that we have).