Hiera Data Bindings and Puppet

I first heard about Hiera data bindings in a blog entitled "Say Hello to Puppet 3" by a puppetlabs employee, however I was not able to find any documentation on how to use it or even what it was exactly, The best explanation I could find was puppetconf 2012 video by Kelsey Hightower, skip to the last 2 minutes and you will get a whirlwind tour.

In a nutshell Hiera data bindings enables Puppet to automatically look up parameters in Hiera. Before I learnt about it I had a whole lot of code that looked like this:

 $relayhost        = hiera('relayhost')  
 $mynetworks       = hiera('mynetworks')  
 $inet_interfaces  = hiera('inet_interfaces')  

with a yaml file that looked like this:

 relayhost: mail.example.co.za
 inet_interfaces: localhost
 mynetworks: 192.168.1.0/24 

With Hiera data bindings you can get rid of all this random hiera('param_to_lookup') in your code and hiera will look up the values automatically. The first thing you will need to do is use parametrized classes, which you should be using already. My environment has a lot of legacy modules which had/have to be updated to use parametrized classes, we typically declared most of our variables in a params class.

A parametrized class has all your variables set at the top of the class. This makes it easily configurable and more portable. An example of a parametrized class:

 class postfix::client (  
  $relayhost,  
  $inet_interfaces = 'localhost',  
  $mynetworks,     
 ...  
 ) {  

Here we define our variables with an optional default value. We can then use this class by overriding the defaults when we call it (This also overrides Hiera):

  class { postfix::client:  
   relayhost => "test",  
   inet_interfaces => "192.168.1.5",  
   mynetworks => "192.168.1.0/24",  
  }  

The problem with this method is if you have many hosts with custom setups, you have all these scattered class definitions all over the place, instead of just declaring it in once. If you are using a parametrized class, puppet can now automagically look up the values from Hiera. You need to structure your Hiera .yaml files to have have the following structure "module::class::variable".

 postfix::client::relayhost: mail.example.co.za  
 postfix::client::inet_interfaces: localhost  
 postfix::client::mynetworks: undef  

Now you can declare your class in your base class as follows:

 class { postfix::client: }  

when puppet runs it will look up the variables in Hiera overriding the modules defaults.

Our new puppet architecture on puppet 3.x

I recently redesigned our puppet infrastructure at the office. Not much thought went into the original overall architecture as not everyone was really sold on the idea of puppet and I don’t like to invest time in something that is technically a proof of concept.

The initial implementation had one puppet master with multiple environments (testing, development, production and individual user environments). The puppet master had been built from gems on Ubuntu 10.04 with a custom ruby interpreter. Puppet dashboard and foreman where hosted on the same machine with MySQL back-ends  The primary git repository was also stored on the puppet master. It handled 500 nodes reasonably well with 8 cores and 6GB of ram.

The primary issue with only one puppet master server for us is patching. Whilst I have never experienced a regression that affected production clients when upgrading the master, in theory it is possible that I have done something non standard in one of the manifests that could affect them. I suppose the removal of dynamic scoping in puppet 3 is a good example where things could have gone wrong.

Our new architecture looks like this:

The puppet infrastructure has been separated into several virtual machines. This should be a lot easier to maintain and scale outwards. All the puppet components have been built using the puppetlabs repositories. The development server was built using the original puppet server and all clients where migrated to connect to it. The new production server was then built using the new development server (puppet servers building puppet servers, awesome!) and the production hosts where then pointed to the new production server using the development server as the CA. If the CA server goes down we cannot sign new certificates but the puppet clients to continue to retrieve catalogs (I am not sure for how long). I am somewhat satisfied with the redundancy. I would really like to make the CA highly available but I have not figured out how yet.

The move from puppet 2.7.x to 3.1.0 has been well worth it. It was smooth migration with very few issues. Unfortunately we have not been able to use ruby 1.9.x on our master servers due the dependency issues of the passenger package on Ubuntu 12.04's ruby 1.8.7 packages. We had some very minor issues with dynamic scoping and some modules had to be updated. From a performance perspective there has been a massive improvement. I would say the average puppet runtime has improved from between 50% and 75%.

Whilst these features not new to puppet 3.x, I have only started to use them now and they are must haves for any puppet configuration.

Hiera:

It has been built into the base puppet package for puppet 3.x. Hiera has allowed us to build more portable modules and cater for all the special cases where we have nodes with unique settings. We had many modules with a params.pp file with many variables set depending on hostnames, locations, distributions and just about every fact you can think of. One of the downside of working in the financial industry is we have lots of legacy systems and applications that will never be replaced and Hiera allows us to cater for these unique snow flakes.

Classes can be assigned to nodes using Hiera, we plan to use Hiera as or node terminus and move all of our class assignments from the default site.pp. 

The Hiera documentation is rather poor, specifically a feature which is new in puppet 3, called Hiera data bindings.

Puppetdb:

We have yet to reap any massive benefit from puppetdb right now; currently no applications seem to use it. The puppet dashboard can only use it for the inventory service at the moment. Reports can be stored in puppetdb (currently experimental), I am guessing the dashboard will fully integrate with it soon. Sadly there seems to be no intention for foreman to use puppetdb. All that being said, I am very excited about the potential of puppetdb and I have been playing around with the API for queries.

I noticed Africa was sorely missing from the puppetconf 2012 talk on puppetdb. So here is a screen shot of a puppetdb dashboard all the way from Johannesburg, South Africa.

Moved my blog

I have moved my blog to blogger.com. My old blog is available here: http://oldblog.krakowitzer.com
I have ported my last 3 articles from my old blog, not sure if I am going to bother with the rest. Feels good to not be using google app engine any more.

Kanban in operations

I first heard about Kanban when listening to a devops cafe podcast, it sounded pretty interesting so I decided to do some more research on Kanban.

It did not take me very long to realise that Kanban was a perfect fit for my sysadmin team. In a nutshell Kanban is a way to visualise and limit your work in progress (WIP). The design of the kanban board I decided to use was based on a blog by systemsoup.org and it looks like this:



The board is divided into 3 buckets.

• The main bucket is where my team spends most of its time with project work. The person on standby always has 1 of his WIP slots filled with "Support". A maximum if 2 WIP slots per person is allowed
• The second bucket is called support, I found we primarily use this bucket by moving cards from our WIP to the awaiting customer and awaiting dev/storage/oracle/db2/whatever teams. If feedback is positive the card gets moved into done. If feedback is negative the card gets moved back into WIP.
• The third bucket is for emergencies. This is typically some kind of severity 1/2/3 outage that is impacting the business. If we have a card in this bucket all work in progress is blocked until the emergency is resolved.

The cards

• Blue - 1 days work, High priority card.
• Green - 1 days work, Low priority card.
• Purple - 5 days work, High priority card.
• Pink - 5 days work, Low priority card.
• White - 2 hour task.
• Yellow - Really large chunk of work, typically broken down into smaller cards when we are ready to move it into the ready lane.

The minimum detail on the card should be:

• Detail of the work required.
• Date moved from READY lane to WIP lane.
• Date moved from WIP lane to DONE lane.

Our board on its first day:



Every morning we have a quick 5 minute meeting where we discuss our current work in progress and if we are on track to complete the cards we own within the estimated time. We decide if any of the cards in the BACKLOG should be moved into the READY lane. If a yellow card is ready to be moved to the READY lane, we break it down into smaller cards and put those cards into the READY lane.

New cards can be added into the BACKLOG at any time and will be discussed in the next morning meeting.  It's pretty important to meet with all the key stake holders involved on a regular bases. I meet once per month with my boss to discuss what work is in our backlog and if there are any new items he feels we should be working on. I do the same for some of our customers (lead developers, project managers etc.)

Our board after 30+ days:



I am pretty happy with Kanban from a management perspective. It is very easy to see who is working on what and by when the team estimated that particular task should be completed. It is easy for everyone who walks into our area to see who is busy on what. It helps the team to achieve forward momentum. It helps team members who struggle to complete tasks by preventing them from taking on new tasks until there current tasks are complete. If you are interested in Kanban I can recommend the following:

book - Kanban: Successful Evolutionary Change for Your Technology Business
blog - Our first kanban board for it operations
video - Kanban in operations by Mattias Skarin.
web based Kanban board - AgileZen

Behavior Driven Infrastructure

I wrote in a previous blog about how we decide when a system is ready for production and how I would be looking into an automated process (Behavior Driven Infrastructure) using cucumber-nagios. I finally have time to blog a bit about it.

In a nutshell I hacked up a script to generate the natural language (gherkin) that is used by cucumber-nagios. When I run the script it will ask some of questions on how I expect the server to look and then it generates all the features and scenarios that I have selected for that server. It then runs cucumber-nagios to make sure the server passes all the “tests”, thereafter we can move the server into production or hand the server over to an application team with 100% confidence the system has been built to their specification.

I found the cucumber-nagios learning curve to be pretty easy and it did not take long before I could write decent tests against my servers. My intention now is to continue to add more and more tests to our infrastructure. There is no such thing as to many tests :)

Cucumber-nagios is officially part of my sysadmin toolbox :)