Archive for January, 2011

Proposal for talk at DevOpsDays Boston!

by Oliver on Tuesday, January 25th, 2011.

My proposal is up on the DevOpsDays website, please comment and vote on it!

Tags: , , , , ,

Tuesday, January 25th, 2011 Tech No Comments

Puppet class API verification and testing

by Oliver on Thursday, January 20th, 2011.

I’ve been working recently on building up a more reliable, predictable Puppet infrastructure. I’m not a developer at heart, but I’ve been trying very hard to apply software development principles to what I do, since generally configuration management and software deployment is becoming more and more just another software problem. One of the end goals of the system I am building is to have a pretty GUI (isn’t that always the end goal?) that will let you know before you deploy something what kind of information it requires.

Applying the concept of encapsulation to our Puppet code, we are trying to keep all the details of each module relatively hidden from the outside so that you don’t need to look at the Puppet code to use it – not exactly revolutionary stuff. All that should matter is the interface to the code – and we’re going down the path of Puppet 2.6 parametrised classes here. So, you have a bunch of parametrised classes and you want to see what the interface is, or in my example, have some GUI figure out what variables you need to pass to the class in order to use it. I know many software tools already do this, but I have no idea exactly how they do it (although it is probably just a variation on the standard code parsing phase).

Armed with this information we can also do verification on the interface to a module, and see when it changes. Dan Bode and Luke Kanies of Puppet Labs helped me out with the following code.

Let’s set up a very basic class in a file:

# test API - test.pp
class test ($foobar) {
    file { '/tmp/foo':
        ensure => present,
        content => $foobar;
    }
}

Now we call Puppet directly from Ruby on this file, and render the class as PSON (although it could just as easily be YAML, or probably other formats):

#!/usr/bin/env ruby
require 'rubygems'
require 'puppet'
Puppet[:manifest]='/home/ohookins/test.pp'
puts Puppet::Resource::Type.find('test').render(:pson)

And here’s the output:

{"name":"test","arguments":{"foobar":null},"line":2,"doc":"test API - test.pp\n","code":"[File[\"/tmp/foo\"]]","type":"hostclass","file":"/home/ohookins/test.pp"}

This is ridiculously easy to use later to pretty-print our class interface or pre-load HTML forms etc etc. If we change that variable to have a pre-set default value, you can easily see the change in output PSON:

{"name":"test","arguments":{"foobar":"\"mydefault\""},"line":2,"doc":"test API - test.pp\n","code":"[File[\"/tmp/foo\"]]","type":"hostclass","file":"/home/ohookins/test.pp"}

What is even cooler is that comments preceding the class definition are parsed and included as a doc pair in the hash, which could have any number of cool uses. I assume this code would work equally well for printing interesting information about defined types or unparametrised classes (although in that case I’m not sure there would be as much value). Let me know if this helps you at all, or if you find more cool uses for it!

Tags: , , , ,

Thursday, January 20th, 2011 Tech No Comments

MySQL replication and the idempotence myth

by Oliver on Tuesday, January 11th, 2011.

I had a fairly frustrating time today trying to explain to someone that indeed the order of actions matters very much when trying to recover failed replication on a MySQL server. We had a good dump from the master taken with --single-transaction --master-data=2 (yes, all InnoDB tables here), and had imported it successfully. However, at the critical moment of reading the binary log name and position from the comments at the start of the dump file, the operator in charge had mistakenly entered an old CHANGE MASTER statement from the history and started the replication at an earlier position than it should have been.

The completely expected happened, and a short while later replication failed again on a duplicate insertion error. At this point there was some disagreement between myself and the operators nursing the system back to health. Their position was that we can simply start the replication from the correct point and let it continue safely from there. My position was that once you have restarted replication from an incorrect point in the binary logs, you can no longer be assured that your data is consistent with what is on the master.

Consider if you will a brief timeline of events:

Timeline 1 Timeline 2
insert into foo values 1,2 delete from foo where bar > 2
insert into foo values 3,4 insert into foo values 1,2
delete from foo where bar > 2 insert into foo values 3,4

OK, so that’s fairly nasty PseudoSQL but you get the idea. Regardless of which (contrived) timeline you judge to be the “correct” one, reordering them will lead to a different set of data at the end. In Timeline 1 you will have only values 1 and 2 in your table, whereas in Timeline 2 you will have 1 through to 4. Relying on the fact that you can skip past duplicate insertion errors will not save you from the inevitable updates or deletes that can also change the dataset in non-idempotent ways.

“But that’s a completely contrived example” you might very well say. “It’s unlikely something like that would happen on the real dataset” may be another phrase used. My answer to that is – as a sysadmin do you know 100% how the application works? Do you know every SQL query that will occur?

Consider Marty going back to 1955 in Back to the Future. The past is changed, and he has to spend a bunch of effort fixing things up so that he is eventually born, and still alive at least until 1985. He gets back to 1985 and a lot of things are not as he remembers. In order to ensure that it was just as he left it, he would have to ensure that every event from 1955 to 1985 happened just as it did in his own timeline. Similarly, to have any surety that your dataset will end up just “similar” to how it should be, you would have to trace through the entire binary log, fixing problems as you go (and even then the outcome would be doubtful).

There are far better uses of our time than making mistakes and pretending they don’t exist.

Tags: , ,

Tuesday, January 11th, 2011 Tech No Comments