Archive for October, 2010

Puppet – create a resource within a function using hash2resource

by Oliver on Tuesday, October 26th, 2010.

I searched everywhere for an example of this but couldn’t find one, so I stumbled through and got it working myself and am documenting it here in case it ever shows up on Google and helps someone. The premise is: you have some crazy variables that you want to turn into resources of some kind, but there’s no way to do it using the declarative methods within Puppet. Most notably lacking is usually the for loop, and that’s what I’ll be hacking around to provide.

This slightly contrived example takes a bunch of variables which represent things we might to give to libvirt in order to create a storage pool. We have a fixed number of fields, but no limit to how many entities are created (so the arrays can be as long as we want). Perfect for the missing for loop to create:

modules/virtualization/plugins/puppet/parser/functions/storagepool_construct.rb:
module Puppet::Parser::Functions
    newfunction(:storagepool_construct, :type => :statement) do |args|
        # Hard-coded number of parameters
        args.length == 5 or raise Puppet::Errors.new("storagepool_construct takes 5 arguments!")

        # Process the parameters in this order:
        #  - storagepool_src_name
        #  - storagepool_type
        #  - storagepool_src_host
        #  - storagepool_src_path
        #  - storagepool_src_target
        if args[0].class == Array then
            (0..(args[0].length - 1)).each do |i|
                catalog.add_resource Puppet::Type.type(:file).hash2resource({:title => "/tmp/#{args[0][i]}", :content => "#{args[1][i]} #{args[2][i]} #{args[3][i]} #{args[4][i]}"})
            end
        elsif args[0].class == String then
            catalog.add_resource Puppet::Type.type(:file).hash2resource({:title => "/tmp/#{args[0]}", :content => "#{args[1]} #{args[2]} #{args[3]} #{args[4]}"})
        end
    end
end
modules/virtualization/manifests/storage.pp:
class virtualization::storage {
    storagepool_construct($storagepool_src_name, $storagepool_type, $storagepool_src_host, $storagepool_src_path, $storagepool_src_target)
}
modules/virtualization/test/storage.pp:
$storagepool_src_name = ['iscsi-host5', 'iscsi-host6']
$storagepool_type = ['iscsi', 'iscsi']
$storagepool_src_host = ['host5', 'host6']
$storagepool_src_path = ['iqn.2010-01.master:esitbi036lab', 'iqn.foo']
$storagepool_src_target = ['/dev/disk/by-path', '/dev/disk/by-path']

include virtualization::storage

So, we have a couple of test storagepools we want to set up. For the moment I’m just creating file resources as a proof of concept – the next step is to write a libvirt type and provider for storagepool creation. The output when we run Puppet on the test manifest is now as follows:

$ puppet -v --modulepath ../ test/storage.pp
info: Autoloaded module virtualization
info: Autoloaded file virtualization/storage from module virtualization
info: Applying configuration version '1288079226'
notice: /File[/tmp/iscsi-host6]/content: defined content as 'unknown checksum'
notice: /File[/tmp/iscsi-host5]/content: defined content as 'unknown checksum'

And that’s how it is done – probably not in the best style, but it does work.

Tags: ,

Tuesday, October 26th, 2010 Tech 2 Comments

Linux kernel fun

by Oliver on Sunday, October 24th, 2010.

One of my favourite pastimes (sad as it is) is finding new and cool things that the Linux kernel can do. Short of reading through the actual source code, a relatively sane way to find your way around the kernel is by reading the documentation provided by the Linux Cross Reference. Seriously, bookmark it now.

Ever since my baptism of fire with our Sun OpenStorage cluster shortly after joining Nokia, I’ve actually quite admired the data security aspects of ZFS and RAIDZ. Former colleagues of mine at Anchor Systems are probably laughing at me now, as they’ve been running it for years. If you are running fairly regular scrubs on your pools there is very little chance of losing data due to random bits being corrupted on faulty drives. The software takes care of finding out when your hardware is likely to fail.

Wouldn’t it be wonderful if Linux could do this as well with its md RAID driver? It turns out that it can, actually. If you run a simple command such as:

echo "check" > /sys/block/md0/md/sync_action

you will trigger a check of the consistency of your software RAID set. I have no idea yet if it is as reliable as ZFS scrub, but it seems like a worthwhile thing to schedule to run once a week. You can then check the contents of mismatch_cnt to see if any errors were found. Let’s hope it outputs on which device the inconsistencies were found somewhere useful…

Sunday, October 24th, 2010 Tech No Comments

Devopsdays 2010

by Oliver on Friday, October 15th, 2010.

I’m in Hamburg! If you’ve read through my previous articles I’ve written for Anchor you would know that I love going to tech conferences. I’m currently at Devopsdays which is being hosted in the very stylish East Hotel, just down the road from Hamburg’s red-light district. I haven’t actually been outside during the day yet since we caught the EC train up from Berlin last night, but I’m enjoying the conference so far.

We’re trying to implement a Devops culture at work, which so far has just resulted in a lot more work as the initial chaos subsides somewhat but I’m always cautiously optimistic about these things. In stark contrast to the Linux Conf AU conferences I’ve been to (3 so far, but I missed the last one sadly) there is definitely a sense of this conference being in its infancy. There is a lot of excitement about the potential of Devops, and a lot of people very interested in implementing it but not really sure how to. There seems to be only a few “experts” and many “novices” eager to learn, whereas LCA seems to be a more even mix (with the odd number of wizards thrown in too).

I actually managed to score my own Open Space session to talk about configuration file distribution issues which was informative, and confirmed I’m at least on the right track. On another note, if you’re looking for interesting work as a Developer or Sysadmin in our Devops team ( 😉 ) Nokia is hiring in Berlin!

Tags: ,

Friday, October 15th, 2010 Germany, Tech No Comments

ZBOXXXXXXX

by Oliver on Saturday, October 9th, 2010.

If you’ve been following my excessively techy posts you will be aware that my personal notebook computer has been on the verge of utter failure for some time now. It has come to the point where I can no longer tolerate holding a finger in the “magic” spot to keep the screen alive long enough to update my Facebook status or other meaningless activities, and standing up in front of the TV with the notebook connected by a short cable just wasn’t fun. Not that I need more computers, but the time had come to buy the much-desired Zotac ZBOX HD-ID11.

No, this isn’t a picture of the ZBOX. We’ve been going a bit crazy ordering things from Amazon recently, including a bunch of books which are probably over my head. I recently read through Stasiland and After The Wall on recommendation from some friends (and especially relevant given we are living in the former East Berlin). They both whet my appetite for reading which has been stagnant for some time. Ange has been a long-time fan of Steven Pinker having studied his work during university and we both really enjoyed his talks on TED (if you haven’t seen them, I highly recommend them) usually centred around cognitive science. So I’m currently reading his book How The Mind Works which is a thoroughly good read. The Language Instinct actually precedes the book I’m reading but has been updated recently so that will be next on the reading list.

So, I was most pleased this morning when we returned from our regular visit to Bar Centrale to find a message from DHL in our letterbox (yes, they deliver on Saturdays in Germany!) and went to visit the random neighbour they had decided to leave our stuff with. There is never any indication of exactly what has been dropped off so the excitement levels are always high. I was expecting just books but sure enough the ZBOX was there. I cracked it open and the insides were a little underwhelming (in fact not unlike what a couple of my self-made wireless KVMs looked like) but hey, it certainly gets the job done.

Within minutes I had the 2.5″ SSD and RAM installed. It is really just minimal hardware to facilitate the operation of the box, the real storage will be outside of the unit as large 3.5″ spinning media tends to be large, noisy and hot. Not long after I had the thing back together I realised I was missing a couple of key components – a video cable and some sort of input device (all the other computers in the apartments are notebooks). Medimax furnished me with a very cheap HDMI cable (no, I didn’t spring for the gold-plated Monster cable!) and a wireless keyboard and mouse.

Huzzah! It was a little easier than I expected, getting the unit working with the TV but having never played with HDMI before I didn’t know what to expect. There will be the compulsory compiling of latest ALSA drivers I suspect, to get the audio-over-HDMI functionality working.

Fortunately we already had a USB DVD drive so it was a piece of cake getting Ubuntu installed. At some point I will install the Mythbuntu additions and get this thing functioning as a proper HTPC. There goes my weekends for the next few weeks 😉

Tags: , , , ,

Saturday, October 9th, 2010 Tech 1 Comment