adrift on a cosmic ocean

Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We have lived in Berlin since 2009, have two kids, and have far too little time to really justify having a blog.

Chef's attribute system is broken

Posted by Oliver on the 19th of December, 2013 in category Tech
Tagged with: attributeschefrant

Disclaimer: I'm not what I would call a "hard-core" Chef user, but I'll explain in a moment why that is irrelevant.

I've been working on and off with Chef for almost a year and a half now; certainly not something I use every day or even every week but perhaps a handful of times per month something will come up that requires me to jump into Chef land and get some real work done. There are definitely people in the organisation with a lot more Chef knowledge than I, and I utilise their knowledge to make up for my own knowledge gaps.

If you don't know me, I have a fairly good background already in configuration management. I would safely say I fit into the "power-user" category of Puppet users at my previous employer, having worked quite a lot on custom Ruby bits and pieces for our installation, and even giving a main-stage talk at Puppet Camp in Amsterdam in 2011. So I know plenty about the general landscape.

One of the main features of the system I helped build around that time was a very flexible data store and layered configuration engine - an External Node Classifier - which, mostly due to user demand from those using our system, went from a handful of layers at which you could define data to over 12. These allowed you to express variables tied to a given app, role within that app's deployment, datacenter, region, environment, and machine itself. Originally we had just a few of these, and there were just a few combinations in which you could use them. Demand grew until we had covered a reasonable amount of combinations, at which point the system became a little unwieldy.

We tried documenting the order of overrides permissible, which turned into a fairly awful document, and it confused the hell out of new users who had much simpler requirements. But despite supporting so many combinations already, there would always be someone else coming up and asking for some other new combination we hadn't yet supported. When I left, we were just starting to gear up for a rewrite of the engine that would support a completely different mechanism for configuration that wasn't limited to just facts about the app or its deployment environment, but I'm not sure how well that went.

My point is, if you haven't already guessed, is that Chef replicates all of that flexibility and all of the pitfalls that go along with it. Initially I was comforted by its familiarity, but now I find it to just painfully shackled to the same inescapable problems I was trying to forget, and even introduces problems of its own. Take a look at the documentation for deep merging attributes.

A feature in this area ("knockout" prefix for deep merging) was even removed in the last major release. Deep merging makes so little sense in Chef, it pains me. Chef has all the flexibility of raw Ruby at its fingertips, and yet deep merging was decided to be necessary anyway - even though you can more or less accomplish any data transformations you want with native code. And not all of it is even correct.

I've seen several examples in our own codebase of deep merging being used as intended - with hashes - but also several examples where arrays have been used "to work around deep merging". If you read the documentation, you'll see that both hashes and arrays are candidates for deep merging and yet apparently even on the latest version (11.8) this is not really the case. Another point that pains me is that in order to avoid the deep merging behaviour encourages all of the bad habits that go along with Ruby's duck-typing system - if you have an array already in place in a cookbook default attribute, and define a hash (or any other type) in a role default attribute it will be overwritten wholesale. This potentially puts a large onus on cookbook authors to excessively type check and stamps a large "WAT" all over this.

Finally, the complexity of the attribute definition layers is just excessive. I've previously done a lot of stuff like this in Puppet and it still makes very little sense to me. I was able in one instance to override a hash defined in the cookbook default attributes with another hash defined in the role override attributes despite them apparently being candidates for merging, and yet when the role override attributes were moved down to role default attribute level they WERE merged. The system is just far too complex and complicated, even for relatively savvy users. If a configuration management system requires you to do excessive juggling of config data just to get it working properly, it's actually costing rather than saving you time.

Fortunately there are now other (what I will still call "current generation", since they haven't made any huge noticeable improvements over current offerings) configuration management systems in the marketplace such as Ansible, Salt and Juju. In my brief examinations of them, none of them appear to do much of a better job but at least it is good to have other alternatives than just Chef and Puppet (and I suppose CFEngine3, if you're a real masochist). Will there be another generation? Based on my experiences with containerised application hosting in the style of Docker, Heroku and others, perhaps today's configuration management systems are trying to do too much already and we just don't need more - we need less.

© 2010-2018 Oliver Hookins and Angela Collins