adrift on a cosmic ocean

Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We have lived in Berlin since 2009, have two kids, and have far too little time to really justify having a blog.

Checkpoints and Gatekeeping versus Principles and Values

Posted by Oliver on the 9th of December, 2018 in category Tech
Tagged with: alignmentdevelopmentengineeringleadershipproductivity

Picture the scene: you have a new project to start working on and deliver in a relatively short amount of time. You have done lots of research, know the problem to be solved, have worked diligently on whittling down the scope to just the bare elements that will prove or disprove the utility of the idea. You know how long it will take to build a first iteration, have a pretty good idea of how to build it and which technologies you'll use. You've already consulted with the relevant business and product people and now things look good to move ahead with building it!

As the next step in due diligence, you write up a proposal which will be shared around engineering to gather any relevant feedback on how you will build the new service, and catch any problems in your design before they are irrevocably committed. You set a reasonable period for review of your plans and send it off into the world. What follows is a storm of comments on everything from naming to purpose to whether the service should exist at all. Some more friendly folks might schedule some time with you to sit down and discuss why it is that you are building this, and try to address some of the concerns they have in a more personal manner.

Does this sound familiar? Even if everyone involved in the process has the best of intentions, and you've followed your engineering processes to the letter, what has transpired is still a fairly fundamental failure of an engineering organisation. If you are reading this post, then you've probably worked in an organisation with very traditional, gatekeeping operations engineers. I can't fault them - in these old siloed organisations, many teams had conflicting priorities. However, most of the time today we don't even need one team to distinctly say "No. This won't ever ship." for there to still be a big problem in our software delivery process.

I'm talking about shared principles and values across engineering. What do I mean by that? Ideally, you should be able to ask yourself "How do we build stuff at X" (replace with your company name). One aspect (which I'm not dealing with here) is Engineering Culture - a separate topic altogether. What I mean is, does everyone have a shared understanding (another one of my favourite phrases) of how they design, build and ship software within your company? If it's a big company, it's quite possible that there are multiple ways this can happen, and you may have even optimised for this kind of autonomy. A couple of examples at opposite ends of the spectrum might be Google (where everyone must interact with the Monorepo and use very widely shared tooling and libraries) or Zalando/Spotify/Netflix (where a high degree of autonomy is granted, as long as the end result achieves the business goals). Of course I've never worked at these companies so I cannot say for sure.

What might such a set of principles or values look like? Here are some examples (and you can associate positive or negative connotations to them as you like - it's entirely contextual):

  • We never deploy on Fridays.
  • Adding a new critical dependency to our technology stack always requires approval from an architecture group.
  • We only use languages X, Y, or Z.
  • We allow any language, as long as it runs on the JVM.
  • Code review is a mandatory step towards production.
  • Critical services must always implement circuit breakers, exponential backoffs and back-pressure on upstream services.
  • All services must implement a healthcheck endpoint and provide metrics for request rates, errors, and latency.
  • Any service that utilises object caching must be built to service production traffic loads even without caching for short periods of time.
  • We allow any data-store to be utilised as long as it is supported by AWS.
  • We prefer boring but understandable naming.
  • We prefer buy over build for areas that are not our core competencies.
  • Any new features must be supported by canary releases, percentage rollouts etc. No big bang releases!

This is an artificial list off the top of my head, but hopefully you get the idea. Where should it come from? Who gets to write this and hold the stick of values over the engineering department's head henceforth? If you found a startup yourself, in a lot of ways these principles and values are implicit and you help to cement them through your actions. Others learn from how you work and the positive reinforcement of public praise, and learn the negative behaviours through what receives (hopefully constructive) criticism. You may even decide to commit these principles to (virtual) paper, in wikis or other documentation. You might like to incorporate these principles into your onboarding processes, so that new joiners to the company can already begin work with the right ideas in their mind as they begin building.

Later on, once the engineering organisation is established and you as a leader are no longer so deeply involved, it becomes harder. Should you be the "Benevolent Dictator", set the rules to be obeyed (even if from a place of good intention) and continue to wield the stick rather than the carrot? Or, should you institute a democratic process, collecting ideas via the inevitable Google Forms that are sent around via email - I'm sure you are familiar with this pattern.

I believe that behaviours that should be encouraged will emerge naturally, should be publicly (and I mean internally within the company, but visible and transparent to all) praised but then committed to some kind of lasting record. If you notice that an unhealthy culture of gatekeeping or checkpoints has arisen to deal with the lack of alignment, it's a good sign that not enough shared principles have been captured. It's worthy of having some kind of Retrospective or information gathering mechanism, sourcing current and desired practices from those who are building the product, and encode them in your lasting record. One option is to gather ideas on sticky notes and have a voting process where you crowd-source the new principles the majority of your team would like to start implementing. Then, all you have to do is provide the time and space for them to actually do it!

These are some, admittedly very basic ideas. I would say that in general, building up this kind of shared vision around how your engineering is actually performed day-to-day is a very challenging task. We only know about the companies that appear to have gotten it right, because they are still around. How they got there is partially a product of their culture and partially a number of elements of their secret sauce - things that they might not be so willing to share, or perhaps wouldn't work very well elsewhere. What seems reasonably clear to me though, is that these principles and values MUST be captured, easily referenceable and easily shared from mind to mind in your organisation. Without them you are doomed to implement efficiency-killing synchronisation/coordination processes.

Have you encountered this pattern in your company? How did you identify it? Have you solved it or attempted to address it? I would love to know your experiences so please leave a comment below if you can share anything!

© 2010-2018 Oliver Hookins and Angela Collins