adrift on a cosmic ocean

Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We have lived in Berlin since 2009, have two kids, and have far too little time to really justify having a blog.

  • LZ* compression algorithms
    Posted by Oliver on the 25th of May, 2014 in category Tech
    While I was implementing a small, naive log aggregation tool I had a moment to consider the type of compression I wanted to use on the log files at rest. The main implication this has outside of the efficiency of compression and how much space the files will take up is how they can be used once stored. If you are using tools like gzip/gunzip with awk or other simple command-line tools, or even…
  • Excluding files from logrotate globbing matches
    Posted by Oliver on the 21st of May, 2014 in category Tech
    A few days ago I wrote about logging with Upstart and the various implications that has if you want more frequent log rotation than the defaults provide, or do something other than rotation with the files that logrotate cannot do. There doesn't seem to be any good canonical source of information about how to exclude certain files from the logrotation so I thought I'd make a small note about it…
  • Understanding Upstart's logging options
    Posted by Oliver on the 18th of May, 2014 in category Tech
    I have a couple of services I own that run in EC2 that, sadly, still lack any log aggregation. In-house we have our own system for this, that technically can also run in EC2 but for a variety of reasons I consider it overkill for my use case in EC2. Basically, the solution I am after is to do the minimum amount of work to see these logs aggregated to an S3 bucket, making them available for…
  • (something something) Big Data!
    Posted by Oliver on the 12th of April, 2014 in category Tech
    I recently wrote about how I'd historically been using Pig for some daily and some ad-hoc data analysis, and how I'd found Hive to be a much friendly tool for my purposes. As I mentioned then, I'm not a data analyst by any stretch of the imagination, but have occasional need to use these kinds of tools to get my job done. The title of this post (while originally a placeholder for something more…
  • Tools that make your life harder
    Posted by Oliver on the 20th of March, 2014 in category Tech
    This post title is inspired by this Google Plus post , although I've been meaning to write this post for a few days anyway (it's just a catchier title to describe the same thing). I'm not a data scientist, analyst or even a hard-code Hadoop user by any stretch of the imagination, but on occasion I need to do some log analysis when there is simply too much data to force through Awk (as much as I…
  • Cloudformation and the data transformation nightmare
    Posted by Oliver on the 7th of March, 2014 in category Tech
    The background to this story is that I spent the bulk of one week recently working on getting a prototype service deployed with AWS CloudFormation , and the experience was still reasonably painful. My team has other services deployed with CloudFormation, which is working perfectly fine (now) but I had hoped there would be some improvements available since the last time we went through the process…
  • Elastic MapReduce and data in S3
    Posted by Oliver on the 28th of February, 2014 in category Tech
    I don't have to do much data analysis fortunately, but when I do there are two options: either the data is local to our own datacenter and I can use our own Hadoop cluster, or it is external and I can use Elastic MapReduce. Generally you don't run an Elastic MapReduce cluster all the time, so when you create your cluster you still need to get that data into the system somehow. Usually the easiest…
  • Setting goals for learning for 2014
    Posted by Oliver on the 18th of February, 2014 in category Tech
    Perhaps a little late in the year to be conducting a personal retrospective on the years past, but I feel at this point I'm starting to wonder about the challenges ahead. The last two to three years I've distinctly changed my career direction from systems engineering, to "DevOps" (whatever that means anymore), to developer. Sure, I'm technically a Computer Scientist by tertiary education standards…
  • The current state of feature flags
    Posted by Oliver on the 11th of February, 2014 in category Tech
    Feature flags, flippers, toggles, selective rollouts and various other terms have been used to describe systems that allow you to deploy code without necessarily forcing it on all of your users. I won't describe it here as it has already been covered in depth by a variety of companies and people, notably (and not necessarily in date order) Flickr , Github , Forrst , Martin Fowler and many…
  • Seven Languages - Clojure
    Posted by Oliver on the 10th of February, 2014 in category Tech
    I notice my pace has yet again slowed between the last chapter of the book - Erlang - and this one. Another five months has passed since I finished the chapter on Erlang! In actual fact, I haven't been slaving away on the next language that whole time - decompression of sorts has to follow each chapter, and dealing with a manic three-year-old, finding some time for a bit of exercise and trying to…
© 2010-2018 Oliver Hookins and Angela Collins