adrift on a cosmic ocean

Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We have lived in Berlin since 2009, have two kids, and have far too little time to really justify having a blog.

Tools that make your life harder

Posted by Oliver on the 20th of March, 2014 in category Tech
Tagged with: hadoophivelog analysispigproductivity

This post title is inspired by this Google Plus post, although I've been meaning to write this post for a few days anyway (it's just a catchier title to describe the same thing). I'm not a data scientist, analyst or even a hard-code Hadoop user by any stretch of the imagination, but on occasion I need to do some log analysis when there is simply too much data to force through Awk (as much as I hate to admit it).

For perhaps the last year I've been using Pig. I say "using", but really it is learning, trying, failing, learning again, scratching your head and maybe eventually using. Since my usage is fairly sporadic I forget everything after a month or so of not using it and then start again the next time I have to use it. I've been away from Java for so long it's better to just say I never knew it in the first place, so whenever my job fails and I get a 50-line stack trace it is usually fairly difficult for me to piece together why it failed, and let's not even talk about trying to write UDFs. It's a tool that I needed, but it undoubtedly made my tech life harder in some respects.

But, on the whole it has made me more productive with Hadoop. I don't have to write any Java, and when it does work it works fairly well. However, the Hadoop ecosystem is by now fairly rich (even if you consider Apache projects exclusively) and there are alternatives to Pig at similarly high levels of abstraction above the basic Map/Reduce system. I've been meaning to look into Hive for a while, especially as I read up recently on Redshift and concluded that the SQL-based approach is gaining popularity on multiple fronts. If you are not familiar with either Pig or Hive, essentially Pig has its own high-level language whereas Hive has a derivative of standard SQL called HiveQL (and if you already know SQL there is not much difference, at least from what I've seen so far).

I was slightly shocked (as well as pleased) at how much of a difference there is in expressiveness and understandability between the two. Here's an example of analysing some logs to find out the top 5 contributors to cache misses by User Agent, using Pig:

cache_misses = FILTER raw_logs BY sc_status MATCHES 'TCP_MISS.*';
cache_misses_by_ua = GROUP cache_misses BY c_user_agent;
cache_miss_count_by_ua = FOREACH cache_misses_by_ua {
    COUNT(cache_misses) AS cnt;
ord = ORDER cache_miss_count_by_ua BY cnt DESC;
top5 = LIMIT ord 5;
DUMP top5;

This is actually a simplified version from what I had previously which involved nested blocks, but that only serves to further illustrate my point. I find it very hard to wrap my head around nested blocks and how to use them to get the data out in the way I want (usually involving ordering and limiting). Maybe my Pig fragment here could have been described more concisely, but I doubt I would understand it in any fewer lines.

For comparison, here is what I wrote to get the equivalent data out of Hive:

select count(*) as cnt, c_user_agent from raw_logs where sc_status like 'TCP_MISS%' group by c_user_agent order by cnt DESC limit 5;

Pretty straightforward, right? The Hive query above and the Pig script fragment both produce the exact same result set (ignoring the minor formatting differences). Both Hive and Pig require approximately the same amount of lines to set up the log parsing, mostly because it involves setting up each field label and data type individually and then a regex to parse the fields out of the input files. If you have a deserializer UDF this is made much easier in either case.

It appears I may have to knuckle down and write a deserializer in Java for at least one format that some of the logs I use are stored in, but outside of that I feel I would be much more productive in Hive. I can only recommend trying it out if you are wanting to get your hands dirty with Map/Reduce on Hadoop and don't want to dive down to the murky Java depths of the system. Where Pig made my life harder, it seems Hive has the potential to make it vastly easier again.

© 2010-2018 Oliver Hookins and Angela Collins