Archive for July, 2011

OpenCV on the N900

by Oliver on Wednesday, July 27th, 2011.

I’ve been playing around with OpenCV over the last few weeks while reading/coding along with the Learning OpenCV book. Overall I’m pretty impressed with how easy it is to do quite complex things, thanks to the rich library of visual computing functions. The library is natively in C, but there are also python bindings using SWIG. It is actually an interesting exercise taking the examples in the Learning OpenCV book (which were written for C) and rewriting them in Python. Incidentally, my worked examples are here if you want to reuse them or take a look at my terrible code.

Actually there are not too many changes necessary and you don’t have to worry about pointers or memory management which is handy. The great thing about having a Python library to use is that it is generally cross-platform. Sure enough, someone has packaged OpenCV for Maemo 5. Unfortunately the default Python version in the Scratchbox build environment for the platform is 2.3, which apparently doesn’t play nicely when trying to build python-opencv. So last night I rebuilt against Python 2.5 which is also available in the regular “Fremantle” repositories for the phone and today have a working python-opencv environment.

I ran one of the example programs which simply opens the default system camera and displays it on the screen, both on my laptop and on my phone and this is the result:

I’m not sure if OpenCV has any camera focusing calls, but in any case none are used in this example obviously so the quality is not great. There is also some colour-correction that perhaps the standard imaging utilities in the phone uses but OpenCV lacks so the colours are a bit off. Rest assured though, that both the computer and the phone are indeed running the same (very simple) script (chapter2/example9.py).

A bit more code and you could have the computer and phone tracking each other, and after a few more iterations lead to some kind of T-800/T-1000 cyber-stand-off 😉 I’m talking to the python-opencv package maintainer to get the fixes incorporated into the public repositories (extras-devel right now) but if you want them earlier please let me know.

Tags: , , ,

Wednesday, July 27th, 2011 Tech 2 Comments

Self-rating on the Spolsky/Limoncelli tests

by Oliver on Monday, July 25th, 2011.

Tom Limoncelli just released his own version of The Joel Test – except his one is for sysadmins. I was only vaguely aware of Joel Spolsky’s test and only just read through it and rated my current team, and I’m glad to say we are just about at twelve for twelve. It hasn’t really been on my radar until now as this is the first time I can actually say I’ve been on a development team. Given all the buzz around DevOps, this is quite an exciting phase of my career as we are basically a development team of Systems Engineers, working on automation and all the things sysadmins dream about.

We actually spent a good amount of time planning out our product, developing our ways of working and a few sprints down (yes, we are using Agile) things are generally going pretty smoothly and enjoyably. The team is a handful of geeks with a range of interests not just related to pure sysadminery so it makes for a pretty enjoyable place to be right now.

The only areas I can say we don’t entirely meet Joel’s criteria are:

  • Daily Builds. We develop in Python and Ruby and our test suites are pretty quick, so we build on every commit (or push for Git). Instead of having a single monstrous test suite for everything we split things up pretty aggressively and have all the simple tests done in the first stage of the pipeline, and more intensive tests done in later stages so as to enable fast feedback. I’ve written (and talked) about this a few times before.
  • We don’t have testers. We do however have a rigorous code review mentality, actually do pairing when writing code (it’s totally worth it) and test each other’s work constantly. I know this doesn’t meet up to Joel’s testing criteria but we’re only a team of five or six right now. Hopefully this will change in future (although we do have a very willing army of beta testers right now).
  • Candidates don’t yet write code in interviews, but this is something we definitely intend to change. Sadly the DevOps movement is still young and it is extremely difficult finding people who are both competent sysadmins and coders. Usually they will be one or the other and I have read several times now how it is easier training a developer to be a good sysadmin than vice-versa (and this certainly on face-value would seem to make most sense). If you want to prove me wrong, send me your CV and come in for an interview! 😉

So overall I think we’re doing pretty well. On Tom’s (extremely comprehensive) test I have to say we (or rather, my old team now) don’t fare so well. To be completely fair, we maintain a huge array of services, maintain environments which span development VMs where developers germinate the first seedlings of an idea to the production machines on which they serve millions of users world wide, datacenters not only local and abroad for R&D but in many countries around the world serving our production services. Some of us even manage to fit on-call time into our crazy lives.

We are sometimes at the mercy of development teams pressing to get their products live so I can’t say we have a strangle-hold on all of Tom’s test items – but if we did, what would be the challenge and fun that keeps us coming back day after day? I also see Tom’s test differing significantly from Joel’s. If you score 10 or less on Joel’s test you are in pretty big trouble. Tom’s test has only a handful of such critical items among the 32 questions, so it is much easier to fall into the trap of either thinking you are doomed (and you’re not) or thinking you are fine (and you’re not). Not to say that his test is any less valuable, and of course such things are very useful for applying good DevOps practices to sysadmin teams.

So my advice is, take a look at what you didn’t answer yes to and think about what you can do to get there. I can personally vouch for many of the items being painful to implement when you don’t already have them, but once you have it done you won’t regret it and you won’t look back (unless you maintain your documentation in LaTeX… never again!)

Tags: , ,

Monday, July 25th, 2011 Tech No Comments

Reliably finding processes with ps by name

by Oliver on Sunday, July 17th, 2011.

I imagine that there are two groups of people who might read this post:

  1. When you need to find a process by name, you run ps -ef or similar and pipe into grep processname.
  2. You gladly suffer the presence of your own grep process being shown in the output, or maybe even grep -v it out (these are the “heathens”).

  3. When you need to find a process by name, you run ps -C processname
  4. (these are the “enlightened ones”).

If you fall into the first category, you fail my interview tests. Perhaps you smugly fall into the second category, but surely you have seen this occur:


ohookins 4410 0.0 0.2 212804 9384 ? S 20:10 0:00 /usr/lib/bamf/bamfdaemon
$ ps u -C /usr/lib/bamf/bamfdaemon
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
$ ps u -C bamfdaemon
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ohookins 4410 0.0 0.2 212804 9384 ? S 20:10 0:00 /usr/lib/bamf/bamfdaemon

I don’t particularly care about bamfdaemon, but given that the process listing shows the full path to the binary, why can’t we search for it by this process name? Why does the unqualified filename work? OK, perhaps it is just basing the match on the unqualified filename…


ohookins 3710 0.0 0.9 468988 38216 ? Sl 19:49 0:05 /usr/bin/python /usr/bin/terminator
$ ps u -C terminator
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
$ ps u -C python
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
$ ps u -C /usr/bin/python
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
$ ps u -C /usr/bin/terminator
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ohookins 3710 0.0 0.9 468988 38216 ? Sl 19:49 0:05 /usr/bin/python /usr/bin/terminator

OK, what the heck is going on here exactly? I’m not terribly familiar with the POSIX specification, so let’s take a look at the source code of procps:


109 break; case SEL_COMM: i=sn->n; while(i--)
110 if(!strncmp( buf->cmd, (*(sn->u+i)).cmd, 15 )) return 1;

In select.c of ps, we see these two lines in the case statement which selects between different process identification mechanisms. -C actually allows you to select multiple processes by different name since it iterates through the list of selectors (which I didn’t know before looking at the code – very cool).

A limited string comparison is done between the argument given to -C and the process being examined. You can see that this limit is 15 characters, and in the union inside the selection node only 16 characters are stored anyway. Let’s have a look at what this proc_t buf struct looks like so we can figure out what the comparison is being done on. This sits in proc/readproc.h:


char cmd[16]; // stat,status basename of executable file in call to exec(2)

Now we are getting somewhere. We can easily verify that the limit in the character comparison is being done:


$ ps u -C upstart-socket-
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1123 0.0 0.0 15004 400 ? S 18:46 0:00 upstart-socket-bridge --daemon

So if you are lazy, you only have to type 15 characters at most of your process name. Let’s look at the more complicated case of when processes are just really hard to find by any name we can see in the process listing – my candidate case for this is Jenkins, which is notoriously hard to track down especially if you are running several Java-based services on the one machine (for example Jenkins itself, Nexus and perhaps Sonar which all logically fit together as part of a typical Java build server):


$ ps uwww -U jenkins
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
jenkins 19517 9.3 30.8 1823496 1245768 ? Ssl Jul08 1255:29 /usr/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -Xmx1024m -Xms768m -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

Nothing amazing here, let’s find this process by the command name:


$ ps uwww -C java
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
nexus 3853 0.0 3.5 1442536 145520 ? Sl Jun06 3:17 java -Dsun.net.inetaddr.ttl=3600 -Dbasedir=. -Djava.io.tmpdir=./runtime/tmp -Djava.library.path=bin/jsw/linux-x86-64/lib -classpath bin/jsw/lib/wrapper-3.2.3.jar:./runtime/apps/nexus/lib/plexus-classworlds-1.4.jar:./conf/ -Dwrapper.key=ybUhRQr9hU88aJwC -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=3837 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.codehaus.plexus.classworlds.launcher.Launcher

Wait, where is Jenkins? Didn’t we confirm that the process running was in fact /usr/bin/java, and we know only java is used as the executable basename inside of ps? How is it possible that ps is now not showing us the Jenkins process? Let’s have a slightly different look at it:


$ ps -U jenkins
PID TTY TIME CMD
19517 ? 20:55:47 exe
$ ps uwww -C exe
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
jenkins 19517 9.3 30.7 1823496 1245124 ? Ssl Jul08 1255:50 /usr/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -Xmx1024m -Xms768m -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

Aha! We have found the errant Jenkins process. But why is the basename exe? As it turns out, it is a peculiarity of Jenkins itself, documented in https://issues.jenkins-ci.org/browse/JENKINS-9206 which also causes problems with the init script (when it tries to find the process with the incorrect method, as we found above, due to certain assumptions).

In any case, now we’ve seen how ps operates and even how to find a process using the correct method, even when that process is playing hard to get.

Tags: , ,

Sunday, July 17th, 2011 Tech 7 Comments

Finding inspiration

by Oliver on Wednesday, July 13th, 2011.

I fully own up to and take responsibility for my unending perfectionist streak (although of course I am far from perfect) – I am never happy with the work I have done, am always looking for new challenges, and constantly think that I’m not doing enough (in various different areas of my life). This generally applies to work and associated pursuits but also to just about any other area I care to mention – learning languages, fitness, music etc. The inevitable time sucks aside (being a parent, Facebook, movies, RSS feeds, writing blog posts etc) I’ll try to describe how I find inspiration from day to day. This may sound rather vague but I guess this is mainly a post to give some anonymous credit to some people who have inspired me and continue to inspire me, mainly on a technical level.

I was fortunate enough to have found my last job through a friend, and while I didn’t have much idea of his tech background at the time, when he left I ended up filling his position until I left a little less than two years ago now. He ended up going to a rather large, well-known search engine company so you can guess he set a fairly high standard. Looking back, I was extremely green coming into that job and his presence was invaluable to me. He was always working on some system or writing some code that was a couple of steps out of my reach. I didn’t realise it much at the time but I unconsciously strived to catch up to him in some way (a futile effort) and he definitely inspired me to get into programming again, which has served me very well the last year or so.

When he left the company, I felt very uneasy stepping into his shoes, feeling completely unworthy of the role. It was actually a very good thing and I feel like I grew into it, and eventually felt that I left my mark after I myself had left. It was a big challenge stepping into the position of the lead tech for systems and networking in the company but definitely rewarding, and provided a lot more scope for bridging the gap between what I would call more standard sysadmin work and lead tech/architect work which is where I find myself these days.

Something I feel like I needed a lot more at that time was someone I could always fall back on who had all the answers when I came up stumps. The technical director of that company is extremely experienced, knowledgeable beyond all reason and very fastidious. I learned my sometimes excessive attention to detail and obsessive compulsive approach to work from him, and while it also comes with bonuses like constantly washing your hands, doubling back to make sure you have turned off light switches and so on (or maybe that’s just me) it has served me well. It was also incredibly reassuring to know that someone around would have the answer even if you didn’t, and when we were trying to solve problems in often unexplored ways, his presence was invaluable in that process of development.

Right now I feel a lot more self-confident in my own decisions and capability to find the right answers either through brute-force or learning that I don’t need that reliance, but it is still useful having peers you respect to bounce ideas off. You might call these things “knowing yourself” and “distrusting yourself”. Something I only discovered over the last few years is how my mind actually operates. I know I don’t take knowledge in immediately, but percolate through it unconsciously – so it doesn’t bother me as much when I don’t understand things as soon as I read them or someone explains it to me. I take a break away from thinking, or sleep on it and can relax knowing the answer or realisation will be there without forcing it.

And so, I feel like what is almost two years now at Nokia have been some of my most productive, mentally and developmentally speaking. I think I have done much less typing but much more thinking, which is a change, but also appreciate my inspirations and goals a lot more. I did accomplish the goal of programming more, and I do enjoy it (which was the whole point in the first place). I’m not sure if I’m good at it yet, but that’s another matter. So the last person I wanted to mention is actually a reasonably new recruit to my team (and ironically from that large search engine company). He again seems to be constantly two steps (actually a lot more) ahead, but has an odd array of hobbies strongly rooted in Electrical Engineering (which is where I started out at Uni).

This is another area that perhaps I had only vaguely wanted to get back into, but exploring it now has got my imagination going wild. Right now I’m having a look at building some kind of motion detection system (for various reasons which may be revealed over the coming months) using OpenCV on his suggestion. There is a wealth of other embedded systems and electronic projects that could be extremely fun and hopefully rewarding both in terms of what can be built and the knowledge acquired. It’s good having someone around to guide me in this field that I have been out of for perhaps a decade now.

The main takeaway from this all could be something along the lines of the trite adage of standing on the shoulders of giants, but I think it is simpler than that. You don’t need to be surrounding by renaissance men (although it helps) who can pull you up – you just need one or two people in your life who inspire you and do things you hadn’t perhaps thought of doing, or it hadn’t occurred to you to do. If you can find the time to better yourself by exploring these curiosities rather than just leaving them as such, I think you’ll find yourself a happier person in the long run.

Tags: , ,

Wednesday, July 13th, 2011 Tech, Thoughts No Comments