adrift on a cosmic ocean

Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We currently reside in Sydney after almost a decade in Berlin, have three kids, and have far too little time to really justify having a blog.

Exceptional circumstances

Posted by Oliver on the 24th of November, 2011 in category Tech
Tagged with: exceptionsfrustrationihaterubymochamockingrspecrubytesting

I'm still building up to my article on how to properly mock calls to Rake's sh to facilitate testing of your Rakefile tasks but I haven't quite worked up the strength to trace the calls through the libraries and into Eigenclass wonderland. For the moment, I've got just enough fodder to write a bit about determinism in testing.

Since Friday, both a coworker and I have been hit by the same testing failure. Not the exact same one mind you as we are working on two different pieces of software but the same failure in that the order of our tests matter. Anyone experienced in TDD will tell you that you need to make sure your tests don't depend on the order in which they run so that your units of testing don't build upon previous assumptions and don't fully test each unit in isolation. Easier said than done unfortunately.

It would be nice if we could test the cartesian product of all of our units in all possible orders to rule out any unplanned dependency but it is simply not feasible for anything but the smallest programs. In any case, our small programs managed to cause a small bit of hell just fine, thankyou! Enough babbling and onto the code:

desc 'Run tests''test') do |t|
  t.pattern = 'spec/*_spec.rb'

This is a fairly standard task that sets up RSpec tests using RSpec's Rake helpers. Not much to see here - except for the fact that my two test files run in opposite orders on different machines. I managed to rule out different Ruby versions (exactly the same, installed in both places by RVM), Gem versions (same of Rake, RSpec and all relevant dependencies), even the locale on both machines was identical to rule out sorting order differences in the filenames.

RSpec uses RSpec::Core::RakeTask#pattern to assemble a FileList with the pattern you have set. FileList (defined by Rake) basically uses Dir.glob to get its dirty work done:

# Add matching glob patterns.
    def add_matching(pattern)
      Dir[pattern].each do |fn|
        self << fn unless exclude?(fn)

Dir[] just aliases Dir.glob, and within the source of that you can find the following:

dp = readdir(dirp->dir);

readdir(3) just returns the directory entries in the order they are linked together, which is also not related to inode numbering but as best as I can tell is from outer leaf inwards (since the most recently created file is listed first).

Now I have some idea of why the testing order can be different, but I'm no closer to the cause of the problem - my tests succeed when run in one order and not in the opposite order. The errors look something like this:

  1) Nagios#new raises an exception when the command file is missing
     Failure/Error: expect {, @status_file) }.to raise_error(NagiosFileError, /not found/)
       expected NagiosFileError with message matching /not found/, got #
     # ./spec/nagios_spec.rb:26

Hmm, a bare Exception object with no information about where it came from. I had a few suspicions:

  • I have no idea how to code Ruby
  • I changed the code in some subtle way and broke the tests legitimately
  • Some Mock object's lifetime is longer than expected and sticking around

This last idea seemed most plausible since I was able to put some debugging code into the constructor of the object and it was not being called at all. Mocha usefully has an unstub method which allows you to remove stubs on an object/class you had previously set up and return it to its previous state, but this seemed to be a no-go:

     Failure/Error: Nagios.unstub(:new)
       The method `new` was not stubbed or was already unstubbed

I installed the very useful ruby-debug and invoked that just before the failing tests started and did some poking around but wasn't able to find much (although that's probably more to do with lack of skill than the debugger lacking functionality). How to proceed? Unfortunately the easiest way forward seemed to be the brute force method - comment out tests in the preceding file until it starts working then narrow the field. Fortunately the problem uncovered itself immediately:

  # Change the existing mock to be something we can pick up in output
  before(:each) do
    Nagios.stubs(:new).returns('surprise !')

Then in the test output:

       undefined method `cmd_file' for "surprise !":String
     # ./spec/nagios_spec.rb:55

Hmm, that's no good. The stubbing is surviving between test files. TL;DR - It's good to read the documentation! RSpec supports several mocking frameworks - FlexMock, Mocha, RR, bring your own framework and of course, RSpec itself. I already knew of this before but since I've been working mostly with Test::Unit up until recently it didn't register in my brain whatsoever and I just reached for Mocha out of habit.

It turns out that if you use a different mocking framework but don't tell RSpec about it, weird stuff happens - that is to say, the kind of behaviour you see above. You add a small block to your test code something like this:

RSpec.configure do |config|
  config.mock_framework = :mocha

Once I had confirmed this was the actual problem and my tests were passing, I decided to rewrite all my mocking code to use RSpec mocking anyway - less gems, easier to maintain.

So the main takeaways from this experience are:

  • Read the documentation!
  • Ruby is powerful and frequently easy to use, but can still not be easy to interrogate when there are problems happening that don't seem to make sense.
  • Having source code on hand to see what your software is really doing is an awesome thing. Admittedly, the metaprogramming behind testing and mocking frameworks can be some mind-bending stuff, but the code is there when you want to dive in.
  • Testing can be frustrating, but awesome.
  • Ordering is not always the problem. Try to get your tests order-independent, but just like code coverage it can be a large time sink and the law of diminishing returns applies.
© 2010-2022 Oliver Hookins and Angela Collins