Archive for September, 2011

Mocking fun with Python, mox and socket

by Oliver on Friday, September 9th, 2011.

I’ve been doing most of my work with Ruby this last year, as a lot of it was Puppet-related. I went through the pain and pleasure of learning Ruby and the intricacies of unit testing with test::unit and RSpec, and mocking objects with the excellent Mocha library. Now I have slightly returned to my roots by doing a bit of Python programming on another part of our codebase.

We use Python for some of the code we use to test our infrastructure – initially because it most suited the engineers who were implementing it and Python was already on our list of supported languages. We don’t want our language set to grow unwieldy but at this point in time we have a fairly well-defined set of domains in which we restrict our language choice. Finally in this sprint my chance came up to do some work on this code base so it was time to freshen my memory on Python unit testing.

The particular task at hand was verifying connectivity to the management interface of a piece of hardware. Without giving too much away, you might for example do an extremely simple check of a device that has an SSH interface as follows:


    # Create a TCP connection to the SSH port and grab the output
    try:
      try:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.settimeout(5)
        sock.connect((ip, socket.getservbyname('ssh')))
        sock_output = sock.recv(100)
      except socket.timeout:
        return False, "SSH Connection Timeout"
      except socket.error:
        return False, "SSH Connection Error"
    finally:
      sock.close()

    result = re.search('SSH-2.0-OpenSSH_5.1', sock_output)
    if not result:
      return False, "Valid SSH interface not found"
    else:
      return True, "Valid SSH interface was found"

No, this is not great code (and you can blame Python 2.4 for that crazy nested exception handling block), but it works and reasonably catches the obvious errors. How do we go about testing this? I have another test class which verifies connectivity to an HTTP interface of a piece of hardware, but the decision was just about made up for me with that. Since the interface actually requires SSL, it would be quite difficult, tiring and time consuming to make a real HTTPS connection to a fake server and pass some dummy text across the connection. Therefore in that case I simply stubbed the calls to urllib2.urlopen and passed the dummy text without any network being involved.

In the case of a much simpler interface like a socket, we have a quandary on our hands:

  • Create a listening socket, have it call a real object which passes data over the connection which goes back to the connecting object.
  • Create a listening socket which mocks out the send method in order to send some pre-cooked output back to the connecting object.
  • Mock the sending socket and have pre-cooked output going back to the calling object.

Ideally we’d keep things as real as possible, spinning up a connected pair of sockets on the loopback adaptor and performing reasonably real communication, but after being away from Python for a while it was far too taxing on the brain. To cut a long story short I decided to take the last option and mock the socket object itself, although as it turned out, it was a bit tricky:


  def setUp(self):
    """ Create a mock socket for later use """
    super(TestSshTest, self).setUp()
    self.mock_socket = self.mox.CreateMockAnything(socket.socket)
    self.mox.StubOutWithMock(socket, 'getservbyname')
    self.mox.StubOutWithMock(socket, 'socket')
    socket.socket(socket.AF_INET, socket.SOCK_STREAM).AndReturn(self.mock_socket)

  def test_ssh_success(self):
    """ Standard successful test """
    socket.getservbyname('ssh').AndReturn(33333)
    self.mock_socket.connect(('192.0.2.1', 33333))
    self.mock_socket.recv(100).AndReturn('SSH-2.0-OpenSSH_5.1')
    self.mock_socket.close()
    self.mox.ReplayAll()

    # real method runs here

Don’t ask me exactly why all the above was necessary, as I’m still coming to grips with it. Part of it I believe is due to the intricacies of exactly how different entities work in Python, and part of it is how sockets are set up. Pymox appears to be more complicated to use than Mocha but I’m reasonably certain that is just added flexibility and power that Mocha hides from you – I’m still waiting for the lightbulb moment.

If anybody out there has successfully mocked out sockets in Python using Pymox and in a way that is simpler than this, I’d love to hear how it was done!

Tags: , , , ,

Friday, September 9th, 2011 Tech No Comments

Bike upgrade

by Oliver on Saturday, September 3rd, 2011.

I’ve had a cheap, crappy bike for a while now and so far it has served me fairly well. It is not in great condition and has its problems but generally works despite my recent fitness kick which has seen me taking my son out for morning rides before work in the trailer which hangs off the rear axle. The additional weight of course puts a strain on the axle and gears (not to mention ME) and the already warped gears and chainrings started to complain about it. The chain started skipping off the rear sprockets more and more regularly despite my efforts to adjust the rear derailleur.

So, never the one to shy away from a bit of do-it-yourself modification work I decided to investigate replacing the gears. Berlin is fairly flat and with three front chainrings and seven sprockets on the rear cassette I was amply supplied with gear overkill. I had looked into the Shimano Nexus internal hub series and the Nexus3 had taken my fancy so I ordered what I thought were the parts I required from a shop selling their stuff on Ebay, and began the process.

Above you can see basically what I had to deal with (prior to realising that taking the bike apart inside was a bad idea, not only for the mess but because of my toddler running up and grabbing greasy parts). It’s a pretty standard Herrenrad made by Kreidler. First up I just had to remove the rear wheel and all unnecessary bits. This amounted to:

  • Removing the chain, which is made pretty easy by the derailleur.
  • Removing the rear derailleur which just requires one hex bolt to be taken out.
  • Removing the nuts and washers securing the rear wheel.
  • Loosening the rear brakes. There is a shortcut here which I didn’t realise before – you can just slide the “spaghetti” tube out of one side of the brakes to loosen them completely. I’d been loosening the cable instead all the previous times.
  • Cutting the unneeded shifter cable to the back derailleur and removing related garbage.
  • Removing the front derailleur, cabling etc as well since I was only going to use a single chain-ring up front after the modification.
  • Removing the actual rear wheel.

Here you can see the new wheel slotted into the dropouts. The three gear internal hub is surprisingly small compared to some of the seven and eight gear beasts out there. This is more or less all I had after I initially ordered parts. Actually, to be precise I had:

  • Rear wheel with internal gear hub.
  • Shifter mechanism, cable and attachment to hub
  • Chain tool

Shortly after these parts arrived I realised I was missing most of the rest of the rear wheel assembly, most noticeably the sprocket.

So I ordered that part as well from the same place, hoping that was all I needed. Not so. There are a bunch of other washers, rings, fasteners and the funny little pin that goes into the hub as part of the shifting mechanism which you can see sticking out in the photo above. Apparently there is a kit that contains all of these additional parts (Schaltungszubehör) but I could not source it easily here, nor did any of the shops I visited have it.

By the time I had taken the above picture I had received parts ordered from two stores on Ebay, some of the accessories from a bike shop on Prenzlauer Allee (after fruitlessly checking two other local shops) and a new tyre and inner tube from Stadler (an absolutely massive bike shop just next to Storkower Straße S-Bahn). I also had an incorrect accessories bag which suits an SRAM seven gear internal hub. Huzzah!

I’ve tried to take a frugal approach to this bike wherever possible, applying the “if it ain’t broke don’t fix it” mentality reasonably successfully. So I was slightly torn when I was getting the rear sprocket properly attached at bike shop #3 and the guy asked if I needed a chain. I initially said no, then decided I should get a new one in case the old one was too stretched. As you can see, the old one is completely the incorrect type for the new sprocket so it would have been a total failure and required yet another trip out for the chain.

A word of caution – don’t perform bike surgery over an open drain outside. My first snap lock chain link dropped out of my fingers and into the drain below almost immediately and I was forced to make a stop at bike shop #5 right before it closed on a Saturday (and none are open on Sundays). Aside from that, these links are invaluable if you need to mess about with your chain, while finding the best length. Take it slowly and don’t remove too many links at once! Test before taking out more, as it is almost impossible to reattach links using the regular tool.

Here are the bad boy front chainrings. The initial plan was to use the largest (which I believe was something like 42 teeth), although the angle from it to the rear sprocket was significant and I wasn’t interested in doing funny chainring surgery to reduce the angle.

After the initial chain shortening and connection, here is the first working example of the drive train post-modification.

The shifter was pretty easy to install and fit in reasonably well with the existing half-length handlebar grip, although it is a little short now. The exit point of the cable is also a bit irritating as it gets in the way of the bell. But it works.

Here is my ghetto shifter cable installation with synthetic string. Hey, it was only temporary.

Shortly after putting the bike back together, I took it for a test ride and the chain slipped off almost immediately after putting a bit of load on it. I assumed that the problem was the angle between the front chainring and rear sprocket so I moved the chain to the middle chainring and shortened it again. This seemed to fix the problem at least for just myself riding the bike at reasonable speeds and in all three gears.

Here is a different view of the rear hub and shifter mechanism attached. It just houses a small lever that is pushed and pulled by the shifter cable. In turn, it pushes on the pin that goes into the hub and causes the gear changes in three different pin positions (depths).

After taking the bike for a real ride with trailer attached, the chain got progressively looser and looser until it started skipping, and by only 5km I was getting rather nervous about it. I turned back but by 9km it jumped off and I had to coast home. The axle on the chain side had obviously moved out of the dropout slighltly, so I had either under-tightened the nuts or not had enough locking washers or nuts or both. I also suspected the front chainrings to be slightly warped (as the rear cassette had been before) which didn’t help.

At this stage I had not been able to ride for about two weeks so I had run out of patience. I took the bike to my local shop (literally 100m away) and asked them to fix whatever was wrong. They tightened up the axle and replaced the front chainrings with what you see pictured, a single 38 tooth ring and new crank arms. As a bonus whatever they did fixed the noisy bearings in bottom bracket and my pedals don’t wobble about as they used to.

The end result was extremely good. The combination of 38 teeth in front and 16 in rear give me a very good set of ratios between the three gears – 48.7, 66.5 and 90.7 gear inches respectively. Even with the trailer they work very well for the local terrain. I would not do it the same way again but I don’t regret anything about the upgrade.

For any budding bike DIYers out there I would suggest a few things if you are contemplating this:

  • Before you do anything, read Sheldon Brown’s excellent site.
  • Find the most well-stocked bike shop in your city and go there. Tell them what you want to do to your bike and just get them to supply all the required parts in one hit. It is much better than having to find out for yourself that you are missing one obscure part that nobody carries.
  • Get the right tools to do the job, and make sure they are high quality (this goes for just about anything mechanical).
  • Pester your friends with bicycle repair knowledge and mine it.
  • Allow plenty of time! If you will go crazy if you can’t ride for more than a couple of days, make doubly sure you can get it all done, or you have a rollback plan (or better yet, another bike).

If you have done similar upgrades, please leave a comment as I’d love to know what you did and how you got on.

Tags: , ,

Saturday, September 3rd, 2011 Germany 2 Comments