adrift on a cosmic ocean

Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We have lived in Berlin since 2009, have two kids, and have far too little time to really justify having a blog.

Efficiently handling the deluge of mail

Posted by Oliver on the 30th of August, 2011 in category Tech
Tagged with: exchangeimapmailmutttime

I took a week off work this week for an "Urlaub auf dem Balkon" - in other words a holiday at home. With a demanding one year old child, a bit of relaxation and time to ourselves is about as much as we can ask for at the moment since actually going away on holiday can turn into a bit of a stressful nightmare. Since I have the week to relax, naturally I've used the opportunity to get my geek on and scratch some minor technical itches.

Since I'm reasonably obsessive compulsive, I tend to have some real end goals that get swallowed up in a never-ending stack of things I encounter along the way to getting to those end goals. For example, I'm attempting to build some motion-detection-driven home systems and the way I'm getting to that is by refactoring the way I handle my work email. Straightforward, nicht wahr?

I've been a disciple of the church of Mutt for at least five years now, and can't imagine going back to any other email client. I briefly considered using Gnus but the need to use Emacs and holding seventeen meta-keys down for any given command just doesn't sit well with me (most of my Mutt shortcuts are single-key).

My email habits as of joining Nokia have changed slightly to what they were before, and what I imagine is the norm at most workplaces. I of course have a personal Gmail account, which nowadays I don't even bother to download from that frequently with POP3 but use the web interface and so enjoy having access to the same emails wherever I am. Since the volume of my personal emails is relatively low I don't need the speed of Mutt, but even if I do download the emails onto a computer so that I could use such an MUA, Google made the right decision in having all emails archived and still available. Not only that, but sent emails as well if you have configured your MTA to send via Google's authenticated SMTP servers. I don't worry too much about categorisation of emails due to Gmail's excellent search and tagging capabilities.

Work emails on the other hand are high in volume and desperately in need of categorisation. At my previous job I would typically only access work emails at work or at home using my work laptop. Now, I quite frequently need access to them on my various mobile devices as well, and it's not just because my employer is a mobile phone manufacturer:

  • Meetings where I don't want to bring my laptop (as much as screen is a godsend for persistent shell sessions, sometimes I just don't want to break my network connection)
  • Commuting between offices
  • Being on-call but not having my laptop out
  • Checking emails late at night or on weekends

All of these add to the need (or at least, desire) to check my emails on my phone. This last point may set off warning bells for you, but given that we work with many teams around the globe in different timezones, it is frequently in my best interest if I can fire off a late-night missive to a network technician just before I go to bed, to cut out a globe-rotation from my wait time for some requests. It's just life in a multi-national organisation.

How does this fit in with my previous set up? Not particularly well. I have a three-component combo in the form of fetchmail, procmail and the aforementioned mutt. If I were only retrieving to my local machine and not caring about anything else this would give me reliable retrieval, powerful and flexible filtering, and a speedy and efficient MUA. I've actually been operating like this for the last almost-two-years, but a few problems are starting to become rather bothersome:

  • While my laptop is on the office network or VPN from outside of the office, all new emails are deleted from the mail servers every minute meaning my phone generally never sees them. Unfortunately fetchmail has no way to figure out which emails to download if you are leaving them on the server, but you have already read them e.g. on your phone or through a web interface. So, I just download them all and delete them.
  • I can't use Mutt's built-in IMAP client with our mail servers since there are bugs when using it with Exchange.
  • The same bugs cause problems even if you proxy through Dovecot (which actually makes perfect sense, only the authentication part of the session is really modified).
  • Any emails that happen to stay on the mail server (e.g. when my laptop is not running) don't get the awesomeness of my procmail rules applied to them, so whenever a service dies and spams me with two thousand emails that I would usually consign to a low-priority folder (and perhaps even mark automatically as read), my phone suffers the full force of this load of email.

A former co-worker blogged about a solution that he found to this problem but it doesn't quite cover the full gamut of what I wanted to achieve. Imapfilter is a cool piece of software, but it still needs you to run it on something (and that something sure as hell won't be your company's Exchange servers). So I checked out the Exchange server-side filters, and to be honest they are not too bad. Not nearly as powerful as procmail's regular expression matching and options, but fortunately most of my email is relatively easy to filter based on simple text matching.

So now I have the following setup:

  • Exchange server-side filters categorise emails into the correct folders before they even hit a client.
  • Offlineimap synchronises my email between the Exchange servers and local Maildirs.
  • Mutt is still my MUA

Unfortunately before I got to a fully working end-state, I had almost two years of emails that I had already downloaded and stored locally on my laptop that needed to be uploaded again to the IMAP server. OK, perhaps they didn't need to be there, but by virtue of having them in my local Maildirs, Offlineimap will want to sync them up anyway. Having ~2GB and 50k odd emails (and that's after a bit of archiving) lying around may not be the most useful thing on the planet but far too often I find that I need to go back and search for some archived email that I need some information from. So yes, I need them there.

An important safety tip on this item is that you need to have a low latency to your IMAP server. Offlineimap is multi-threaded but can only use a single thread for syncing with one remote mailserver (at least in the version I am using) and hence each folder is synced serially and email by email. With even latencies of ~120ms over my VPN the syncing was crawling along. I also noticed that once you get over a few thousand emails in a folder, the sync rate slows down further and further. I'm not sure if this is a problem at the local end or in Exchange but it is a safe bet to say it is Exchange.

At this point I feel like I have as best as possible solved my problems. Since emails are fairly well-categorised before they reach a device there are minimal emails in my mobile inbox which helps the device's synchronisation speed. I have an identical representation of my emails on my laptop, my mobile devices and the Outlook Web App, regardless of which one is running or which one I'm using. About the only thing I feel I lack at this point is a really, really good search on my emails both on the Web App and in Mutt (I know it's just not feasible on my phone). OWA just doesn't seem to find all the emails that it should, and Mutt has no recursive search (although Mairix looks interesting). That combined with intelligently retrieving the entire thread involved in a search result are big features for which I doubt I'll find very good alternatives outside of Gmail.

The only other thing I realise I'm now lacking is automatic addition of mail senders to my Mutt aliases list (or with abook) which is now not possible since I've removed procmail from my delivery chain. I'll scratch that itch another day!

© 2010-2018 Oliver Hookins and Angela Collins