Archive for August, 2015

Pre-warming Memcache for fun and profit

by Oliver on Wednesday, August 12th, 2015.

One of the services my team runs in AWS makes good use of Memcached (via the ElastiCache product). I say “good” use as we manage to achieve a hit rate of something like 98% most of the time, although now I realise that it comes at a significant cost – when this cache is removed, it takes a significant toll on the application. Unlike other applications that traditionally cache the results of MySQL queries, this particular application stores GOB-encoded binary metadata, but what the application does is outside the scope of this post. When the cached entries aren’t there, the application has to do a reasonable amount of work to regenerate it and store it back.

Recently I observed that when one of our ElastiCache nodes are restarted (which can happen for maintenance, or due to system failure) we already saw a less desirable hit to the application. We could minimise this impact by having more instances in the cluster with less capacity each – for the same overall cluster capacity. Thus, going from say 3 nodes where we lose 33% of our cache capacity to 8 nodes where we would lose 12.5% of our cache capacity is a far better situation. I also realised we could upgrade to the latest generation of cache nodes, which sweetens the deal.

The problem that arises is: how can I cycle out the ElastiCache cluster with minimal impact to the application and user experience? To save a long story here, I’ll tell you that there’s no way to change individual nodes in a cluster to a different type, and if you maintain your configuration in CloudFormation and change the instance type there, you’ll destroy the entire cluster and recreate it again – losing your cache in the process (in fact you’ll be without any cache for a short period of time). I decided to create a new CloudFormation stack altogether, pre-warm the cache and bring it into operation gently.

How can you pre-warm the cache? Ideally, you could dump the entire contents and simply insert it into the new cluster (much like MySQL dumps or backups), but with Memcached this is impossible. There is the stats cachedump command to Memcached, which is capable of dumping out the first 2MB of keys of a given slab. If you’re not aware of how Memcached stores its data, it breaks the memory allocation into various “slabs” of increasing sizes and stores values in the closest-sized slab that will fit it (although always rounding up). Thus, internally the data is segmented. You can list stats for all of the current slabs with stats slabs, then perform a dump of the keys with stats cachedump {slab} {limit}.

There are a couple of problems with this. One is the aforementioned 2MB limit on the returned data, which in my case did in fact limit how useful this approach was. Some slabs had several hundred thousand objects and I was not able to retrieve nearly the whole keyspace. Secondly, the developer community around Memcached is opposed to the continued lifetime of this command, and it may be removed in future (perhaps it already is, I’m not sure, but at least it still exists in 1.4.14 which I’m using) – I’m sure they have good reasons for it. I was also concerned that using the command would lock internal data structures and cause operational issues for the application accessing the server.

You can see the not-so-reassuring function comment here describing the locking characteristics of this operation. Sure enough, the critical section is properly locked with pthread_mutex_lock on the LRU lock for the slab, which I assumed meant that only cache evictions would be affected by taking this lock. Based on some tests (and common sense) I suspect that it is an LRU lock in name only, and more generally locks the data structure in the case of writes (although it does record cache access stats somewhere as well, perhaps in another structure). In any case as mentioned before, I was able to retrieve only a small amount of the total keyspace from my cluster, so as well as being a dangerous exercise, using the stats cachedump command was not useful for my original purpose.

Later in the day I decided to instead retrieve the Elastic LoadBalancer logs from the last few days, run awk over them to extract the request path (for some requests that would trigger a cache fill) and simply make the same requests to the new cluster. This is more effort up-front since the ELB logs can be quite large, and unfortunately are not compressed, but fortunately awk is very fast. The second part to this approach (or any for that matter) is using Vegeta to “attack” your new cluster of machines, replaying the previous requests that you’ve pulled from the ELB logs.

A more adventurous approach might be to use Elastic MapReduce to parse the logs, pull out the request paths and using the streaming API to call an external script that will make the HTTP request to the ELB. That way you could quite nicely farm out work of making a large number of parallel requests from a much larger time period in order to more thoroughly pre-warm that cache with historical requests. Or poll your log store frequently and replay ELB requests to the new cluster with just a short delay after they happen on your primary cluster. If you attempt either of these and enjoy some success, let me know!

Tags: , , , ,

Wednesday, August 12th, 2015 Tech No Comments

Running and energy gels

by Oliver on Monday, August 10th, 2015.

I’ve been taking part in longer-distance events of various kinds (running, obstacle racing, cycling and most distantly kayaking) for quite some time now and have had various nutritional techniques over the years and for the different events. Something that has been more of a staple for me personally in cycling and running is energy gels, and if you’ve used them yourself you’ll know that they can be a hit-and-miss affair; often the choice is very personal. What works for one person may not work for another.

That being said, at the last couple of events in which I participated, I think I did a particularly poor job at choosing my gels. At the point of consumption it was a matter of purely getting the calories into my body, but it wasn’t particularly pleasant. I’m not a picky person when it comes to food and drink, but I am obsessed with details and quite thorough so I decided to try as many different gels as I could get my hands on and record my thoughts on them. Again, choice can be a personal thing but perhaps it will be interesting to others, and also handy for my own memory’s sake.

A couple of notes before the first review: I’m consuming these after at least ~45 minutes of reasonable exercise so they are somewhat realistic impressions as I’d expect “in the field”. Of course, if you are low on energy and haven’t had anything in a while, generally you’ll be ravenous and slurp down anything, but assuming you time everything correctly you’ll want something that is reasonably palatable.

Clif Shot Energy Gel – Razz flavour

96 calories from a compact, easy to carry and open 34 gram packet. It also has a “litter leash” on the side of the packet which I haven’t worked out how to use – I just roll it back up and put it in my pocket. I think I ate one of these at home late at night a week or so ago, in one of my more desperate fits of midnight-snacking, and remember the taste wasn’t that great, but after running for 10km it was actually not bad. Not sure if it was the fact that it was around 0°C outside, but it seemed very thick – probably too thick for my liking.

Ideally I would not have to follow this up immediately with water to get it down, but it was just so thick I felt compelled to drink afterwards. The flavour was ok – not great initially but by the time I’d finished it, I wasn’t too unimpressed. Would use again if there was nothing better. Raspberry, in that kind of synthetic-tasting raspberry you get in lollies, mixed with a bit of saltiness. But I guess that’s quite predictable – most gels are going to taste synthetic, sickly sweet and salty all at the same time.

Somewhat interesting note: the packaging has a small paper sticker with the word “shot” on it, that covers “90% organic” on the plastic packaging. I wonder if the formula is different in Germany, or if it is simply bullshit that wasn’t allowed to be advertised here (unlike perhaps in other markets).

XenoFit Carbohydrate – Redberry Gel

68 calories from one 25g “stick” (in reality a tear-to-open soft packet like all of the others). Noticeably smaller than the others, and when I opened it I didn’t taste anything initially – although I suspect this was due to my taste buds being slightly frozen (I was running in about 3ºC). The flavour, when it came through, was pretty similar to the Clif Shot but perhaps not quite as sweet – which is good.

Quite thick consistency again, similar to the Clif Shot. Again, this could be due to the temperature. Overall ok, would use again but it didn’t blow me away. The relatively small amount of energy in one packet might be an issue for some – I would probably want more in one packet to avoid having to continually open them to stay energised.

Nutrixxion Energy Gel “Force” flavour/variety

40g easy-open sachet with 80mg caffeine, 500mg taurine, 118 calories.
This one really tasted quite foul. It doesn’t list the flavour on the packet and I couldn’t identify what it might have been from the flavour of it – generically sweet and goopy texture. I guess it is not trying to be user friendly but has the advantage of a shot of caffeine to redeem itself with. Not sure that is enough for me. Wouldn’t use this one again.

PowerBar L-Carnitine liquid drinking ampoule (before) and Amino Mega Liquid drinking ampoule (after)

25mL each.
Technically not energy gels, these were in the same area of the shop and caught my eye. They are intended as some kind of vaguely snake-oily type of muscle enhancer through combining their consumption with exercise. The L-Carnitine ampoule was pleasant enough tasting, the liquid being quite liquidy and having a nice flavour (although somewhat unidentifiable) but outside of the main ingredient it has practically no energy content at all. The Amino Mega Liquid does contain 52 calories and a host of strange chemicals, but also a rather horrid flavour. For those reasons I couldn’t consider these worthwhile for energy during exercise, and probably wouldn’t use them for their supposed muscle building or regeneration effects either.

MultiPower MultiCarbo Gel, Lemon flavour

40g easy-open sachet with 104 calories.
Very similar to many of the others I’ve previously written about, thickish consistency and sickly-sweet lemon flavour that wasn’t terribly unpleasant or pleasant. Again, wouldn’t avoid it if there were no other options, but also wouldn’t pick it necessarily over other available gels.

Sponser Sport Food Liquid Energy BCAA, strawberry-banana flavour

70g “toothpaste tube” style container with 173 calories.
Contains 500mg of “BCAA” (branched-chain amino acid), and also calls out (on the front of the tube) the presence of additional sodium, potassium and taurine. Whether or not those elements of the liquid energy actually helped me or not, I found two things about this one detrimental – the container and the flavour. The very artificial-tasting strawberry-banana combination is not exactly revolting, but it’s not great. Why would you combine these flavours?!? Secondly, the tube format with the cap was far more difficult to consume from than other gels, and this was just when I was running. They recommend that you consume half the tube every 20 minutes, combined with 200mL of water on each “dose”, which doubles the difficulty. I couldn’t possibly recommend this, and would avoid it unless I literally had no other options.

Xenofit Carbohydrate for sport, citrus-mix gel

25g sachet with 68 calories.
This one surprised me twice. Firstly the flavour, citrus (although basically lemon) was actually quite pleasant although the texture is more pasty/floury than most gels, which was not expected. When I first started swallowing the gel I was thinking “I’m not sure I like this texture”, but after just a few seconds this changed to “actually I quite like this”. It is definitely a bit on the thicker side and takes a little effort getting it out of the small sachet, but not thick like some of the other gels are that just makes them unpalatable. Incidentally, Xenofit is also the brand of sports drink powder I have. The main downside to these gels is their smaller size, and so you don’t get as much out of each sachet as some of the others. Would definitely buy again.

Dextro Energy Liquid Gel, Apple flavour

60mL packet with screw-off lid, 114 calories.
I was saving this one for last but couldn’t help myself and took it on my run today. I’m fairly certain this is my favourite of the whole lot of gels (or possibly one of the other flavours of the same product). The consistency is much more fluid than other gels, in fact it’s almost like drinking a very sweet shot of juice or cordial. They are quite sweet but not sickly sweet, and the flavour, while somewhat artificial (like most other gels) is not too bad. This combination makes these my favourite energy shot. The only thing against them is the fairly impractical screw-off top of the packet. It’s ok if you are running, but quite fiddly if you are on a bike.

BSc BodyScience Athlete Series Energy Gel, Super Berry flavour

35g easy-tear sachet, 97 calories.
Unfortunately I left my big bag of gels at home, forgetting to take them with me on vacation, but this gave the opportunity to pick up something new in a different country. I haven’t seen this particular brand before but was interested to try it. The consistency was quite thick, even compared to others I’ve already tried, and the temperature is mid-20s Celsius at the moment so in colder temperatures it might be difficult to consume! That being said, it was not quite as sickly sweet as other gels that are on the thicker side, although the “super berry” flavour is not as great as it would have you believe. I didn’t have any water immediately available to me, so it was good that there wasn’t a terrible aftertaste. Quite convenient size and easily tearable on a bike without slowing down much – would probably buy again over most of the other gels I’ve tried.

PowerGel Fruit Dual Source Carb Mix Gel, Mango Passionfruit flavour

41g easy-tear sachet, 108 calories.
I’ll admit up front that PowerGels are one of my least favourite tasting gels, and I’ve used them in several races before when nothing else was available. That being said, the “fruit” variety wasn’t nearly as bad as the non-fruit flavours I’ve had before. Almost pleasant, if I dare go so far. Still, the easy-tear sachet was anything but – unless I made a terrible mistake, the consistency of the gel made it practically impossible to get out of the sachet. If I’d been on a bike it would have been a disaster, but even running it made things tricky. Luckily I had to stop at some traffic lights and wrestled with the packet for a minute. Would possibly buy again if I could be sure the sachet would open properly.

MultiPower MultiCarbo Gel, Orange flavour

40g easy-open sachet with 104 calories.
Proving once again that enjoyment heavily depends on how tired and in need of a psychological energy boost you are, I didn’t mind consuming this gel – despite having had one before in lemon flavour and not really enjoying it. Given that the lemon flavoured gel of this kind was consumed quite a while ago I think it must have been in one of the colder months, and I can definitely confirm that on a slightly warmer day the consistency was not as thick. The flavour was reasonable, not quite as sickly sweet as the lemon (somehow, despite the orange still being quite artificial tasting).

I’d be tempted to have this one again – but only in orange flavour. It’s definitely one of the more handy sizes and packagings to carry around for use on a run or ride without much effort in consuming.

Nutrixxion Energy Gel, Cola-Lemon flavour

40g easy-open sachet with 40mg caffeine, 118 calories.
Contrary to the “Force” flavour of the same brand and type, I have a soft spot for Cola-Lemon flavour in general and so I kinda enjoyed this gel. The consistency was still quite goopy, even on a hot evening after sitting in my pocket. I’d already run out of water so it was not the best experience, but still not too bad. On flavour alone I’d probably use this again and consider it on par with everything but my most favourite gel (currently the Dextro Energy Liquid).

PowerGel “Original” Dual Source Carb Mix Gel, Vanilla flavour

41g easy-tear sachet, 107 calories.
I had low expectations for this one as I tend to prefer the fruity or cola flavours of gels, but actually was surprised that I preferred this one over the other flavours of PowerGels that I’ve tried in the past and generally disliked. I could actually stomach it, and the consistency was not bad. However, unless I’m making a terrible mistake every single time I open one of these sachets, there is something about the consistency of the gel and the opening at the top of the packet that makes it very difficult to get anything out of it easily. Smaller sachets from other brands don’t seem to be as problematic.

In any case, if I remembered to be more careful opening it, I would probably try this one again.

Xenofit Carbohydrate, Maracuja (Passionfruit) flavour

60mL sachet, 103 calories.
Unlike the smaller gels in the same brand, this one was quite a bit bigger. The reason became evident when I opened it (and also hinted at by the calorie content) – it’s much less dense and “gel-like”. Almost (but not quite) liquid. At the other end of the scale is the Dextro Energy Liquid which is very liquid, but this one was about half-way in-between. Perhaps as as result, the gel itself was not as sweet (and I was anticipating some fairly awful synthetic-tasting passionfruit flavour) and not overpowering. I wouldn’t say it particularly tasted like actual passionfruit, but that wasn’t a problem.

Might try again, although like many of the other rip-open sachets it proved tricky to actually get it open without squirting the contents all over my hand.

Sponser Liquid Energy Long, Salty flavour

40g sachet, 94 calories.
Knowing that the flavour was already described as “salty”, I was already dreading this one somewhat. This gel’s advantage is that it has 180mg of sodium to counteract losing sodium during exercise through respiration. Hence, most exercise drinks also contain a reasonable amount of sodium. In practice though, it was the consistency that was the killer for me. Despite it being a quite hot day (perhaps 30ºC in the shade when I was running) the gel did not soften up much, and felt like a slug sitting in my mouth, it was so thick and unyielding. It was quite a battle to swallow the first mouthful, and continue through the rest of the meager 40g.

To summarise: the flavour is not actually noticeably saltier than other gels or exercise drinks but the consistency is way too thick to be pleasant. I would not try this one again, even if it were the only option.

Xenofit Carbohydrate, Orange flavour

60mL sachet, 103 calories.
This was a sweet (literally) relief after the salty, thick gel I had consumed about an hour before. Not much more to say about the orange flavour that I didn’t already say about the passionfruit variety. The warm weather made this gel seem even more liquid-like compared to other gels, and it was easier to consume since I managed to tear the sachet properly this time. It was a bit more synthetic tasting than the passionfruit.

The only thing not in its favour is the size, and at 60mL (rather than perhaps 40mL/40g) it doesn’t fit so well in my pockets, but that’s probably more to do with my running kit than this gel. Would try again if my more favoured options are not available.

Dextro Energy Liquid Gel, Cola flavour

60mL packet with screw-off lid, 114 calories.
Just another flavour of the same one I’ve tried before (and wrote about above). I only just realised it was cola flavour just now, when I started to write about it – it wasn’t obvious to me while I was out for my run, and I assumed it was some kind of very synthetic fruit flavour. In any case, as before, this is definitely my preferred energy source while exercising. The only thing not in its favour for me personally is the additional size due to it being a liquid and not a more-dense gel, and the screw-off top (although if you feel like having just half of it now and half later, it is useful).

So there you have it.
It took something like 8 months to actually get through all of these gels since I wasn’t doing a lot of long distances to warrant it until recently. To sum up, my preferred gel is the Dextro Energy Liquid Gel (if that wasn’t already obvious), but since the amount of energy you get from them isn’t as much as others, I’d probably supplement that with a banana or energy bars. However, before you base your own selection on my reviews here, I’d recommend trying them for yourself – it’s a very personal choice and everyone will have their own preferences.

Tags: ,

Monday, August 10th, 2015 Health 2 Comments