adrift on a cosmic ocean

Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We have lived in Berlin since 2009, have two kids, and have far too little time to really justify having a blog.

Migrating the blog to Gatsby/Contentful/Netlify

Posted by Oliver on the 3rd of June, 2018 in category Tech
Tagged with: yaksweb developmentprogrammingfrontendblogcontentfulnetlifygatsbyreact

If you have previously read anything on this blog (directly, rather than via RSS) you will have noticed that the visuals have been updated. This is the culmination of a very long and arduous process - entirely my responsibility and self-inflicted - and as a result, it is running via Netlify, built with Gatsby and stored in Contentful. For styling I relied heavily on Tachyons and for the sake of completeness, used a header image from Unsplash.

If you didn't see it before, this is what it looked like:

Screen Shot 2018-06-03 at 18.40.56

It was hosted on Wordpress since about 2010 when I started blogging, initially on a former employer's shared hosting service, and then later I moved it to a VPS I was managing myself. It has automatically upgraded through the various versions of Wordpress as they became available, and used the Decoder theme, which felt modern when I chose it but is really starting to look dated now.

The work itself occurred in several stages: overcoming my fears of Javascript, CSS and web development in general; migrating the backend content from Wordpress to Contentful, and building the front-end itself. I'll describe each of these in sections below.

Web Development and Javascript

Outside of the fact that for the last few years I've stepped away from being a direct technical contributor and have been managing engineers instead, I don't believe I've ever felt comfortable with front-end web development. My origins in my field stem from computer/networks support, operations and what today we might call DevOps or SRE. I don't think I touched any Javascript at all until literally 2011, and even then it was just some simple form field validation.

I wrote a very simple Node application and corresponding front-end code for an interview challenge in 2012, but wouldn't say I had a great grasp on what it was doing. Somewhere in 2014 I had built a very simple website for my wife's yoga website (using a now-deprecated framework called "Famous") but I still came away from the experience not really "getting" it. I was exposed to a lot more general Javascript app development as part of my team's work on web audio streaming at SoundCloud, but it was only once I came to Contentful that it rose in importance - Javascript and Ruby are basically the only two languages in use (with Go a very remote third place).

In order to understand the challenges the engineers were experiencing, I went through a few online courses, completed all of the exercises on ES6Katas, and built a few small dashboards. These initially were very horribly coded - creating DOM elements manually and populating the page piece by piece. Jan pointed me in the direction of Flexbox Froggy which helped immensely in the styling of some of my build status dashboards and generally helped me understand how styling for the web works. We are really spoiled today with built-in development tools in the browser and dynamic editing of the DOM!

More recently, I learned React since it seemed to be gaining traction (and we've selected it as our future standard framework for our web app at Contentful), and I dutifully migrated the various dashboards I'd created over to that. It drastically simplified the code and made it more modular (and hopefully more maintainable, should anyone be so adventurous). I had made a somewhat false start on migrating the blog off Wordpress towards the end of 2016, but it had involved a Node service retrieving content live from the Contentful API and server-side rendering with Jade (now Pug) templates.

After watching Khaled's excellent four part tutorial on Gatsby I resolved to finally get it done.

Migrating The Data

Back in 2016 I made a first attempt, using the tooling that was around at the time which was the wordpress-exporter. Wordpress itself facilitates data exporting by letting you download an XML-encoded dump of the database. The format is pretty horrible, and the result in my Contentful space looked like a 1:1 facsimile of Wordpress's data structure, which is fairly generic and one-size-fits-all. I did one initial import of the data into a space and started developing the previously mentioned Node service, and built a couple of simple views. It felt a bit difficult and some other priorities popped up so that effort got shelved until earlier this year.

In the meantime my writing output dropped off significantly, partially due to working in management rather than as an individual contributor, and also due to lack of time. The couple of blog posts I decided to make ended up going still in Wordpress which felt like continuing a bad habit. This year I started thinking of some new topics to write about but felt that I didn't want to put them into the Wordpress site - hence jumping back into the data migration. I pulled Gatsby into the old codebase before realising I wanted a more understandable content model to work with. Content Modelling is a pretty key concept in our product, so not only did I want to do it the right way but also wrap my head around exactly how my content would fit together.

I decided it would be fun to use our very alpha Go SDK to write a data migration tool, and the resulting code is here on Github. The SDK didn't support some key features like adding entries, specifying your own IDs for some entities like Content Types, unpublishing assets etc. I ended up making my own fork of the SDK and adding the functionality I needed (although it needs some work to be merged upstream). My workflow was basically this:

  • Delete all entries and content types from Contentful space via the Content Management API (CMA).
  • Add code to parse another entity from the Wordpress XML dump.
  • Add the Contentful content type to match.
  • Iterate through entries and add them via the CMA.
  • Check the result of the import in the Contentful editing interface.

I went through many cycles like this, deleting everything from the space, adding everything back, checking the results in the web app, tweaking something and repeating. Everything was done serially so some operations like adding/deleting tags took a while. Later I added code to re-write URLs that were inside of the post body data to relative URLs (especially for links between blog posts that I use a lot). I also used this "rewrite table" to generate a redirects file for Netlify to use - I am now using "slugs" in the post URLs whereas the old Wordpress site used a query parameter to refer to the post ID. Old links from search engines or RSS feeds will redirect to the new location of the post in question.

Another interesting aspect of the migration is that usually people are interested in converting from Markdown to HTML, not the other way around. Pandoc solves that problem, being able to convert between any two given formats, from a rich list of options. My migration tool just executes another process for pandoc on temporary files, reads the output in again and loads the result into the CMA as an entry (after rewriting URLs and removing other cruft in the output).

Last week, once I was fairly happy with the results, I did one last final import of all of the content and today I've spent several hours combing through each post checking for broken links and other manual fixes that the tool wasn't easily able to do itself. Hopefully the job is done now and I won't have to reimport again, but if so, I have 90% of the automation encoded. Perhaps someone else can use it as well.

Developing the front-end

As mentioned above, I made an initial start with Gatsby in the same repository in which I had developed the Node app, but it became quickly evident that I was not going to want (or need) the backend component, and I also needed more of the content structure and entries available to continue building out the website. I used some placeholder content to start building the React components and overall layout. Had this been perhaps two or more years ago I'd have been tempted to use Bootstrap for much of the layout and styling (in fact I've used this in the past for some dashboards and simple front-ends to infrastructure systems) but I'd heard of Tachyons as a lighter-weight alternative to Bootstrap so I decided to give that a try. I started out styling components myself, and gradually moved styles to standard classes already present in Tachyons. There are a couple of things missing which I styled "manually" - text shadows (not box shadows) and some fine tuning of padding/margins. I feel I actually have a pretty good grasp on these elements now, which is nice after such a long time.

The GraphQL querying of the Contentful API was a little tricky to work with as I'd not worked with GraphQL before, but Khaled's tutorial series referenced above helped immensely with this (as well as some personal assistance from him answering my questions 😍). I also borrowed his strategy on pagination of the index page. Font and colour selection are two aspects I have not trained my eye enough for (probably), but I can live with my choices! Similarly, I'm not a "pixel perfect" styler, but prefer to ensure there are no obvious misalignments or visual gaps.

Since it's a static site there isn't that much extra Javascript required actually - most of the effort went into building React components, tieing them together and styling everything. These are historically major weaknesses of mine in software development, so I'm pretty happy about that. I would take on more front-end work in the future, should the need arise - I think that's a dramatic improvement from the past where I would actively avoid it.

The other dramatic improvement in this area was in hosting. Netlify has been gaining a lot of traction (we use it for our main website at Contentful, and it received a mention recently on the Thoughtworks Tech Radar. Deploying a Gatsby site to Netlify is even simpler than deploying my old Node app to Heroku, and is what seems like a perfect fit for a site like this. Let's Encrypt certificates are supported (and configured/renewed automatically), DNS management is also part of the offering and integrates seamlessly, and I was able to hook it up to the development pipeline by both linking to the Github repository and sending webhook notifications from Contentful on entry publishing. Amazingly, I think I don't even have very extensive use of their features, which go far beyond this. The redirects file as mentioned above is also great for not losing clicks to old URLs from the Wordpress site.

What Was Not Implemented

I was conscious that I didn't want to re-implement all of the functionality of the Wordpress site, to keep the work involved to a minimum. The admin side is taken care of by the Contentful webapp, but outside of that, everything is up for grabs. Here is the list of what didn't make the cut:

  • Comments - I'd migrated these already to Disqus, but I haven't bothered to integrate that yet.
  • RSS feeds - I was worried about how existing RSS subscribers would see the posts and what I'd need to do to ensure they don't get every post from the beginning again. Gatsby has an RSS plugin, in fact, but for the moment I am avoiding this topic entirely (at the cost of perhaps some small amount of readership).
  • Categories - These were actually imported and I link a category from each post, but they are only printed as part of the post metadata and not available as a way to query for matching posts. I might add this functionality in future.
  • Tags - Also imported and used, but only for additional post metadata. You can't filter posts by tag (at least on the site).
  • Responsive or Mobile Design - This is still something of a mystery to me. The dashboards I've built at work are woefully un-responsive, and look very different depending on your screen resolution. Tachyons has some kind of responsive web support so this might come in the future.
  • Grid Layout - This has been gaining a lot of traction over the last 6-12 months. I don't believe I need it very badly, but it's a clear successor to Flexbox and a worthwhile addition to my web knowledge.
  • Monthly archives, tag cloud, search, page numbers, etc... - Many other things I just didn't want to implement (yet) and have unknown or little value at the moment. Best to validate that what I've done is good, and iterate where necessary.

Given that the "project" in a sense should have been done in the early months of me starting at Contentful, it is way overdue, and needed to be kept lean. Obviously some knowledge pre-requisites stood in the way, but I'm glad I could stay relatively strict with myself about feature-set and get it completed quickly once I'd set my mind to it.

Conclusion

Not much else to say here except that the combination of Contentful, Gatsby and Netlify is a good one and I can recommend it. Styling and front-end development is not to be feared, and can be mastered at least to the point where one can build a content-driven site such as this. React and similar frameworks make this a lot easier, I have to say.

I hope with this long-overdue site migration I can get back to writing a few more technical posts, hopefully at a rate of slightly more than one a year! Any comments, questions or feedback - you'll have to direct to me on Twitter for the moment - I'm happy to chat more about my experiences or answer any questions if you want to go about a similar migration or site build.

Thanks for reading.

© 2010-2018 Oliver Hookins and Angela Collins