Continuous Integration/Testing/Delivery/Deployment

Manoj Kumar K asked me via LinkedIn for some feedback on his article about the differences between Continuous Integration/Testing/Delivery and Deployment.

Here’s my response (left as a comment on the original article):

Continuous Integration is continually integrating several developer’s code, compiling, deploying and automatically testing that code (Continuous Testing), and that can be done on a branch, or in trunk/master.

When you have continual production readiness of every build, but you don’t deploy every build automatically, instead you choose to deploy say x times a week, that is Continuous Delivery.

When you automatically deploy every build to production after it’s been automatically integrated, built and tested, that is Continuous Deployment.

I’ve previously briefly explained the semantics here.

So Continuous Integration (CI) and Continuous Testing together allow Continuous Delivery. Continuous Delivery allows Continuous Deployment (should you choose to do this).

Martin Fowler explains this in more detail here.

 

 

Test in production?

This post is part of the Pride & Paradev series.


With continuous deployment, it is common to release new software into production multiple times a day. A regression test suite, no matter how well designed, may still take over 10 minutes to run, which can lead to bottlenecks in releasing changes to production.

So, do you even need to test before going live? Why not just test changes in production?


Test changes in production

The website for The Guardian, the UK’s third largest newspaper, deploys on average 11 times a day, of which all changes are tested in production.

“Once the code is in production, QA can really start.”

“Sometimes deployments go wrong. We expect that; and we accept it, because people (and machines) go wrong. But the key to dealing with these kind of mistakes is not to lock down the process or extend the breadth, depth and length of regression tests. The solution is to enable people to fix their mistakes quickly, learn, and get back to creating value as soon as possible.”

~ Andy Hume on Real-time QA [The Guardian Developer Blog]

The key to testing changes as soon as they hit production is to have real time, continuous real user experience monitoring. This includes metrics like page views and page load time, which directly correlate to advertising revenue, an incentive to keep these healthy.

More comprehensive automated acceptance tests can be written in a non-destructive style that means they can be run in production. This means that these can be run immediately following a fresh production deployment, and as feedback about the tests is received, any issues can be remedied immediately into production and tested again.

Test changes before production

There are not many businesses that are able to release software without any form of testing into production: whether there be legislative requirements requiring testing, or the risk of introducing errors is too high for its target market.

Whilst automated regression tests do take longer to run than unit or integration tests, there are ways to manage these to ensure the quickest path into production. These strategies include running tests in parallel, only running business critical tests, only running against the single most popular browser, or only running tests that are directly related to your changes.

You can set up a deployment pipeline that runs a selected subset of tests before deploying into production then running the remaining tests (in a test environment). Any of the issues found in subsequent tests are judged to see whether they warrant another immediate release or whether they can be included in the next set of changes being deployed into production.

Whilst you definitely should run tests before deploying to production, it doesn’t mean that this has to drastically hinder your ability to continuously deploy.

SSD: super super drives

If your development machines don’t have solid state drives (SSDs): go buy some.

If your continous integration (build) box doesn’t have a solid state drive: go buy one.

Solid state drives are the quickest and cheapest way to speed software development. As most continuous integrations/builds are IO, solid state drives provide tremendous speed increases: I’ve seen builds and tests run in less than a fifth of the time a traditional moving hard disk would take. Solid state drives have reached a price point where the lower capacity ones are on par with higher capacity hard disks (which you don’t need unless you’re storing movies etc.)

It used to be that the build machine would use some crappy hardware lying around: often an old development machine someone no longer used. This is one of the worst decisions you can make on a project.

Having a super fast SSD based build machine will ensure fast, reliable build results, fast feedback and a happy development team.

Automated WCAG 2.0 accessibility testing in the build pipeline

The web application I am currently working on must meet accessibility standards: WCAG 2.0 AA to be precise. WCAG 2.0 is a collection of guidelines that ensure your site is accessible for people with disabilities.

An example of poor accessibility design is missing an alt tag on an image, or not specifying a language of a document, eg:

<HTML lang="fr">

Building Accessibility In

Whilst later we’ll do doing accessibility assessments with people who are blind or have low vision, we need to make sure we build accessibility in. To do this, we need automated accessibility tests as part of our continuous integration build pipeline. My challenge this week was to do just this.

Automated Accessibility Tests

First I needed to find a tool to validate against the spec. We’re developing the web application locally so we’ll need to run it locally. There’s a tool called TotalValidator which offers a command line tool, the only downside is the licensing cost, as to use the command line version you need to buy at least 6 copies of the tool at £25 per copy, approximately US$240 in total. There’s no trial of command line tool unless you buy at least one copy of the pro tool at £25. I didn’t want to spend money on something that might not work so I kept looking for alternatives.

There are two sites I found that validate accessibility by URL or supplied HTML code: AChecker and WAVE.

AChecker: this tool which works really well. It even supplies an REST API, but I couldn’t find a way to call the API to validate HTML code (instead of by URL) which is what I would like to do. The software behind AChecker is open source (PHP) so you can actually install your own version behind your firewall if you wish.

WAVE: a new tool recently released by WebAim: Web Accessibility in Mind. Again this is an online checker that allows you to validate by URL or HTML code supplied, but unfortunately there’s no API (yet) and the results aren’t as easy to programatically read.

My Solution

The final solution I came up with is a specific tagged web accessibility feature in our acceptance tests project. This has scenarios that use WebDriver to navigate through our application capturing the HTML source code from each page visited. Finally, it visits the AChecker tool online and validates each piece of HTML source code it collected and fails the build if any accessibility problems are found.

AChecker Results

Build Light

We have a specific build light that runs all the automated acceptance tests and accessibility tests. If any of these fail, the light goes red.

Build Status Lights

It’s much better if it looks like this:

Build Lights All Green

Summary

It was fairly easy to use an existing free online accessibility checker to validate all HTML code in your local application, and make this status highly visible to the development team. By building accessibility in, we’re reducing the expected number of issues when we conduct more formal accessibility testing.

Bonus Points: faster accessibility feedback

Ideally, as a page is being developed, the developer/tester should be able to check accessibility (rather than waiting for the build to fail). The easiest way I have found behind a firewall is to use the WAVE Firefox extension, which displays errors as you use your site. Fantastic!

(Apple and Microsoft each have one known accessibility problem, Google has nine!)

Apple.com accessibility

Continuous delivery and ruining the user experience

“I was chatting with a random Canadian woman. She found out I worked for Mozilla and I got that sinking feeling, like “here we go again” because I knew exactly what was coming up. She proceeded to tell me the story of how she switched to Chrome because Firefox kept breaking her extensions and asking her to restart.

I’ve heard this story a lot in the last year.

I used to be proud to say I worked (or had worked) for Mozilla, but a careful listener might detect a certain sheepish quality that has crept into my voice lately when I name-check my former employer. And this is why. Even on the opposite side of the world, it’s always the same story: “I used to love Firefox”, but “I switched to Chrome because my extensions stopped working” or “I switched to chrome because Firefox kept asking me to restart”.

I’ve had this conversation with dozens of people across three continents. Not one person has had anything good to say about the rapid release process. Nearly 100% of my highly unscientific survey volunteered the information — unasked, unprompted — that the rapid release process had ruined Firefox for them.

Of course nobody says “rapid release process” because people don’t know that’s what it was called. They might start out complaining about version numbers, or some plugin that doesn’t work right, but when I ask enough questions to get to the root of the problem, it’s always the rapid release process.”

This article, by an ex-Mozilla developer all but confirms what I wrote about here nearly two years ago:

“The only downside I see to continuous delivery is when it’s used in an environment that needs to be actively upgraded by users; it’s no point pushing out new functionality daily if your users have to do an upgrade daily.”

I am at a point where after 8 years of using Firefox as my primary day to day web browser, I am seriously considering switching to using Google Chrome primarily. I find the Firefox rapid release process ridiculous: who releases a major version upgrade every few six weeks that appears the same as any previous version? And each version requires new WebDriver binaries, unlike Chrome.

I see the iOS update issue becoming more and more prevalent as we have more iDevices in our household that need constant updating. Other annoyances include Adobe Flash on the desktop (every second day it seems), Mac OSX installing updates that need restarts and paid OSX apps now require running App Store on Mac to check for updates. Madness!

Running Watir-WebDriver tests on Travis CI: a distributed build system

I recently came across Travis CI, a distributed build system that has close links to Github. I’ve seen quite a few projects use it as a CI system, but none that run headless browser tests.

Leveraging off the work I had done recently setting up my own Jenkins server in the cloud to run headless Watir-WebDriver tests, I thought I would have a go at running my WatirMelonCucumber and EtsyWatirWebDriver headless browser tests using Travis CI.

What I didn’t realize is how easy it’d be. The only things I had to do was make sure my Gemfile included rake, and also make sure there was some file existance checking happening for some log files, and it pretty much ran straight away. Wow!

Caveat Emptor

This is pretty new territory, so there’s a few things to watch out for:

  • Every now and, Travis complains about not having Firefox installed. I am not sure why this happens, maybe something to do with the different agents in use;
  • The locale of the build agents seems to be German, so when running my Google tests, the page content is in German, so it fails because it can’t find my expected (English) results; and
  • I can’t seem to capture my test results html file nor screenshot pngs, so it’s a red/green only affair at the moment.

But still, neat distributed free headless CI!