npm ci

I recently discovered npm ci which you can use instead of npm install when running a node project on continuous integration (CI) system and want to install your npm dependencies. It does this in a more lightweight, more CI friendly way.

If you use npm test to run your tests, this can be shortened to npm t (much like npm i is npm install), and therefore you can run npm cit to install dependencies and run tests in CI.

Running Mocha e2e Tests in Parallel

I recently highlighted the importance of e2e test design. Once you have well designed e2e tests you can start running them in parallel.

There are a couple of approaches to scaling your tests out to be run in parallel:

  1. Running the tests in multiple machine processes;
  2. Running the tests across multiple (virtual) machines;

These aren’t mutually exclusive, you can run tests in parallel processes across multiple virtual machines – we do this at Automattic – each test run happens across two virtual machines, Docker containers on CircleCI, each of which runs six processes for either desktop or mobile responsive browser sizes depending on the container.

I have found running tests in multiple processes gives best bang for buck since you don’t need additional machines (most build systems charge based on container utilisation) and you’ll benefit from parallel runs on a local machine.

We write our e2e tests in WebDriverJs and use Mocha for our test framework. We currently use a tool called Magellan to run our e2e tests in separate processes (workers), however the tool is losing Mocha support and therefore we need to look at alternatives.

I have found that mocha-parallel-tests seems like the best replacement – it’s a drop in replacement runner for mocha tests which splits test specification files across processes available on your machine you’re executing your tests on – you can also specify a maximum limit of processes as the command line argument --max-parallel

There is another parallel test runner for mocha: mocha.parallel – but this requires updating all your specs to use a different API to allow the parallelisation to work. I like mocha-parallel-tests as it just works.

I’ve updated my webdriver-js-demo project to use mocha-parallel-tests – feel free to check it out here.

Bailing with Mocha e2e Tests

At Automattic we use Mocha to write our end-to-end (e2e) automated tests in JavaScript/Node.js. One issue with Mocha is that it’s not really a tool suited to writing e2e tests where one test step can rely on a previous test step – for example our sign up process is a series of pages/steps which rely on the previous step passing. Mocha is primarily a unit testing tool and it’s bad practice for one unit test to depend on another, so that is why Mocha doesn’t support this.

A more simplified example of this is shown in my webdriver-js-demo project:

describe( 'Ralph Says', function() {
	this.timeout( mochaTimeoutMS );

	before( async function() {
		const builder = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() );
		driver = await builder.build();
	} );

	it( 'Visit the page', async function() {
		page = await RalphSaysPage.Visit( driver );
	} );

	it( 'shows a quote container', async function() {
		assert( await page.quoteContainerPresent(), 'Quote container not displayed' );
	} );

	it( 'shows a non-empty quote', async function() {
		assert.notEqual( await page.quoteTextDisplayed(), '', 'Quote is empty' );
	} );

	afterEach( async function() { await driver.manage().deleteAllCookies(); } );

	after( async function() { await driver.quit(); } );
} );

Continue reading “Bailing with Mocha e2e Tests”

Executing JS in IE11 using WebDriverJs

We write our e2e tests in JavaScript running on Node.js which allows us to use newer JavaScript/ECMAScript features like template literals.

We have a subset of our e2e tests – mainly signing up as a new customer – which we run a few times a day against Internet Explorer 11: our lowest supported IE version.

I recently added a function that sets a cookie to set the currency for a customer:

setCurrencyForPayments( currency ) {
  const setCookieCode = function( currencyValue ) {
    window.document.cookie = `landingpage_currency=${ currencyValue };domain=.wordpress.com`;
  }
return this.driver.executeScript( setCookieCode, currency );
}

This code works perfectly when executing against Chrome or Firefox, but when it came to executing against IE11 I would see the following (rather unhelpful) error:

Uncaught JavascriptError: JavaScript error (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 69 milliseconds

I couldn’t work out what was causing this so I decided to take a break. On my break I realised that WebDriverJs is trying to execute a new JavaScript feature (template literals) against an older browser that doesn’t support it! Eureka!

So all I had to do was update our code to:

setCurrencyForPayments( currency ) {
  const setCookieCode = function( currencyValue ) {
    window.document.cookie = 'landingpage_currency=' + currencyValue + ';domain=.wordpress.com';
  }
return this.driver.executeScript( setCookieCode, currency );
}

and all our tests were happy against IE11 again 😊

Having not found a lot about this error or the cause I am hoping this blog post can help someone else out if they encounter this issue also.

AMA: JavaScript in Test Automation

Max asks…

First of all thanks for sharing all your insights on this blog. I regularly try to come back to your blog and I helped me grow as a QA Engineer quite a lot.
I wanted to ask you on your opinion on a seemingly new generation of test tools for the web that are written in JavaScript to help deal with all the asynchronicity on modern front end frameworks. Our company is in the process of redesigning our website and it seems that it just gets harder and harder to deal with all the JavaScript in test automation. I recently started looking into some other tool like cypress.io, but as of now they still seem quite immature.
Could you help me and point me in the right direction?

My response…

I think there’s two parts to the JavaScript asynchronicity issue which are not necessarily related.

Modern JavaScript front-end web frameworks (like React and Angular) are designed to be asynchronous and fast and therefore this can cause issues when trying to write automated tests against them. I don’t believe writing automated tests in an asynchronous way actually makes the tests easier to write, or more maintainable or robust, but just a result of writing these tests in the same way the web frameworks work.

You can write synchronous tests (like using Watir/Ruby) against asynchronous web interfaces, you just need to use the waiting/polling mechanisms (or write/extend your own) – the same as you need to do in asynchronous test automation tools.

We choose WebDriverJS for automated end-to-end testing of our React application as it was the official WebDriver for JavaScript project and seems to be a good choice at the time. I somewhat regret that decision now as using a synchronous third party implementation like webdriver.io seems like an easier way to write and maintain tests.

I have tried to use cypress.io but the way it controls sites (through proxies) has (current) limitations like not working on iFrames and cross-domains which are deal-breakers for our end-to-end testing needs at present.

If you don’t need to write your end-to-end tests in JavaScript I’d say avoid it unless absolutely necessary and stick to another non-asynchronous programming language.

I’m glad I’ve been able to help you grow over the years.

Puppeteer for Automated e2e Chrome Testing in Node

I recently noticed the new Google Chrome project Puppeteer:

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

As someone who only runs WebDriver tests in Google Chrome anyway, this looks like a promising project that bypasses WebDriver to have full programmatic control of Google Chrome including for automated end-to-end (e2e) tests.

The thing I really love about this is no Chromedriver dependency and how installing the library installs Chromium by default which can be controlled headlessly with zero config or any other dependencies.

You can even develop scripts using this playground.

I set up a (very basic) demo project that uses Mocha + Puppeteer and it runs on CircleCI with zero config. Awesome.

AMA: Moving automated tests from Java to JavaScript

Anonymous asks…

I am currently using a BDD framework with Cucumber, Selenium and Java for automating a web application. I used page factory to store the objects and using them in java methods I wanted to replace the java piece of code with javaScript like mocha or webdriverio. could you share your thoughts on this? can I still use page factory to maintain objects and use them in js files

My response…

What’s the reasoning for moving to JavaScript from Java? Despite having common names, there’s very little otherwise in common (Car is to Carpet as Java is to JavaScript.)

I wouldn’t move for moving sake since I see no benefit in writing BDD style web tests in JavaScript, if anything, e2e automated tests are much harder to write in JavaScript/Node because everything is asynchronous and so you have to deal with promises etc. which is much harder to do than just using Java (or Ruby).

Aside: I still dream of writing e2e tests in Ruby: it’s just so pleasant. But our new user interface is written extensively in JavaScript (React) so it makes sense from a sustainability point of view to use JS over Ruby.

 

Minesweepers Anonymous

Once upon a time there lived a couple who loved playing massively multiplayer online role-playing video games with each other (World of Warcraft or Destiny or something). The thing about these games is that they’re a bit like exercise/fitness where you need to keep playing them to keep your score high, and if you don’t you play them (or exercise) enough your score (or fitness) starts dropping which makes it much less fun overall.

So playing these games all went well for the couple, for a while.

Continue reading “Minesweepers Anonymous”

Handling JavaScript alerts when leaving a page with WebDriver

You’ve most probably seen the sometimes-useful-but-often-annoying browser alerts when navigating away from a page:JavaScript onbeforeunload alert

How do we deal with these using WebDriver?

Continue reading “Handling JavaScript alerts when leaving a page with WebDriver”

AMA: Learning JavaScript automation & starting e2e tests

Edwin asks…

How should someone approach learning JavaScript strictly for the purpose of test automation? Should they learn it the way a front-end developer would or is there a better way to learn it for purposes of writing automation?

anil also asks…

I need to do automation by using mocha chai framework with selenum webdriver. can you please guide me or suggest any site or video on how to start and what are the basic things need to know.

Continue reading “AMA: Learning JavaScript automation & starting e2e tests”

AMA: more details about our JS e2e tests

Satyajit Malugu asks…

Saw your recent post on JS vs Ruby. As a recent migrant from Ruby to JS for mobile test automation – couple of questions 1) how are you doing page objects (ES6 works too) 2) Are you able to run your E2E tests in parallel? 3)Is there anything like binding.pry for debugging JS whilst execution?)

My response…

  1. This post explains how we’re doing our page objects in ES6, and since all our e2e tests are open source, you can see for yourself :)
  2. We use CircleCI which supports parallel test execution, which I enabled in the past, but it’s currently disabled as it was consuming too many build containers which are shared across our organization. Since we have a lot more containers available now I have it on my list to renable these.
  3. I believe WebStorm supports debugging Node Mocha tests; I’ve been meaning to investigate this over console.log statements; I’ll report back when I get around to it.

AMA: Mocha this.timeout Duplication

 Alex C asks…

I notice that the “this.timeout” code is duplicated across the before and test suite. This seems mentally uncomfortable to me and the logical question is: “Why duplicate code that is in a before method?” Right now, the only thing I can think would be an explanation is “because it’s just a workaround to how the Javascript binding does things.”

import assert from 'assert';
import webdriver from 'selenium-webdriver';
import test from 'selenium-webdriver/testing';
import config from 'config';
import RalphSaysPage from '../lib/ralph-says-page.js';

let driver;

const mochaTimeoutMS = config.get( 'mochaTimeoutMS' );

test.before( function() {
	this.timeout( mochaTimeoutMS );
	driver = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() ).build();
} );

test.describe( 'Ralph Says', function() {
	this.timeout( mochaTimeoutMS );

	test.it( 'shows a quote container', function() {
		var page = new RalphSaysPage( driver, true );
		page.quoteContainerPresent().then( function( present ) {
			assert.equal( present, true, 'Quote container not displayed' );
		} );
	} );

	test.it( 'shows a non-empty quote', function() {
		var page = new RalphSaysPage( driver, true );
		page.quoteTextDisplayed().then( function( text ) {
			assert.notEqual( text, '', 'Quote is empty' );
		} );
	} );
} );

test.afterEach( () => driver.manage().deleteAllCookies() );

test.after( () => driver.quit() );

My response…

Mocha has a low default timeout (2000ms) as it wasn’t really written to be used for e2e testing. In the way I’ve written the above code, all the four test. blocks don’t share any instance state, so setting this.timeout in one will not effect the other, hence why I wrote it that way.

Having a few months experience behind me, I can see it’s easy to rewrite the above tests to avoid this duplication (and to have less lines of code):

import assert from 'assert';
import webdriver from 'selenium-webdriver';
import test from 'selenium-webdriver/testing';
import config from 'config';
import RalphSaysPage from '../lib/ralph-says-page.js';

let driver;

const mochaTimeoutMS = config.get( 'mochaTimeoutMS' );

test.describe( 'Ralph Says', function() {
	this.timeout( mochaTimeoutMS );

	test.before( function() {
		driver = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() ).build();
	} );

	test.it( 'shows a quote container', function() {
		var page = new RalphSaysPage( driver, true );
		page.quoteContainerPresent().then( function( present ) {
			assert.equal( present, true, 'Quote container not displayed' );
		} );
	} );

	test.it( 'shows a non-empty quote', function() {
		var page = new RalphSaysPage( driver, true );
		page.quoteTextDisplayed().then( function( text ) {
			assert.notEqual( text, '', 'Quote is empty' );
		} );
	} );

	test.afterEach( () => driver.manage().deleteAllCookies() );

	test.after( () => driver.quit() );
} );

But you still would have to duplicate this across different test files though, even though it’s stored as a config value. I’ve since realised Mocha allows you to globally override this timeout using test/mocha.opts.

AMA: R.Y.O. Page Objects 2.0

Michael Karlovich asks…

Do you have any updated thoughts on rolling your own page objects with Watir? The original post is almost 4 years old but is still the basis (loosely) of every page object framework I’ve built since then.

My response…

Wow, I can’t believe that post is almost four years old. I have also have used this for the basis of every page object framework I have built since then.

I recently had a look at our JavaScript (ES2015) code of page objects and despite ES2015 not having meta-programming support like ruby, our classes are remarkably similar to what I was proposing ~4 years ago.

I believe this is because some patterns are classic and therefore almost timeless, they can be applied over and over again to different contexts. There’s a huge amount of negativity towards best practices of late, but I could seriously say that page objects are a best practise for test automation of ui systems, which isn’t saying they will be exactly the same in every context, but there’s a common best-practice pattern there which you most likely should be using.

Page objects, as a pattern, typically:

  • Inherit from a base page object/container which stores common actions like:
    • instantiating the object looking for a known element that defines that page’s existence
    • optionally allow a ‘visit’ to the page during instantiation using some defined URL/path
    • provides actions and properties common to all pages, for example: waiting for the page, checking the page is displayed, getting the title/url of the page, and checking cookies and local storage items for that page;
  • Define actions as methods which are ways of interacting with that page (such as logging in);
  • Do not expose internals about the page outside the page – for example they typically don’t expose elements or element selectors which should only be used within actions/methods for that page which are exposed; and
  • Can also be modelled as components for user interfaces that are built using components to give greater reusability of the same components across different pages.

The biggest benefit I have found from using page objects as a pattern is having more deterministic end-to-end tests since instantiating a page I know I am on that page, so my tests will fail more reliably with a better understanding of what went wrong.

Are there any other pattern attributes you would consider vital for page objects?

AMA: Async JavaScript Tests?

David asks…

Now that you work with javascript and asynchronous code. It would be interesting to get your thoughts on asynchronous REST/HTTP calls in javascript with respect to testing, if you ever get around to that. I’m familiar with using it in web UIs on client side, but haven’t wrapped my head around that on the test/server side in node.js. I have so far opted to use a synchronous HTTP request library in node.js when I can.

My response…

This is an interesting question David. Whilst WebDriverJs and Mocha etc. are asynchronous on Node, we aren’t using any of that asynchronicity, but dealing with it using promises to resolve to a value we are looking for. It would be very cool to be able to have every WebDriver test we write truly asynchronous to spin up a browser do some testing and wait for any results all whilst running other asynchronous tests, but I believe to do this would be very complicated and pushing the limits of current technology/browser process threads etc. It would be possible if you were writing pure node/ES unit tests though.

So to answer your question we have a bunch of e2e tests written in Mocha that run sequentially one after another in a synchronous manner (using promises), even though they utilise asynchronous libraries.

AMA: bdd frameworks, api test tools & unit testing responsibility

Swapnil Waghmare asks…

Are BDD frameworks like Specflow, Cucumber better for E2E tests?

My response…

There’s two ways I can interpret this question (my additions are in bold).

Firstly:

Are BDD frameworks that use ‘Given/When/Then’ feature files like Specflow, Cucumber  better than frameworks that use ‘describe/it’ blocks like RSpec and Mocha for e2e tests?

As I previously explained, Automattic’s unit tests are written in Mocha, so that was a logical choice for writing e2e tests as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.

Secondly:

Are BDD frameworks like Specflow, Cucumber better for E2E tests than using them for automated integration, component or unit tests?

I think you could write integration tests or even unit tests in Given/When/Then format as most unit tests follow the same arrange/act/assert pattern anyhow which is exactly what Given/When/Then is.

Keep in mind there is overhead in maintaining all the step definitions and feature files for Cucumber/Specflow that give you non technical readability so if you don’t require that readability it is probably overkill. But a personal preference nonetheless.

Swapnil Waghmare also asks…

Which tools have you used for API testing, which ones would you recommend?

My response…

At the moment my main focus is on automated unit test and automated e2e test coverage in JavaScript.

I try to keep things simple, so in the past when I’ve written REST integration tests I’ve just called them using in built libraries, and have used Postman quite a lot for manual testing and debugging.

Swapnil Waghmare finally asks…

What is the reason behind writing E2E tests using JavaScript? Isn’t any Object oriented language a better choice for writing E2E tests? Should QA’s write Unit tests?

My response…

 

Firstly, JavaScript is actually object oriented, it’s just not class-based object oriented like Java, C# or Ruby. The newer versions of JavaScript, called ECMAScript, or ESScript are more class-based object oriented.

I’ve actually already answered why I chose JavaScript in a previous answer, so I’ll summarise that here and you can read the rest in that answer.

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for WordPress.com where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

As for whether QA’s should write unit tests? No, I don’t think so, as I believe unit tests should drive software development, and writing code by writing unit tests is much easier than trying to add unit tests after by someone else, as the original code will not likely be very testable as it wasn’t written with testability in mind. One benefit of code written with unit tests at the time is that it will mostly be better code as the tests are consuming your API that you are developing.

AMA: JS vs Ruby

Butch Mayhew asks…

I have noticed you blogging more about JS frameworks. How do these compare to Watir/Ruby? Would you recommend one over the other?

My response…

I had a discussion recently with Chuck van der Linden about this same topic as he has a lot of experience with Watir and is now looking at JavaScript testing frameworks like I have done.

Some Background

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for WordPress.com where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

Whilst I see merit in both views: I still think having your automated acceptance tests in the same language as your application leads to better maintainability and adoptability.

I still think writing automated acceptance tests in Ruby is much cleaner and nicer than JavaScript Node tests, particularly as Ruby allows meta-programming which means page objects can be implemented really neatly.

The JavaScript/NodeJS landscape is still very immature where people are using various tools/frameworks/compilers and certain patterns or de facto standards haven’t really emerged yet. The whole ES6/ES2015/ES2016 thing is very confusing to newcomers like me, especially on NodeJS where some ES6+ features are supported, but others require something like Babel to compile your code.

But generally with the direction ES is going, writing page objects as classes is much nicer than using functions for everything as in ES5.

Whilst there’s nothing I have found that is better (or even as good) in JavaScript/Mocha/WebDriverJS than Ruby/RSpec/Watir-WebDriver, I still think it’s a better long term decision for WordPress.com to use the JavaScript NodeJS stack for our e2e tests.

AMA: JavaScript & Mobile App Automation Tools

Justin Watts asks…

Have you looked at Chimp.js ? Any thoughts? I was very closely following your posts on picking a new framework for Automattic and I was quite surprised when you chose an async tool and stepped away from Cucumber. As a ruby dev looking to get comfortable with testing web-apps in JavaScript, I am torn on what library to dig into.

My response…

I had a quick look at Chimp.js when I was evaluating tools, it looks quite impressive, but quite opinionated and integrated, so I find these types of libraries are good to start with, but quickly become frustrating when you want to start doing advanced things (like the custom Slack reporter we just wrote that pings slack when tests fail with screenshots). I am curious about how Chimp.js is synchronous when the underlying WebDriver.IO tool uses promises?

I think the asynchronous thing is less of a big deal than most people make out. Once you get used to using promises whenever querying a value from the browser, then you just do that. I have set up my own es6 page objects so these take away most of this complication anyway.

The one synchronous tool I did seriously contemplate was webdriver-sync but since it wraps the underlying WebDriver Java driver, it was quite heavyweight as you need Java, the node-java bridge, and I couldn’t get this working on CircleCI which is what Automattic’s uses as its CI platform.

As for stepping away from Cucumber. Automattic’s unit tests are written in Mocha, so that was a logical choice as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.

Justin Watts also asks…

Appium seems to be getting more unstable as time goes on. Early Grey caught my eye. Do you have any thoughts / outlooks on the mobile app testing landscape?

My response…

I must say I haven’t had a lot of recent experience with mobile UI automation tools. A couple of years back we started using Appium but abandoned this effort as the tests were so fragile we could never get green builds, so we focused on writing unit tests for the specific mobile code, testing webviews from webdriver, and doing human end to end testing on real devices. This worked well, and unless I worked on complex native apps I would probably end up doing something similar again should this arise.

 

 

Testing end-to-end with Mocha

As part of my excellent Excellence Wrangler role at Automattic, one of my key tasks has been establishing some end-to-end tests for WordPress.com using Mocha with WebDriverJs. Our testing pyramid doesn’t look much like a pyramid:

wordpress.com test pyramid.png

We’ve got lots of React unit tests at the bottom: these are to speed development.

We’re intentionally missing a middle: the REST API we consume has its own unit tests, we don’t need integration tests for it. We don’t have detailed full stack acceptance tests of our UI: these are too slow and brittle.

We have a handful of e2e flow tests at the top, these are to protect the user experience, we run these on every deployment and frequently in production. These can be brittle on such a fast moving code base, but we limit their number (depth) so they still give us good confidence everything is working well together but limiting our overhead.

So what do these end-to-end tests look like?

I hadn’t used Mocha before and I was used to writing end-to-end tests in Gherkin format in tools like Cucumber and Specflow so I initially began writing end-to-end tests that looked like this:

test.describe('WordPress.com Sign Up', function() {
  test.beforeEach(function() {
    driver.manage().deleteAllCookies();
  });

  test.it('Can Create A Free Blog', function() {
    var signupFlow = new SignUpFlow( driver, 'desktop' );
    signupFlow.createFreeBlog( 'en' );
  });

  test.it('Can Create A New Site With a Paid Domain Upgrade', function() {
    var signupFlow = new SignUpFlow( driver, 'desktop' );
    signupFlow.CreateSiteWithDomainPaidByCreditCard( 'en' );
  });
});

I was pushing the code down into flow classes which I have used before, but the issue with this was the output I was getting from Mocha wasn’t very rich:

WordPress.com Sign Up
      ✓ Can Create A Free Blog
      ✓ Can Create A New Site With a Paid Domain Upgrade

I then realized by looking at an end-to-end test written by another developer that you can nest describe and it statements to give you much more expressive end-to-end tests.

test.describe( `Sign Up (${screenSize})`, function() {

  test.describe( 'Free Site:', function() {
    test.before( 'Delete Cookies and Local Storage', function() {
      driverManager.clearCookiesAndDeleteLocalStorage( driver );
    } );

    test.describe( 'Sign up for a free site', function() {

      test.describe( 'Step One: Themes', function() {
        test.before( 'Can see the choose a theme page as the starting page', function() {
          this.chooseAThemePage = new ChooseAThemePage( driver, { visit: true } );
          return this.chooseAThemePage.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a theme start page is not displayed' );
          } );
        } );

        test.it( 'Can select the first theme', function() {
          return this.chooseAThemePage.selectFirstTheme();
        } );
      } );

      test.describe( 'Step Two: Domains', function() {
        test.before( 'Can then see the domains page ', function() {
          this.findADomainComponent = new FindADomainComponent( driver );
          return this.findADomainComponent.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a domain page is not displayed' );
          } );
        } );

        test.it( 'Can search for a blog name', function() {
          return this.findADomainComponent.searchForBlogNameAndWaitForResults( blogName );
        } );

        test.it( 'Can see a free WordPress.com blog address in results ', function() {
          return this.findADomainComponent.freeBlogAddress().then( ( actualAddress ) => {
            assert.equal( actualAddress, expectedBlogAddress, 'The expected free address is not shown' )
          } );
        } );

        test.it( 'Can select the free address', function() {
          return this.findADomainComponent.selectFreeAddress();
        } );
      } );

This gives us rich feedback:

Sign Up (desktop)
    Free Site:
      Sign up for a free site
        Step One: Themes
          ✓ Can see the choose a theme page as the starting page
          ✓ Can select the first theme
        Step Two: Domains
          ✓ Can then see the domains page
          ✓ Can search for a blog name
          ✓ Can see a free WordPress.com blog address in results
          ✓ Can select the free address

The mistake I had made which I didn’t realize was not creating enough nesting, instead of having Step One, Step Two etc. next to one another, they should be nested within each other. This is because if they’re next to one another, Mocha will run the Step Two, Step Three etc. even if Step One has failed, which is not what we want in an end-to-end scenario where each step is dependent on the previous one.

So, it now looks something like this:

test.describe( 'Free Site:', function() {
    test.before( 'Delete Cookies and Local Storage', function() {
      driverManager.clearCookiesAndDeleteLocalStorage( driver );
    } );

    test.describe( 'Sign up for a free site', function() {
      test.describe( 'Step One: Themes', function() {
        test.before( 'Can see the choose a theme page as the starting page', function() {
          this.chooseAThemePage = new ChooseAThemePage( driver, { visit: true } );
          return this.chooseAThemePage.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a theme start page is not displayed' );
          } );
        } );

        test.it( 'Can select the first theme', function() {
          return this.chooseAThemePage.selectFirstTheme();
        } );

        test.describe( 'Step Two: Domains', function() {
          test.before( 'Can then see the domains page ', function() {
            this.findADomainComponent = new FindADomainComponent( driver );
            return this.findADomainComponent.displayed().then( ( displayed ) => {
              assert.equal( displayed, true, 'The choose a domain page is not displayed' );
            } );
          } );

          test.it( 'Can search for a blog name', function() {
            return this.findADomainComponent.searchForBlogNameAndWaitForResults( blogName );
          } );

          test.it( 'Can see a free WordPress.com blog address in results ', function() {
            return this.findADomainComponent.freeBlogAddress().then( ( actualAddress ) => {
              assert.equal( actualAddress, expectedBlogAddress, 'The expected free address is not shown' )
            } );
          } );

          test.it( 'Can select the free address', function() {
            return this.findADomainComponent.selectFreeAddress();
          } );

          test.describe( 'Step Three: Plans', function() {

which means the output is slightly different but still very useful:

Sign Up (mobile)
  Free Site:
    Sign up for a free site
      Step One: Themes
        ✓ Can select the first theme
        Step Two: Domains
          ✓ Can search for a blog name
          ✓ Can see a free WordPress.com blog address in results
          ✓ Can select the free address
          Step Three: Plans
            ✓ Can select the free plan

These tests are much better written this way. The only issue I am left facing with Mocha is when a before hook fails (such as logging in) the generic afterEach hook we have to take screenshots is not triggered (this is only triggered when an it block is run.