Bailing with Mocha e2e Tests

At Automattic we use Mocha to write our end-to-end (e2e) automated tests in JavaScript/Node.js. One issue with Mocha is that it’s not really a tool suited to writing e2e tests where one test step can rely on a previous test step – for example our sign up process is a series of pages/steps which rely on the previous step passing. Mocha is primarily a unit testing tool and it’s bad practice for one unit test to depend on another, so that is why Mocha doesn’t support this.

A more simplified example of this is shown in my webdriver-js-demo project:

describe( 'Ralph Says', function() {
	this.timeout( mochaTimeoutMS );

	before( async function() {
		const builder = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() );
		driver = await builder.build();
	} );

	it( 'Visit the page', async function() {
		page = await RalphSaysPage.Visit( driver );
	} );

	it( 'shows a quote container', async function() {
		assert( await page.quoteContainerPresent(), 'Quote container not displayed' );
	} );

	it( 'shows a non-empty quote', async function() {
		assert.notEqual( await page.quoteTextDisplayed(), '', 'Quote is empty' );
	} );

	afterEach( async function() { await driver.manage().deleteAllCookies(); } );

	after( async function() { await driver.quit(); } );
} );

In our case, our test shows a quote container relies on Visit the page successfully working, if we can’t visit our page we can’t assert content on it.

If we can’t access our page and we run our tests, we will see that all three results fail, which is technically correct, but it also generates very noisy test results as it can be hard to work out exactly why the tests are failing.

Output:

1) Ralph Says
Visit the page:
TimeoutError: Waiting for element to be located By(css selector, #quote)
Wait timed out after 2003ms
at /Users/alisterscott/Projects/alisterscott/webdriver-js-demo/node_modules/selenium-webdriver/lib/promise.js:2209:17
at process._tickCallback (internal/process/next_tick.js:68:7)

2) Ralph Says
shows a quote container:
TypeError: Cannot read property 'quoteContainerPresent' of undefined
at Context. (specs/ralphsays-spec.js:24:22)

3) Ralph Says
shows a non-empty quote:
TypeError: Cannot read property 'quoteTextDisplayed' of undefined
at Context. (specs/ralphsays-spec.js:28:31)

Mocha has an inbuilt mechanism to stop, or bail, when you encounter a failure. This sounds like what we want, so we can add --bail to our mocha command, and see the following results:

1) Ralph Says
Visit the page:
TimeoutError: Waiting for element to be located By(css selector, #quote)
Wait timed out after 2003ms
at /Users/alisterscott/Projects/alisterscott/webdriver-js-demo/node_modules/selenium-webdriver/lib/promise.js:2209:17
at process._tickCallback (internal/process/next_tick.js:68:7)

 

Much less noisy! However, when using this option I noticed Mocha completely bailed on us, it didn’t run any other tests! So if I have another describe block in the same file, or any other test files, these aren’t run at all. So we don’t know what the status of our all other e2e tests are when using Mocha’s bail.

My colleague wrote a patch to Mocha that allows you to bail from just one test suite rather than the whole run on failure, which is what we currently use at Automattic.

Mocha didn’t accept this as a contribution as it’s not something that Mocha wants to support or encourage, and running our own patched version of Mocha isn’t ideal, so I recently went looking for another alternative.

I found mocha-steps which is a npm package that allows you to do exactly what we want.

Where appropriate, you simply use step instead of it and on failure within a describe block it will stop execution of the subsequent steps 😊

describe( 'Ralph Says', function() {
	this.timeout( mochaTimeoutMS );

	before( async function() {
		const builder = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() );
		driver = await builder.build();
	} );

	step( 'Visit the page', async function() {
		page = await RalphSaysPage.Visit( driver );
	} );

	step( 'shows a quote container', async function() {
		assert( await page.quoteContainerPresent(), 'Quote container not displayed' );
	} );

	step( 'shows a non-empty quote', async function() {
		assert.notEqual( await page.quoteTextDisplayed(), '', 'Quote is empty' );
	} );

	afterEach( async function() { await driver.manage().deleteAllCookies(); } );

	after( async function() { await driver.quit(); } );
} );

Now we pass --require mocha-steps to mocha and we can see the result:

1) Ralph Says
Visit the page:
TimeoutError: Waiting for element to be located By(css selector, #quote)
Wait timed out after 2002ms
at /Users/alisterscott/Projects/alisterscott/webdriver-js-demo/node_modules/selenium-webdriver/lib/promise.js:2209:17
at process._tickCallback (internal/process/next_tick.js:68:7)

The best part is that all the other e2e test scenarios continue to get run as we’re not bailing on the test run.

I’ll propose that we replace our custom mocha bail suite code with mocha-steps 😊