At Automattic we use Mocha to write our end-to-end (e2e) automated tests in JavaScript/Node.js. One issue with Mocha is that it’s not really a tool suited to writing e2e tests where one test step can rely on a previous test step – for example our sign up process is a series of pages/steps which rely on the previous step passing. Mocha is primarily a unit testing tool and it’s bad practice for one unit test to depend on another, so that is why Mocha doesn’t support this.
A more simplified example of this is shown in my webdriver-js-demo project:
describe( 'Ralph Says', function() { this.timeout( mochaTimeoutMS ); before( async function() { const builder = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() ); driver = await builder.build(); } ); it( 'Visit the page', async function() { page = await RalphSaysPage.Visit( driver ); } ); it( 'shows a quote container', async function() { assert( await page.quoteContainerPresent(), 'Quote container not displayed' ); } ); it( 'shows a non-empty quote', async function() { assert.notEqual( await page.quoteTextDisplayed(), '', 'Quote is empty' ); } ); afterEach( async function() { await driver.manage().deleteAllCookies(); } ); after( async function() { await driver.quit(); } ); } );
In our case, our test shows a quote container
relies on Visit the page
successfully working, if we can’t visit our page we can’t assert content on it.
If we can’t access our page and we run our tests, we will see that all three results fail, which is technically correct, but it also generates very noisy test results as it can be hard to work out exactly why the tests are failing.
Output:
1) Ralph Says Visit the page: TimeoutError: Waiting for element to be located By(css selector, #quote) Wait timed out after 2003ms at /Users/alisterscott/Projects/alisterscott/webdriver-js-demo/node_modules/selenium-webdriver/lib/promise.js:2209:17 at process._tickCallback (internal/process/next_tick.js:68:7) 2) Ralph Says shows a quote container: TypeError: Cannot read property 'quoteContainerPresent' of undefined at Context. (specs/ralphsays-spec.js:24:22) 3) Ralph Says shows a non-empty quote: TypeError: Cannot read property 'quoteTextDisplayed' of undefined at Context. (specs/ralphsays-spec.js:28:31)
Mocha has an inbuilt mechanism to stop, or bail, when you encounter a failure. This sounds like what we want, so we can add --bail
to our mocha command, and see the following results:
1) Ralph Says Visit the page: TimeoutError: Waiting for element to be located By(css selector, #quote) Wait timed out after 2003ms at /Users/alisterscott/Projects/alisterscott/webdriver-js-demo/node_modules/selenium-webdriver/lib/promise.js:2209:17 at process._tickCallback (internal/process/next_tick.js:68:7)
Β
Much less noisy! However, when using this option I noticed Mocha completely bailed on us, it didn’t run any other tests! So if I have another describe block in the same file, or any other test files, these aren’t run at all. So we don’t know what the status of our all other e2e tests are when using Mocha’s bail.
My colleague wrote a patch to Mocha that allows you to bail from just one test suite rather than the whole run on failure, which is what we currently use at Automattic.
Mocha didn’t accept this as a contribution as it’s not something that Mocha wants to support or encourage, and running our own patched version of Mocha isn’t ideal, so I recently went looking for another alternative.
I found mocha-steps which is a npm package that allows you to do exactly what we want.
Where appropriate, you simply use step
instead of it
and on failure within a describe block it will stop execution of the subsequent steps π
describe( 'Ralph Says', function() { this.timeout( mochaTimeoutMS ); before( async function() { const builder = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() ); driver = await builder.build(); } ); step( 'Visit the page', async function() { page = await RalphSaysPage.Visit( driver ); } ); step( 'shows a quote container', async function() { assert( await page.quoteContainerPresent(), 'Quote container not displayed' ); } ); step( 'shows a non-empty quote', async function() { assert.notEqual( await page.quoteTextDisplayed(), '', 'Quote is empty' ); } ); afterEach( async function() { await driver.manage().deleteAllCookies(); } ); after( async function() { await driver.quit(); } ); } );
Now we pass --require mocha-steps
to mocha and we can see the result:
1) Ralph Says Visit the page: TimeoutError: Waiting for element to be located By(css selector, #quote) Wait timed out after 2002ms at /Users/alisterscott/Projects/alisterscott/webdriver-js-demo/node_modules/selenium-webdriver/lib/promise.js:2209:17 at process._tickCallback (internal/process/next_tick.js:68:7)
The best part is that all the other e2e test scenarios continue to get run as we’re not bailing on the test run.
I’ll propose that we replace our custom mocha bail suite code with mocha-steps π
How steps are behaving with mocha’s retying mechanism?
They appeared to work exactly the same as it blocks in my testing. Iβm not a big fan of mochas retry as it only retries the step not the describe block (scenario)