Running Mocha e2e Tests in Parallel

I recently highlighted the importance of e2e test design. Once you have well designed e2e tests you can start running them in parallel.

There are a couple of approaches to scaling your tests out to be run in parallel:

  1. Running the tests in multiple machine processes;
  2. Running the tests across multiple (virtual) machines;

These aren’t mutually exclusive, you can run tests in parallel processes across multiple virtual machines – we do this at Automattic – each test run happens across two virtual machines, Docker containers on CircleCI, each of which runs six processes for either desktop or mobile responsive browser sizes depending on the container.

I have found running tests in multiple processes gives best bang for buck since you don’t need additional machines (most build systems charge based on container utilisation) and you’ll benefit from parallel runs on a local machine.

We write our e2e tests in WebDriverJs and use Mocha for our test framework. We currently use a tool called Magellan to run our e2e tests in separate processes (workers), however the tool is losing Mocha support and therefore we need to look at alternatives.

I have found that mocha-parallel-tests seems like the best replacement – it’s a drop in replacement runner for mocha tests which splits test specification files across processes available on your machine you’re executing your tests on – you can also specify a maximum limit of processes as the command line argument --max-parallel

There is another parallel test runner for mocha: mocha.parallel – but this requires updating all your specs to use a different API to allow the parallelisation to work. I like mocha-parallel-tests as it just works.

I’ve updated my webdriver-js-demo project to use mocha-parallel-tests – feel free to check it out here.

AMA: JavaScript in Test Automation

Max asks…

First of all thanks for sharing all your insights on this blog. I regularly try to come back to your blog and I helped me grow as a QA Engineer quite a lot.
I wanted to ask you on your opinion on a seemingly new generation of test tools for the web that are written in JavaScript to help deal with all the asynchronicity on modern front end frameworks. Our company is in the process of redesigning our website and it seems that it just gets harder and harder to deal with all the JavaScript in test automation. I recently started looking into some other tool like cypress.io, but as of now they still seem quite immature.
Could you help me and point me in the right direction?

My response…

I think there’s two parts to the JavaScript asynchronicity issue which are not necessarily related.

Modern JavaScript front-end web frameworks (like React and Angular) are designed to be asynchronous and fast and therefore this can cause issues when trying to write automated tests against them. I don’t believe writing automated tests in an asynchronous way actually makes the tests easier to write, or more maintainable or robust, but just a result of writing these tests in the same way the web frameworks work.

You can write synchronous tests (like using Watir/Ruby) against asynchronous web interfaces, you just need to use the waiting/polling mechanisms (or write/extend your own) – the same as you need to do in asynchronous test automation tools.

We choose WebDriverJS for automated end-to-end testing of our React application as it was the official WebDriver for JavaScript project and seems to be a good choice at the time. I somewhat regret that decision now as using a synchronous third party implementation like webdriver.io seems like an easier way to write and maintain tests.

I have tried to use cypress.io but the way it controls sites (through proxies) has (current) limitations like not working on iFrames and cross-domains which are deal-breakers for our end-to-end testing needs at present.

If you don’t need to write your end-to-end tests in JavaScript I’d say avoid it unless absolutely necessary and stick to another non-asynchronous programming language.

I’m glad I’ve been able to help you grow over the years.

AMA: Moving automated tests from Java to JavaScript

Anonymous asks…

I am currently using a BDD framework with Cucumber, Selenium and Java for automating a web application. I used page factory to store the objects and using them in java methods I wanted to replace the java piece of code with javaScript like mocha or webdriverio. could you share your thoughts on this? can I still use page factory to maintain objects and use them in js files

My response…

What’s the reasoning for moving to JavaScript from Java? Despite having common names, there’s very little otherwise in common (Car is to Carpet as Java is to JavaScript.)

I wouldn’t move for moving sake since I see no benefit in writing BDD style web tests in JavaScript, if anything, e2e automated tests are much harder to write in JavaScript/Node because everything is asynchronous and so you have to deal with promises etc. which is much harder to do than just using Java (or Ruby).

Aside: I still dream of writing e2e tests in Ruby: it’s just so pleasant. But our new user interface is written extensively in JavaScript (React) so it makes sense from a sustainability point of view to use JS over Ruby.

 

Handling JavaScript alerts when leaving a page with WebDriver

You’ve most probably seen the sometimes-useful-but-often-annoying browser alerts when navigating away from a page:JavaScript onbeforeunload alert

How do we deal with these using WebDriver?

Continue reading “Handling JavaScript alerts when leaving a page with WebDriver”

AMA: JS vs Ruby

Butch Mayhew asks…

I have noticed you blogging more about JS frameworks. How do these compare to Watir/Ruby? Would you recommend one over the other?

My response…

I had a discussion recently with Chuck van der Linden about this same topic as he has a lot of experience with Watir and is now looking at JavaScript testing frameworks like I have done.

Some Background

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for WordPress.com where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

Whilst I see merit in both views: I still think having your automated acceptance tests in the same language as your application leads to better maintainability and adoptability.

I still think writing automated acceptance tests in Ruby is much cleaner and nicer than JavaScript Node tests, particularly as Ruby allows meta-programming which means page objects can be implemented really neatly.

The JavaScript/NodeJS landscape is still very immature where people are using various tools/frameworks/compilers and certain patterns or de facto standards haven’t really emerged yet. The whole ES6/ES2015/ES2016 thing is very confusing to newcomers like me, especially on NodeJS where some ES6+ features are supported, but others require something like Babel to compile your code.

But generally with the direction ES is going, writing page objects as classes is much nicer than using functions for everything as in ES5.

Whilst there’s nothing I have found that is better (or even as good) in JavaScript/Mocha/WebDriverJS than Ruby/RSpec/Watir-WebDriver, I still think it’s a better long term decision for WordPress.com to use the JavaScript NodeJS stack for our e2e tests.

AMA: JavaScript & Mobile App Automation Tools

Justin Watts asks…

Have you looked at Chimp.js ? Any thoughts? I was very closely following your posts on picking a new framework for Automattic and I was quite surprised when you chose an async tool and stepped away from Cucumber. As a ruby dev looking to get comfortable with testing web-apps in JavaScript, I am torn on what library to dig into.

My response…

I had a quick look at Chimp.js when I was evaluating tools, it looks quite impressive, but quite opinionated and integrated, so I find these types of libraries are good to start with, but quickly become frustrating when you want to start doing advanced things (like the custom Slack reporter we just wrote that pings slack when tests fail with screenshots). I am curious about how Chimp.js is synchronous when the underlying WebDriver.IO tool uses promises?

I think the asynchronous thing is less of a big deal than most people make out. Once you get used to using promises whenever querying a value from the browser, then you just do that. I have set up my own es6 page objects so these take away most of this complication anyway.

The one synchronous tool I did seriously contemplate was webdriver-sync but since it wraps the underlying WebDriver Java driver, it was quite heavyweight as you need Java, the node-java bridge, and I couldn’t get this working on CircleCI which is what Automattic’s uses as its CI platform.

As for stepping away from Cucumber. Automattic’s unit tests are written in Mocha, so that was a logical choice as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.

Justin Watts also asks…

Appium seems to be getting more unstable as time goes on. Early Grey caught my eye. Do you have any thoughts / outlooks on the mobile app testing landscape?

My response…

I must say I haven’t had a lot of recent experience with mobile UI automation tools. A couple of years back we started using Appium but abandoned this effort as the tests were so fragile we could never get green builds, so we focused on writing unit tests for the specific mobile code, testing webviews from webdriver, and doing human end to end testing on real devices. This worked well, and unless I worked on complex native apps I would probably end up doing something similar again should this arise.

 

 

Running Automated Tests with A/B Testing

Like a lot of modern, data driven sites, WordPress.com uses A/B testing extensively to introduce new features. These tests may be as simple as a label change or as complex as changing the entire sign up flow, for example by offering a free trial.

Since I have been working on a set of automated end-to-end tests for WordPress.com, I have found A/B testing to be problematic for automated testing on this very fast moving codebase, namely:

  1. Automated tests need to be deterministic: having a randomised experiment as an A/B test means the first test run may get an entirely different sign up flow than a second test run which is very hard to automate; and
  2. Automated tests need to know which experiments are running otherwise they may encounter unexpected behaviour randomly.

What we need is two methods to deal with A/B tests when running automated tests:

  1. We need to be able to see which A/B tests are active and compare this to a known list of expected A/B tests – so that we don’t suddenly encounter some unexpected/random behaviour for some of our test runs
  2. We need to be able to set the desired behaviour to the control group so that are our tests are deterministic.

Different sites conduct A/B testing using different tools and approaches, WordPress.com uses HTML5 local storage to set which A/B tests are active and which group the user belongs to.

Luckily it’s easy to read and update local storage using WebDriver and JavaScript. This means our approach is to:

  1. Each time a page object is initialised, there is a call on the base page model that checks the A/B tests that are active using something like return window.localStorage.ABTests; and then compares this to the known list of A/B tests which are checked in as a config item. This fails the test if there’s a new A/B test introduced that isn’t in the list of known tests. This is better than not knowing about the A/B test and failing based upon some non-deterministic behaviour.
  2. When a new A/B test is introduced and we wish to ensure our automated tests always use the control group, we can set this using a similar method window.localStorage.setItem('ABTests','{"flow":"default"}'); and refresh the page.

Ideally it would be good to know and plan every A/B test for our automated e2e tests, but since this isn’t possible, checking against known A/B tests and ensuring control groups are set means our automated tests are at least more consistent and deterministic, and fail a lot faster and more consistently when a new A/B test has been introduced.

How do you deal with non-determinism with A/B tests?

WebDriverJS & Mocha Part 3: Page Objects

I recently shared how to get started with WebDriverJS and Mocha, and how to use hooks.

This post continues on from there: I will share how to set up our own page objects to model our application for maintainability.

Continue reading “WebDriverJS & Mocha Part 3: Page Objects”

Why I don’t like Jasmine JavaScript unit tests (for MVC web apps)

Update 11-Feb-2013: After some feedback, I would like to clarify what I am talking about here: I am talking from experience in applications not purely written in JavaScript: for example a MVC application written with say C# with JavaScript added to pages.

I’m really not a big fan of Jasmine JavaScript unit tests for applications not written purely in JavaScript. Here’s why:

jasmine logo

Most modern web applications use JavaScript as a way to provide client side validation and interaction. Most of this JavaScript manipulates the Document Object Model (DOM) of the browser, therefore the JavaScript is coupled to the DOM using ids or classes of elements.

As most web pages are dynamically rendered by the server, Jasmine, the JavaScript unit test framework, can’t run against the ‘real’ DOM as it appears in the application, so what it does is duplicates parts of the DOM to test the JavaScript is working properly based upon this duplication.

As there’s no reliance on the real application or its DOM, these tests run super fast, you can run thousands of JavaScript tests in seconds which is pretty amazing.

The problem becomes that over time the real DOM in the application is changed but the Jasmine tests and its own duplicated DOM is not! This means that often I find that all Jasmine tests pass (as they have to for a new version to be deployed to QA) but the JavaScript doesn’t work in the application because an id or class name has been changed and someone ‘forgot’ to update the Jasmine tests.

Tests that don’t pick up valid errors are worthless, and I continually find myself getting annoyed that all the Jasmine tests have passed, but JavaScript is broken and our application doesn’t work.

Another downside is that Jasmine tests are only as good as the JavaScript engine they are run against. We run our Jasmine tests in a headless PhantomJS browser, which is fast, but most of our JavaScript bugs are found in Internet Explorer, particularly earlier versions of it, so Jasmine tests again are worthless.

I don’t know of a way to overcome these downsides to Jasmine that I have mentioned. What we do have is a suite of end to end acceptance tests that run in 5 different ‘real’ browsers that check the JavaScript works correctly. Whilst these take 8 minutes to run per browser, they pick up a lot more JavaScript issues than Jasmine.

I am not still not sure whether it is worth investing the time and effort in Jasmine tests when they don’t provide sufficient confidence that our application ‘works’.

What have others experiences been with Jasmine JavaScript unit testing?

Using WebDriver to automatically check for JavaScript errors on every page

Update: please see my newer post on this that doesn’t require changes to your application.


One of the benefits of using a page-object model is that you can perform certain actions on every page in your application that you visit, such as checking for accessibility.

One such check is automatically checking for JavaScript errors on page load. There’s a couple of approaches out there, one involves copying and pasting a small snippet of JavaScript into each page. Since our application we are working on uses a standard template for every page, we simply add some JavaScript to the common page header that is the first thing to load on the page, and catches any JavaScript errors that occur:

define(["amdUtils/string/interpolate"], function(interpolate) {
    window.jsErrors = [];

    window.onerror = function (errorMessage, url, lineNumber) {
        var message = interpolate("Error: [{{0}}], url: [{{1}}], line: [{{2}}]", [errorMessage, url, lineNumber]);
        window.jsErrors.push(message);
        return false;
    };
});

It’s then a matter of checking the jsErrors each time we visit a page, this is example C# code we use in the base page class for every page.

var js = driver as IJavaScriptExecutor;
ICollection javascriptErrors = null;
for (var i = 0; i < 20; i++)
{
  javascriptErrors = js.ExecuteScript("return window.jsErrors") as ICollection;
  if (javascriptErrors != null) break;
  System.Threading.Thread.Sleep(1000);
}
Assert.IsNotNull(javascriptErrors, "Can't seem to load JavaScript on the page to find JavaScript errors. Check that JavaScript is enabled.");
var javaScriptErrorsAsString = javascriptErrors.Cast<string>().Aggregate("", (current, error) => current + (error + ", "));
Assert.AreEqual("", javaScriptErrorsAsString, "Found JavaScript errors on page load: " + javaScriptErrorsAsString);

This code waits until it can read the jsErrors on the page. If it can’t, it means that JavaScript didn’t load and this is an error. Once it gets the jsErrors, it checks that there are none.

This code has been very useful for us. It has caught a number of JavaScript errors, and is especially great for finding cross browser JavaScript issues, as we run our acceptance tests in 5 different browsers.