AMA: Protractor for e2e Testing?

John X asks…

 I have dodged the AngularJS /Protractor bullet for many years now. May last foray, some 5 years ago, was a cluster to put it mildly! Cucumberjs/Angularjs/Protractor/chai/mocha/ the stack was in its infancy and failed miserably!! Many cycles were spent pulling fixes from our own repos instead of waiting for PRs to get done. This was in stark contrast to the ease and stability of the automation I wrote using Watir-Webdriver and eventually Watir.

I am now faced with automating the regression test cases for an Angularjs App.

Question: Do I finally jump back into the using stack that almost caused me to lose my mind, or is it possible to use Watir/Selenium to build out meaningful e2e UI automation for an angularjs app as we dawn on 2019?

My response…

It’s still my opinion in 2018 that writing e2e tests in Node using either Protractor or WebDriverJs is still more difficult than using Watir in Ruby.

Sure using async functions with await commands makes things easier than before (see examples in our project), when you would have first come across Protractor, but I still feel like there’s a lot of catching up to do to get to the stability and ease-of-use of Watir.

My decision would come down to whether others on your project are going to be comfortable maintaining tests in ruby – if they are I’d use Ruby and Watir, otherwise I’d revisit Protractor if they really want to use Node.

npm ci

I recently discovered npm ci which you can use instead of npm install when running a node project on continuous integration (CI) system and want to install your npm dependencies. It does this in a more lightweight, more CI friendly way.

If you use npm test to run your tests, this can be shortened to npm t (much like npm i is npm install), and therefore you can run npm cit to install dependencies and run tests in CI.

Running Mocha e2e Tests in Parallel

I recently highlighted the importance of e2e test design. Once you have well designed e2e tests you can start running them in parallel.

There are a couple of approaches to scaling your tests out to be run in parallel:

  1. Running the tests in multiple machine processes;
  2. Running the tests across multiple (virtual) machines;

These aren’t mutually exclusive, you can run tests in parallel processes across multiple virtual machines – we do this at Automattic – each test run happens across two virtual machines, Docker containers on CircleCI, each of which runs six processes for either desktop or mobile responsive browser sizes depending on the container.

I have found running tests in multiple processes gives best bang for buck since you don’t need additional machines (most build systems charge based on container utilisation) and you’ll benefit from parallel runs on a local machine.

We write our e2e tests in WebDriverJs and use Mocha for our test framework. We currently use a tool called Magellan to run our e2e tests in separate processes (workers), however the tool is losing Mocha support and therefore we need to look at alternatives.

I have found that mocha-parallel-tests seems like the best replacement – it’s a drop in replacement runner for mocha tests which splits test specification files across processes available on your machine you’re executing your tests on – you can also specify a maximum limit of processes as the command line argument --max-parallel

There is another parallel test runner for mocha: mocha.parallel – but this requires updating all your specs to use a different API to allow the parallelisation to work. I like mocha-parallel-tests as it just works.

I’ve updated my webdriver-js-demo project to use mocha-parallel-tests – feel free to check it out here.

Bailing with Mocha e2e Tests

At Automattic we use Mocha to write our end-to-end (e2e) automated tests in JavaScript/Node.js. One issue with Mocha is that it’s not really a tool suited to writing e2e tests where one test step can rely on a previous test step – for example our sign up process is a series of pages/steps which rely on the previous step passing. Mocha is primarily a unit testing tool and it’s bad practice for one unit test to depend on another, so that is why Mocha doesn’t support this.

A more simplified example of this is shown in my webdriver-js-demo project:

describe( 'Ralph Says', function() {
	this.timeout( mochaTimeoutMS );

	before( async function() {
		const builder = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() );
		driver = await builder.build();
	} );

	it( 'Visit the page', async function() {
		page = await RalphSaysPage.Visit( driver );
	} );

	it( 'shows a quote container', async function() {
		assert( await page.quoteContainerPresent(), 'Quote container not displayed' );
	} );

	it( 'shows a non-empty quote', async function() {
		assert.notEqual( await page.quoteTextDisplayed(), '', 'Quote is empty' );
	} );

	afterEach( async function() { await driver.manage().deleteAllCookies(); } );

	after( async function() { await driver.quit(); } );
} );

Continue reading “Bailing with Mocha e2e Tests”

Executing JS in IE11 using WebDriverJs

We write our e2e tests in JavaScript running on Node.js which allows us to use newer JavaScript/ECMAScript features like template literals.

We have a subset of our e2e tests – mainly signing up as a new customer – which we run a few times a day against Internet Explorer 11: our lowest supported IE version.

I recently added a function that sets a cookie to set the currency for a customer:

setCurrencyForPayments( currency ) {
  const setCookieCode = function( currencyValue ) {
    window.document.cookie = `landingpage_currency=${ currencyValue };domain=.wordpress.com`;
  }
return this.driver.executeScript( setCookieCode, currency );
}

This code works perfectly when executing against Chrome or Firefox, but when it came to executing against IE11 I would see the following (rather unhelpful) error:

Uncaught JavascriptError: JavaScript error (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 69 milliseconds

I couldn’t work out what was causing this so I decided to take a break. On my break I realised that WebDriverJs is trying to execute a new JavaScript feature (template literals) against an older browser that doesn’t support it! Eureka!

So all I had to do was update our code to:

setCurrencyForPayments( currency ) {
  const setCookieCode = function( currencyValue ) {
    window.document.cookie = 'landingpage_currency=' + currencyValue + ';domain=.wordpress.com';
  }
return this.driver.executeScript( setCookieCode, currency );
}

and all our tests were happy against IE11 again 😊

Having not found a lot about this error or the cause I am hoping this blog post can help someone else out if they encounter this issue also.

AMA: JavaScript in Test Automation

Max asks…

First of all thanks for sharing all your insights on this blog. I regularly try to come back to your blog and I helped me grow as a QA Engineer quite a lot.
I wanted to ask you on your opinion on a seemingly new generation of test tools for the web that are written in JavaScript to help deal with all the asynchronicity on modern front end frameworks. Our company is in the process of redesigning our website and it seems that it just gets harder and harder to deal with all the JavaScript in test automation. I recently started looking into some other tool like cypress.io, but as of now they still seem quite immature.
Could you help me and point me in the right direction?

My response…

I think there’s two parts to the JavaScript asynchronicity issue which are not necessarily related.

Modern JavaScript front-end web frameworks (like React and Angular) are designed to be asynchronous and fast and therefore this can cause issues when trying to write automated tests against them. I don’t believe writing automated tests in an asynchronous way actually makes the tests easier to write, or more maintainable or robust, but just a result of writing these tests in the same way the web frameworks work.

You can write synchronous tests (like using Watir/Ruby) against asynchronous web interfaces, you just need to use the waiting/polling mechanisms (or write/extend your own) – the same as you need to do in asynchronous test automation tools.

We choose WebDriverJS for automated end-to-end testing of our React application as it was the official WebDriver for JavaScript project and seems to be a good choice at the time. I somewhat regret that decision now as using a synchronous third party implementation like webdriver.io seems like an easier way to write and maintain tests.

I have tried to use cypress.io but the way it controls sites (through proxies) has (current) limitations like not working on iFrames and cross-domains which are deal-breakers for our end-to-end testing needs at present.

If you don’t need to write your end-to-end tests in JavaScript I’d say avoid it unless absolutely necessary and stick to another non-asynchronous programming language.

I’m glad I’ve been able to help you grow over the years.

Puppeteer for Automated e2e Chrome Testing in Node

I recently noticed the new Google Chrome project Puppeteer:

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

As someone who only runs WebDriver tests in Google Chrome anyway, this looks like a promising project that bypasses WebDriver to have full programmatic control of Google Chrome including for automated end-to-end (e2e) tests.

The thing I really love about this is no Chromedriver dependency and how installing the library installs Chromium by default which can be controlled headlessly with zero config or any other dependencies.

You can even develop scripts using this playground.

I set up a (very basic) demo project that uses Mocha + Puppeteer and it runs on CircleCI with zero config. Awesome.