Executing JS in IE11 using WebDriverJs

We write our e2e tests in JavaScript running on Node.js which allows us to use newer JavaScript/ECMAScript features like template literals.

We have a subset of our e2e tests – mainly signing up as a new customer – which we run a few times a day against Internet Explorer 11: our lowest supported IE version.

I recently added a function that sets a cookie to set the currency for a customer:

setCurrencyForPayments( currency ) {
  const setCookieCode = function( currencyValue ) {
    window.document.cookie = `landingpage_currency=${ currencyValue };domain=.wordpress.com`;
  }
return this.driver.executeScript( setCookieCode, currency );
}

This code works perfectly when executing against Chrome or Firefox, but when it came to executing against IE11 I would see the following (rather unhelpful) error:

Uncaught JavascriptError: JavaScript error (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 69 milliseconds

I couldn’t work out what was causing this so I decided to take a break. On my break I realised that WebDriverJs is trying to execute a new JavaScript feature (template literals) against an older browser that doesn’t support it! Eureka!

So all I had to do was update our code to:

setCurrencyForPayments( currency ) {
  const setCookieCode = function( currencyValue ) {
    window.document.cookie = 'landingpage_currency=' + currencyValue + ';domain=.wordpress.com';
  }
return this.driver.executeScript( setCookieCode, currency );
}

and all our tests were happy against IE11 again 😊

Having not found a lot about this error or the cause I am hoping this blog post can help someone else out if they encounter this issue also.

AMA: JavaScript in Test Automation

Max asks…

First of all thanks for sharing all your insights on this blog. I regularly try to come back to your blog and I helped me grow as a QA Engineer quite a lot.
I wanted to ask you on your opinion on a seemingly new generation of test tools for the web that are written in JavaScript to help deal with all the asynchronicity on modern front end frameworks. Our company is in the process of redesigning our website and it seems that it just gets harder and harder to deal with all the JavaScript in test automation. I recently started looking into some other tool like cypress.io, but as of now they still seem quite immature.
Could you help me and point me in the right direction?

My response…

I think there’s two parts to the JavaScript asynchronicity issue which are not necessarily related.

Modern JavaScript front-end web frameworks (like React and Angular) are designed to be asynchronous and fast and therefore this can cause issues when trying to write automated tests against them. I don’t believe writing automated tests in an asynchronous way actually makes the tests easier to write, or more maintainable or robust, but just a result of writing these tests in the same way the web frameworks work.

You can write synchronous tests (like using Watir/Ruby) against asynchronous web interfaces, you just need to use the waiting/polling mechanisms (or write/extend your own) – the same as you need to do in asynchronous test automation tools.

We choose WebDriverJS for automated end-to-end testing of our React application as it was the official WebDriver for JavaScript project and seems to be a good choice at the time. I somewhat regret that decision now as using a synchronous third party implementation like webdriver.io seems like an easier way to write and maintain tests.

I have tried to use cypress.io but the way it controls sites (through proxies) has (current) limitations like not working on iFrames and cross-domains which are deal-breakers for our end-to-end testing needs at present.

If you don’t need to write your end-to-end tests in JavaScript I’d say avoid it unless absolutely necessary and stick to another non-asynchronous programming language.

I’m glad I’ve been able to help you grow over the years.

Puppeteer for Automated e2e Chrome Testing in Node

I recently noticed the new Google Chrome project Puppeteer:

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

As someone who only runs WebDriver tests in Google Chrome anyway, this looks like a promising project that bypasses WebDriver to have full programmatic control of Google Chrome including for automated end-to-end (e2e) tests.

The thing I really love about this is no Chromedriver dependency and how installing the library installs Chromium by default which can be controlled headlessly with zero config or any other dependencies.

You can even develop scripts using this playground.

I set up a (very basic) demo project that uses Mocha + Puppeteer and it runs on CircleCI with zero config. Awesome.

AMA: Moving automated tests from Java to JavaScript

Anonymous asks…

I am currently using a BDD framework with Cucumber, Selenium and Java for automating a web application. I used page factory to store the objects and using them in java methods I wanted to replace the java piece of code with javaScript like mocha or webdriverio. could you share your thoughts on this? can I still use page factory to maintain objects and use them in js files

My response…

What’s the reasoning for moving to JavaScript from Java? Despite having common names, there’s very little otherwise in common (Car is to Carpet as Java is to JavaScript.)

I wouldn’t move for moving sake since I see no benefit in writing BDD style web tests in JavaScript, if anything, e2e automated tests are much harder to write in JavaScript/Node because everything is asynchronous and so you have to deal with promises etc. which is much harder to do than just using Java (or Ruby).

Aside: I still dream of writing e2e tests in Ruby: it’s just so pleasant. But our new user interface is written extensively in JavaScript (React) so it makes sense from a sustainability point of view to use JS over Ruby.

 

Minesweepers Anonymous

Once upon a time there lived a couple who loved playing massively multiplayer online role-playing video games with each other (World of Warcraft or Destiny or something). The thing about these games is that they’re a bit like exercise/fitness where you need to keep playing them to keep your score high, and if you don’t you play them (or exercise) enough your score (or fitness) starts dropping which makes it much less fun overall.

So playing these games all went well for the couple, for a while.

Continue reading “Minesweepers Anonymous”

Handling JavaScript alerts when leaving a page with WebDriver

You’ve most probably seen the sometimes-useful-but-often-annoying browser alerts when navigating away from a page:JavaScript onbeforeunload alert

How do we deal with these using WebDriver?

Continue reading “Handling JavaScript alerts when leaving a page with WebDriver”

AMA: Learning JavaScript automation & starting e2e tests

Edwin asks…

How should someone approach learning JavaScript strictly for the purpose of test automation? Should they learn it the way a front-end developer would or is there a better way to learn it for purposes of writing automation?

anil also asks…

I need to do automation by using mocha chai framework with selenum webdriver. can you please guide me or suggest any site or video on how to start and what are the basic things need to know.

Continue reading “AMA: Learning JavaScript automation & starting e2e tests”

AMA: more details about our JS e2e tests

Satyajit Malugu asks…

Saw your recent post on JS vs Ruby. As a recent migrant from Ruby to JS for mobile test automation – couple of questions 1) how are you doing page objects (ES6 works too) 2) Are you able to run your E2E tests in parallel? 3)Is there anything like binding.pry for debugging JS whilst execution?)

My response…

  1. This post explains how we’re doing our page objects in ES6, and since all our e2e tests are open source, you can see for yourself :)
  2. We use CircleCI which supports parallel test execution, which I enabled in the past, but it’s currently disabled as it was consuming too many build containers which are shared across our organization. Since we have a lot more containers available now I have it on my list to renable these.
  3. I believe WebStorm supports debugging Node Mocha tests; I’ve been meaning to investigate this over console.log statements; I’ll report back when I get around to it.

AMA: Mocha this.timeout Duplication

 Alex C asks…

I notice that the “this.timeout” code is duplicated across the before and test suite. This seems mentally uncomfortable to me and the logical question is: “Why duplicate code that is in a before method?” Right now, the only thing I can think would be an explanation is “because it’s just a workaround to how the Javascript binding does things.”

import assert from 'assert';
import webdriver from 'selenium-webdriver';
import test from 'selenium-webdriver/testing';
import config from 'config';
import RalphSaysPage from '../lib/ralph-says-page.js';

let driver;

const mochaTimeoutMS = config.get( 'mochaTimeoutMS' );

test.before( function() {
	this.timeout( mochaTimeoutMS );
	driver = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() ).build();
} );

test.describe( 'Ralph Says', function() {
	this.timeout( mochaTimeoutMS );

	test.it( 'shows a quote container', function() {
		var page = new RalphSaysPage( driver, true );
		page.quoteContainerPresent().then( function( present ) {
			assert.equal( present, true, 'Quote container not displayed' );
		} );
	} );

	test.it( 'shows a non-empty quote', function() {
		var page = new RalphSaysPage( driver, true );
		page.quoteTextDisplayed().then( function( text ) {
			assert.notEqual( text, '', 'Quote is empty' );
		} );
	} );
} );

test.afterEach( () => driver.manage().deleteAllCookies() );

test.after( () => driver.quit() );

My response…

Mocha has a low default timeout (2000ms) as it wasn’t really written to be used for e2e testing. In the way I’ve written the above code, all the four test. blocks don’t share any instance state, so setting this.timeout in one will not effect the other, hence why I wrote it that way.

Having a few months experience behind me, I can see it’s easy to rewrite the above tests to avoid this duplication (and to have less lines of code):

import assert from 'assert';
import webdriver from 'selenium-webdriver';
import test from 'selenium-webdriver/testing';
import config from 'config';
import RalphSaysPage from '../lib/ralph-says-page.js';

let driver;

const mochaTimeoutMS = config.get( 'mochaTimeoutMS' );

test.describe( 'Ralph Says', function() {
	this.timeout( mochaTimeoutMS );

	test.before( function() {
		driver = new webdriver.Builder().withCapabilities( webdriver.Capabilities.chrome() ).build();
	} );

	test.it( 'shows a quote container', function() {
		var page = new RalphSaysPage( driver, true );
		page.quoteContainerPresent().then( function( present ) {
			assert.equal( present, true, 'Quote container not displayed' );
		} );
	} );

	test.it( 'shows a non-empty quote', function() {
		var page = new RalphSaysPage( driver, true );
		page.quoteTextDisplayed().then( function( text ) {
			assert.notEqual( text, '', 'Quote is empty' );
		} );
	} );

	test.afterEach( () => driver.manage().deleteAllCookies() );

	test.after( () => driver.quit() );
} );

But you still would have to duplicate this across different test files though, even though it’s stored as a config value. I’ve since realised Mocha allows you to globally override this timeout using test/mocha.opts.

AMA: R.Y.O. Page Objects 2.0

Michael Karlovich asks…

Do you have any updated thoughts on rolling your own page objects with Watir? The original post is almost 4 years old but is still the basis (loosely) of every page object framework I’ve built since then.

My response…

Wow, I can’t believe that post is almost four years old. I have also have used this for the basis of every page object framework I have built since then.

I recently had a look at our JavaScript (ES2015) code of page objects and despite ES2015 not having meta-programming support like ruby, our classes are remarkably similar to what I was proposing ~4 years ago.

I believe this is because some patterns are classic and therefore almost timeless, they can be applied over and over again to different contexts. There’s a huge amount of negativity towards best practices of late, but I could seriously say that page objects are a best practise for test automation of ui systems, which isn’t saying they will be exactly the same in every context, but there’s a common best-practice pattern there which you most likely should be using.

Page objects, as a pattern, typically:

  • Inherit from a base page object/container which stores common actions like:
    • instantiating the object looking for a known element that defines that page’s existence
    • optionally allow a ‘visit’ to the page during instantiation using some defined URL/path
    • provides actions and properties common to all pages, for example: waiting for the page, checking the page is displayed, getting the title/url of the page, and checking cookies and local storage items for that page;
  • Define actions as methods which are ways of interacting with that page (such as logging in);
  • Do not expose internals about the page outside the page – for example they typically don’t expose elements or element selectors which should only be used within actions/methods for that page which are exposed; and
  • Can also be modelled as components for user interfaces that are built using components to give greater reusability of the same components across different pages.

The biggest benefit I have found from using page objects as a pattern is having more deterministic end-to-end tests since instantiating a page I know I am on that page, so my tests will fail more reliably with a better understanding of what went wrong.

Are there any other pattern attributes you would consider vital for page objects?

AMA: Async JavaScript Tests?

David asks…

Now that you work with javascript and asynchronous code. It would be interesting to get your thoughts on asynchronous REST/HTTP calls in javascript with respect to testing, if you ever get around to that. I’m familiar with using it in web UIs on client side, but haven’t wrapped my head around that on the test/server side in node.js. I have so far opted to use a synchronous HTTP request library in node.js when I can.

My response…

This is an interesting question David. Whilst WebDriverJs and Mocha etc. are asynchronous on Node, we aren’t using any of that asynchronicity, but dealing with it using promises to resolve to a value we are looking for. It would be very cool to be able to have every WebDriver test we write truly asynchronous to spin up a browser do some testing and wait for any results all whilst running other asynchronous tests, but I believe to do this would be very complicated and pushing the limits of current technology/browser process threads etc. It would be possible if you were writing pure node/ES unit tests though.

So to answer your question we have a bunch of e2e tests written in Mocha that run sequentially one after another in a synchronous manner (using promises), even though they utilise asynchronous libraries.

AMA: bdd frameworks, api test tools & unit testing responsibility

Swapnil Waghmare asks…

Are BDD frameworks like Specflow, Cucumber better for E2E tests?

My response…

There’s two ways I can interpret this question (my additions are in bold).

Firstly:

Are BDD frameworks that use ‘Given/When/Then’ feature files like Specflow, Cucumber  better than frameworks that use ‘describe/it’ blocks like RSpec and Mocha for e2e tests?

As I previously explained, Automattic’s unit tests are written in Mocha, so that was a logical choice for writing e2e tests as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.

Secondly:

Are BDD frameworks like Specflow, Cucumber better for E2E tests than using them for automated integration, component or unit tests?

I think you could write integration tests or even unit tests in Given/When/Then format as most unit tests follow the same arrange/act/assert pattern anyhow which is exactly what Given/When/Then is.

Keep in mind there is overhead in maintaining all the step definitions and feature files for Cucumber/Specflow that give you non technical readability so if you don’t require that readability it is probably overkill. But a personal preference nonetheless.

Swapnil Waghmare also asks…

Which tools have you used for API testing, which ones would you recommend?

My response…

At the moment my main focus is on automated unit test and automated e2e test coverage in JavaScript.

I try to keep things simple, so in the past when I’ve written REST integration tests I’ve just called them using in built libraries, and have used Postman quite a lot for manual testing and debugging.

Swapnil Waghmare finally asks…

What is the reason behind writing E2E tests using JavaScript? Isn’t any Object oriented language a better choice for writing E2E tests? Should QA’s write Unit tests?

My response…

 

Firstly, JavaScript is actually object oriented, it’s just not class-based object oriented like Java, C# or Ruby. The newer versions of JavaScript, called ECMAScript, or ESScript are more class-based object oriented.

I’ve actually already answered why I chose JavaScript in a previous answer, so I’ll summarise that here and you can read the rest in that answer.

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for WordPress.com where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

As for whether QA’s should write unit tests? No, I don’t think so, as I believe unit tests should drive software development, and writing code by writing unit tests is much easier than trying to add unit tests after by someone else, as the original code will not likely be very testable as it wasn’t written with testability in mind. One benefit of code written with unit tests at the time is that it will mostly be better code as the tests are consuming your API that you are developing.

AMA: JS vs Ruby

Butch Mayhew asks…

I have noticed you blogging more about JS frameworks. How do these compare to Watir/Ruby? Would you recommend one over the other?

My response…

I had a discussion recently with Chuck van der Linden about this same topic as he has a lot of experience with Watir and is now looking at JavaScript testing frameworks like I have done.

Some Background

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for WordPress.com where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

Whilst I see merit in both views: I still think having your automated acceptance tests in the same language as your application leads to better maintainability and adoptability.

I still think writing automated acceptance tests in Ruby is much cleaner and nicer than JavaScript Node tests, particularly as Ruby allows meta-programming which means page objects can be implemented really neatly.

The JavaScript/NodeJS landscape is still very immature where people are using various tools/frameworks/compilers and certain patterns or de facto standards haven’t really emerged yet. The whole ES6/ES2015/ES2016 thing is very confusing to newcomers like me, especially on NodeJS where some ES6+ features are supported, but others require something like Babel to compile your code.

But generally with the direction ES is going, writing page objects as classes is much nicer than using functions for everything as in ES5.

Whilst there’s nothing I have found that is better (or even as good) in JavaScript/Mocha/WebDriverJS than Ruby/RSpec/Watir-WebDriver, I still think it’s a better long term decision for WordPress.com to use the JavaScript NodeJS stack for our e2e tests.

AMA: JavaScript & Mobile App Automation Tools

Justin Watts asks…

Have you looked at Chimp.js ? Any thoughts? I was very closely following your posts on picking a new framework for Automattic and I was quite surprised when you chose an async tool and stepped away from Cucumber. As a ruby dev looking to get comfortable with testing web-apps in JavaScript, I am torn on what library to dig into.

My response…

I had a quick look at Chimp.js when I was evaluating tools, it looks quite impressive, but quite opinionated and integrated, so I find these types of libraries are good to start with, but quickly become frustrating when you want to start doing advanced things (like the custom Slack reporter we just wrote that pings slack when tests fail with screenshots). I am curious about how Chimp.js is synchronous when the underlying WebDriver.IO tool uses promises?

I think the asynchronous thing is less of a big deal than most people make out. Once you get used to using promises whenever querying a value from the browser, then you just do that. I have set up my own es6 page objects so these take away most of this complication anyway.

The one synchronous tool I did seriously contemplate was webdriver-sync but since it wraps the underlying WebDriver Java driver, it was quite heavyweight as you need Java, the node-java bridge, and I couldn’t get this working on CircleCI which is what Automattic’s uses as its CI platform.

As for stepping away from Cucumber. Automattic’s unit tests are written in Mocha, so that was a logical choice as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.

Justin Watts also asks…

Appium seems to be getting more unstable as time goes on. Early Grey caught my eye. Do you have any thoughts / outlooks on the mobile app testing landscape?

My response…

I must say I haven’t had a lot of recent experience with mobile UI automation tools. A couple of years back we started using Appium but abandoned this effort as the tests were so fragile we could never get green builds, so we focused on writing unit tests for the specific mobile code, testing webviews from webdriver, and doing human end to end testing on real devices. This worked well, and unless I worked on complex native apps I would probably end up doing something similar again should this arise.

 

 

Testing end-to-end with Mocha

As part of my excellent Excellence Wrangler role at Automattic, one of my key tasks has been establishing some end-to-end tests for WordPress.com using Mocha with WebDriverJs. Our testing pyramid doesn’t look much like a pyramid:

wordpress.com test pyramid.png

We’ve got lots of React unit tests at the bottom: these are to speed development.

We’re intentionally missing a middle: the REST API we consume has its own unit tests, we don’t need integration tests for it. We don’t have detailed full stack acceptance tests of our UI: these are too slow and brittle.

We have a handful of e2e flow tests at the top, these are to protect the user experience, we run these on every deployment and frequently in production. These can be brittle on such a fast moving code base, but we limit their number (depth) so they still give us good confidence everything is working well together but limiting our overhead.

So what do these end-to-end tests look like?

I hadn’t used Mocha before and I was used to writing end-to-end tests in Gherkin format in tools like Cucumber and Specflow so I initially began writing end-to-end tests that looked like this:

test.describe('WordPress.com Sign Up', function() {
  test.beforeEach(function() {
    driver.manage().deleteAllCookies();
  });

  test.it('Can Create A Free Blog', function() {
    var signupFlow = new SignUpFlow( driver, 'desktop' );
    signupFlow.createFreeBlog( 'en' );
  });

  test.it('Can Create A New Site With a Paid Domain Upgrade', function() {
    var signupFlow = new SignUpFlow( driver, 'desktop' );
    signupFlow.CreateSiteWithDomainPaidByCreditCard( 'en' );
  });
});

I was pushing the code down into flow classes which I have used before, but the issue with this was the output I was getting from Mocha wasn’t very rich:

WordPress.com Sign Up
      ✓ Can Create A Free Blog
      ✓ Can Create A New Site With a Paid Domain Upgrade

I then realized by looking at an end-to-end test written by another developer that you can nest describe and it statements to give you much more expressive end-to-end tests.

test.describe( `Sign Up (${screenSize})`, function() {

  test.describe( 'Free Site:', function() {
    test.before( 'Delete Cookies and Local Storage', function() {
      driverManager.clearCookiesAndDeleteLocalStorage( driver );
    } );

    test.describe( 'Sign up for a free site', function() {

      test.describe( 'Step One: Themes', function() {
        test.before( 'Can see the choose a theme page as the starting page', function() {
          this.chooseAThemePage = new ChooseAThemePage( driver, { visit: true } );
          return this.chooseAThemePage.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a theme start page is not displayed' );
          } );
        } );

        test.it( 'Can select the first theme', function() {
          return this.chooseAThemePage.selectFirstTheme();
        } );
      } );

      test.describe( 'Step Two: Domains', function() {
        test.before( 'Can then see the domains page ', function() {
          this.findADomainComponent = new FindADomainComponent( driver );
          return this.findADomainComponent.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a domain page is not displayed' );
          } );
        } );

        test.it( 'Can search for a blog name', function() {
          return this.findADomainComponent.searchForBlogNameAndWaitForResults( blogName );
        } );

        test.it( 'Can see a free WordPress.com blog address in results ', function() {
          return this.findADomainComponent.freeBlogAddress().then( ( actualAddress ) => {
            assert.equal( actualAddress, expectedBlogAddress, 'The expected free address is not shown' )
          } );
        } );

        test.it( 'Can select the free address', function() {
          return this.findADomainComponent.selectFreeAddress();
        } );
      } );

This gives us rich feedback:

Sign Up (desktop)
    Free Site:
      Sign up for a free site
        Step One: Themes
          ✓ Can see the choose a theme page as the starting page
          ✓ Can select the first theme
        Step Two: Domains
          ✓ Can then see the domains page
          ✓ Can search for a blog name
          ✓ Can see a free WordPress.com blog address in results
          ✓ Can select the free address

The mistake I had made which I didn’t realize was not creating enough nesting, instead of having Step One, Step Two etc. next to one another, they should be nested within each other. This is because if they’re next to one another, Mocha will run the Step Two, Step Three etc. even if Step One has failed, which is not what we want in an end-to-end scenario where each step is dependent on the previous one.

So, it now looks something like this:

test.describe( 'Free Site:', function() {
    test.before( 'Delete Cookies and Local Storage', function() {
      driverManager.clearCookiesAndDeleteLocalStorage( driver );
    } );

    test.describe( 'Sign up for a free site', function() {
      test.describe( 'Step One: Themes', function() {
        test.before( 'Can see the choose a theme page as the starting page', function() {
          this.chooseAThemePage = new ChooseAThemePage( driver, { visit: true } );
          return this.chooseAThemePage.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a theme start page is not displayed' );
          } );
        } );

        test.it( 'Can select the first theme', function() {
          return this.chooseAThemePage.selectFirstTheme();
        } );

        test.describe( 'Step Two: Domains', function() {
          test.before( 'Can then see the domains page ', function() {
            this.findADomainComponent = new FindADomainComponent( driver );
            return this.findADomainComponent.displayed().then( ( displayed ) => {
              assert.equal( displayed, true, 'The choose a domain page is not displayed' );
            } );
          } );

          test.it( 'Can search for a blog name', function() {
            return this.findADomainComponent.searchForBlogNameAndWaitForResults( blogName );
          } );

          test.it( 'Can see a free WordPress.com blog address in results ', function() {
            return this.findADomainComponent.freeBlogAddress().then( ( actualAddress ) => {
              assert.equal( actualAddress, expectedBlogAddress, 'The expected free address is not shown' )
            } );
          } );

          test.it( 'Can select the free address', function() {
            return this.findADomainComponent.selectFreeAddress();
          } );

          test.describe( 'Step Three: Plans', function() {

which means the output is slightly different but still very useful:

Sign Up (mobile)
  Free Site:
    Sign up for a free site
      Step One: Themes
        ✓ Can select the first theme
        Step Two: Domains
          ✓ Can search for a blog name
          ✓ Can see a free WordPress.com blog address in results
          ✓ Can select the free address
          Step Three: Plans
            ✓ Can select the free plan

These tests are much better written this way. The only issue I am left facing with Mocha is when a before hook fails (such as logging in) the generic afterEach hook we have to take screenshots is not triggered (this is only triggered when an it block is run.

Running Automated Tests with A/B Testing

Like a lot of modern, data driven sites, WordPress.com uses A/B testing extensively to introduce new features. These tests may be as simple as a label change or as complex as changing the entire sign up flow, for example by offering a free trial.

Since I have been working on a set of automated end-to-end tests for WordPress.com, I have found A/B testing to be problematic for automated testing on this very fast moving codebase, namely:

  1. Automated tests need to be deterministic: having a randomised experiment as an A/B test means the first test run may get an entirely different sign up flow than a second test run which is very hard to automate; and
  2. Automated tests need to know which experiments are running otherwise they may encounter unexpected behaviour randomly.

What we need is two methods to deal with A/B tests when running automated tests:

  1. We need to be able to see which A/B tests are active and compare this to a known list of expected A/B tests – so that we don’t suddenly encounter some unexpected/random behaviour for some of our test runs
  2. We need to be able to set the desired behaviour to the control group so that are our tests are deterministic.

Different sites conduct A/B testing using different tools and approaches, WordPress.com uses HTML5 local storage to set which A/B tests are active and which group the user belongs to.

Luckily it’s easy to read and update local storage using WebDriver and JavaScript. This means our approach is to:

  1. Each time a page object is initialised, there is a call on the base page model that checks the A/B tests that are active using something like return window.localStorage.ABTests; and then compares this to the known list of A/B tests which are checked in as a config item. This fails the test if there’s a new A/B test introduced that isn’t in the list of known tests. This is better than not knowing about the A/B test and failing based upon some non-deterministic behaviour.
  2. When a new A/B test is introduced and we wish to ensure our automated tests always use the control group, we can set this using a similar method window.localStorage.setItem('ABTests','{"flow":"default"}'); and refresh the page.

Ideally it would be good to know and plan every A/B test for our automated e2e tests, but since this isn’t possible, checking against known A/B tests and ensuring control groups are set means our automated tests are at least more consistent and deterministic, and fail a lot faster and more consistently when a new A/B test has been introduced.

How do you deal with non-determinism with A/B tests?

ES2015 Page Classes

As mentioned yesterday, I am updating my WebDriverJs and Mocha demo to use new ES2015 features.

ES2015 supports classes more elegantly than using prototype based ones in older versions of JavaScript.

For example, this old class:

var webdriver = require('selenium-webdriver');
var until = webdriver.until;
var config = require('config');

RalphSaysPage = function RalphSaysPage(driver, visit) {
	this.driver = driver;
	this.url = config.get('url');
	this.explicitWaitMS = config.get('explicitWaitMS');
	this.quoteSelector = webdriver.By.id('quote');
	if (visit === true) {
		this.driver.get(this.url);
	}
	this.driver.wait(until.elementLocated(this.quoteSelector), this.explicitWaitMS);
};

RalphSaysPage.prototype.quoteContainerPresent = function() {
	var d = webdriver.promise.defer();
	this.driver.isElementPresent(this.quoteSelector).then(function(present) {
		d.fulfill(present);
	});
	return d.promise;
};

RalphSaysPage.prototype.quoteTextDisplayed = function() {
	var d = webdriver.promise.defer();
	this.driver.findElement(this.quoteSelector).getText().then(function(text) {
		d.fulfill(text);
	});
	return d.promise;
};

module.exports = RalphSaysPage;

can be better written as a ES2015 class:

import webdriver from 'selenium-webdriver';
import config from 'config';

export default class RalphSaysPage {
	constructor( driver, visit = false ) {
		this.driver = driver;
		this.url = config.get('ralphURL');
		this.explicitWaitMS = config.get('explicitWaitMS');
		this.quoteSelector = webdriver.By.id('quote');

		if (visit) this.driver.get(this.url);

		this.driver.wait(webdriver.until.elementLocated(this.quoteSelector), this.explicitWaitMS);
	}
	quoteContainerPresent() {
		return this.driver.isElementPresent(this.quoteSelector);
	}
	quoteTextDisplayed() {
		return this.driver.findElement(this.quoteSelector).getText();
	}
}

You can see I have also simplified the functions quoteContainerPresent() and quoteTextDisplayed() to directly return the webDriver promise instead of creating our own which is unnecessary.

When I introduce another page class:

import webdriver from 'selenium-webdriver';
import config from 'config';

const by = webdriver.By;
const until = webdriver.until;

export default class WebDriverJsDemoPage {
	constructor( driver, visit = false ) {
		this.driver = driver;
		this.url = config.get('demoURL');
		this.explicitWaitMS = config.get('explicitWaitMS');
		this.expectedElementSelector = by.id('elementappearsparent');

		if (visit) this.driver.get(this.url);

		this.driver.wait(webdriver.until.elementLocated(this.expectedElementSelector), this.explicitWaitMS);
	}
	waitForChildElementToAppear() {
		return this.driver.wait(until.elementLocated(by.id('elementappearschild')), this.explicitWaitMS, 'Could not locate the child element within the time specified');
	}
	childElementPresent() {
		return this.driver.isElementPresent(by.id('elementappearschild'));
	}
}

you can see I have duplicated some common functionality such as the navigation and page waiting across these two classes. This is where we can an ES2015 parent (or base) page class to inherit from.

Our base class might look something like:

export default class BasePage {
	constructor( driver, expectedElementSelector, visit = false, url = null ) {
		this.explicitWaitMS = config.get('explicitWaitMS');
		this.driver = driver;
		this.expectedElementSelector = expectedElementSelector;
		this.url = url;

		if (visit) this.driver.get(this.url);

		this.driver.wait(webdriver.until.elementLocated(this.expectedElementSelector), this.explicitWaitMS);
	}
}

which means our page classes are much nicer now:

export default class RalphSaysPage extends BasePage {
	constructor( driver, visit = false ) {
		const quoteSelector = webdriver.By.id('quote');
		super(driver, quoteSelector, visit, config.get('ralphURL'));
		this.quoteSelector = quoteSelector;
	}
	quoteContainerPresent() {
		return this.driver.isElementPresent(this.quoteSelector);
	}
	quoteTextDisplayed() {
		return this.driver.findElement(this.quoteSelector).getText();
	}
}

export default class WebDriverJsDemoPage extends BasePage {
	constructor( driver, visit = false ) {
		super(driver, by.id('elementappearsparent'), visit, config.get('demoURL'));
	}
	waitForChildElementToAppear() {
		return this.driver.wait(until.elementLocated(by.id('elementappearschild')), this.explicitWaitMS, 'Could not locate the child element within the time specified');
	}
	childElementPresent() {
		return this.driver.isElementPresent(by.id('elementappearschild'));
	}
}

This gives us the ability to add any common functionality across pages (such as checking the page title) quickly and easily without duplication.

 

 

 

WebDriverJs & Mocha in ES2015

A friend of mine, Mark Ryall, recently created a fork of my WebDriverJs and Mocha example project and updated it to use ES2015. I’ve made some further changes and merged these in, and would like to share these.

Background

JavaScript is an implementation of the ECMAScript scripting language standard.

The latest version of ECMAScript, known as ES2015, ES6, ES6 Harmony, ECMAScript 2015, or ECMAScript 6, has some neat features which are handy to use for our WebDriverJs & Mocha tests I have previously written about.

It seems that there will be yearly releases of the ECMAScript standard from 2015 onwards, and the most common way to refer to these will be as ES2015, ES2016 etc.

Enabling ES2015 Support for our Example Tests

There is a node tool called Babel which is a JavaScript compiler that allows you to use new ECMAScript features and compile these into JavaScript. This requires two node packages which we add to our package.json file:

"babel-core": "^6.3.13",
"babel-preset-es2015": "^6.3.13"

This means we have a babel compiler and a babel library to transform ES2015.

The second thing we need to do is add a plugin to actually tell babel to transform ES2015.

We add a .babelrc file to our project with the following content:

{
"presets": ["es2015"]
}

Running our Specs using Babel

Once we’ve done this, we can use Mocha and WebDriverJs with ES2015. Instead of calling mocha specs we now need to use babel like:
mocha --compilers js:babel-core/register specs.

This isn’t as nice, so we can update our package.json file so our test command is set to the longer babel command, and we just need to call npm test to run our Mocha specs.

Updating our code to use ES2015

The great thing about ES2015 is it is backwards compatible, so we don’t need to update all our code at once, we can made gradual changes to use new features available to us.

Mark made changes to the spec and the page object to use some of the pretty ES2015 features:

Import Statements

This:

var assert = require('assert');
var webdriver = require('selenium-webdriver');
var test = require('selenium-webdriver/testing');
var config = require('config');
var RalphSaysPage = require('../lib/ralph-says-page.js');

Becomes:

import assert from 'assert';
import webdriver from 'selenium-webdriver';
import test from 'selenium-webdriver/testing';
import config from 'config';
import { ralphSays } from '../lib/pages.js';

Using let instead of var

let is block scoped so this is better to use.

This:

var driver;

Becomes:

let driver = null;

Arrow functions

The arrow functions make the clean up hooks simpler to read:

From this:

test.afterEach(function() {
  driver.manage().deleteAllCookies();
});

To this:

test.afterEach(() => driver.manage().deleteAllCookies());

Summary

Moving to use ES2015 wasn’t as daunting as I initially thought as once you add support for it using Babel, you can gradually start using the new features.