Adventures of end-to-end automated web testing in Node.js

This is a presentation I gave at ATTAC in Melbourne on Friday 24th May 2019. The full Google Slides are available here.

Good morning everyone and thanks for having me along to speak.
My name is Alister Scott and I’m from Brisbane. I work as a QA generalist at a software company in Brisbane called Console, and I write a blog called WatirMelon. Today I’m going to be sharing my knowledge of end to end automated testing in Node.js. I’ve created very simple but working demos of the main tools I’ll be discussing today and I’ve put these on GitHub as separate repositories you can easily clone and play around with: github.com/alisterscott

Outside of work I enjoy hiking, often to the summits of mountains, and spending time with my wife and our three young kids.
In 2015 I started a paid trial at Automattic. My trial project – on which I would be assessed to gain full time employment – was to establish an automated e2e testing framework for WordPress.com – the first of its kind for Automattic. I quickly spun up a framework using Watir in Ruby – because that’s what I knew – and it worked. However I quickly gained some feedback that of the hundred+ developers at Automattic almost none knew Ruby and creating shared ownership for e2e tests would be a key measure of success so I had to rethink. At the time Automattic was moving from primarily developing in PHP towards Node.js meaning full stack JavaScript and it made sense that the e2e tests were also developed in Node.js.

This of course made my trial project a lot more difficult than I had originally thought as I had to teach myself Node.js and understand what testing tools existed in this space.

Fast forward to 2019 and earlier this year I decided to change jobs as the requirement to travel for Automattic was no longer compatible with my values of supporting my young family. When I started looking at job advertisements for testing and QE positions in Brisbane I noticed just how many mentioned Node.js as their technology stack of choice: in the 3.5 years at Automattic Node.js had become increasingly popular at other companies as well.
But automated e2e testing in Node.js was and is really hard. Much much harder than I was used to in Ruby and Watir where things just worked.

It’s slightly better in 2019 than 2015, however there are some reasons why it’s hard.
There’s also the paradox of choice when it comes to tooling specifically for e2e web testing. Searching for selenium and WebDriver on NPM provides a list of dozens of libraries – and the official Selenium bindings (WebDriverJs) don’t appear when searching for WebDriver. It’s all very confusing.

My aim today is to create some clarity in this space.
I believe there’s more e2e test tools in Node.js that something like Java or C#. I’ll talk about the four most popular/mature ones.
Cypress also doesn’t support cross-domain stuff – so beware if you’re doing anything like that.
You’ll need your own test runner and assertion library.
I’ve distilled these four tools down into an easy to read visual.
And an even easier to understand flow chart.

Scheduling CircleCI Jobs

We use CircleCI to run our automated end-to-end (e2e) tests for WordPress.com.

We run our tests pretty frequently – not only against every individual change coming through – but we also run them in Production every time someone deploys (about 30 times per day) – as well as every 6 hours to cover weekends and quiet periods of deployments to make sure our hardware and other changes haven’t impacted our key customer flows.

Originally CircleCI didn’t support scheduled jobs so we set up our own infrastructure to schedule jobs to call the CircleCI API which executed the tests.

Fortunately version 2.0 of the CircleCI config now natively supports scheduling jobs which is exactly what we want to do. Since it also uses cron, the default scheduling format, it was very easy to create our jobs in CircleCI.

This is what our .circleci/config.yml looks like:

Before – no scheduling

version: 2
jobs:
  test:
    docker:
      - image: circleci/node:10.5.0-browsers
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
            - v1-npmcache-{{ checksum ".nvmrc" }}
            - v1-npmcache
      - run: npm ci
      - save_cache:
          key: v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
          paths:
            - "~/.npm"
      - run:
          name: Execute the e2e tests
          command: npm test

After – with scheduling

version: 2
jobs:
  test:
    docker:
      - image: circleci/node:10.5.0-browsers
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
            - v1-npmcache-{{ checksum ".nvmrc" }}
            - v1-npmcache
      - run: npm ci
      - save_cache:
          key: v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
          paths:
            - "~/.npm"
      - run:
          name: Execute the e2e tests
          command: npm test
workflows:
  version: 2
  commit:
    jobs:
      - test
  nightly:
    triggers:
      - schedule:
          cron: "0 0 * * *"
          filters:
            branches:
              only:
                - master
    jobs:
      - test

The thing I love about CircleCI is that config is checked in as code so it’s super easy to manage and track changes to the config over time. My sample project is available here.

AMA: Protractor for e2e Testing?

John X asks…

 I have dodged the AngularJS /Protractor bullet for many years now. May last foray, some 5 years ago, was a cluster to put it mildly! Cucumberjs/Angularjs/Protractor/chai/mocha/ the stack was in its infancy and failed miserably!! Many cycles were spent pulling fixes from our own repos instead of waiting for PRs to get done. This was in stark contrast to the ease and stability of the automation I wrote using Watir-Webdriver and eventually Watir.

I am now faced with automating the regression test cases for an Angularjs App.

Question: Do I finally jump back into the using stack that almost caused me to lose my mind, or is it possible to use Watir/Selenium to build out meaningful e2e UI automation for an angularjs app as we dawn on 2019?

My response…

It’s still my opinion in 2018 that writing e2e tests in Node using either Protractor or WebDriverJs is still more difficult than using Watir in Ruby.

Sure using async functions with await commands makes things easier than before (see examples in our project), when you would have first come across Protractor, but I still feel like there’s a lot of catching up to do to get to the stability and ease-of-use of Watir.

My decision would come down to whether others on your project are going to be comfortable maintaining tests in ruby – if they are I’d use Ruby and Watir, otherwise I’d revisit Protractor if they really want to use Node.

TestBash Sydney: Automated e2e Testing at WordPress.com

This is an approximate transcript of the talk I delivered at TestBash in Sydney on Friday 19th October 2018.

Alister Scott Test Bash Sydney Oct 2018-1-1

Today I’d like to share my story about how we started with automated end to end testing at WordPress.com since I started at Automattic over 3 years ago.

Alister Scott Test Bash Sydney Oct 2018 2-2

Continue reading “TestBash Sydney: Automated e2e Testing at WordPress.com”

AMA: Separate Repository for e2e Tests?

Liam asks…

“I did enjoy reading the article about e2e test on wordpress. I noted that e2e test are in a separate repo.

My question will be what is the workflow to make sure new changes does not break the e2e test on pull request ?

For example, if a developer work on some changes, then they need to change the e2e test first and make sure everything pass, however the environment on the pull request might not be stable, developer can overwrite each other changes”

My response…

Thanks for your question Liam.

We have reasons for and benefits in having the WordPress.com e2e tests in a separate repository:

  1. The e2e tests test the entire WordPress.com experience so these test things that happen in different repositories (for example our Calypso user interface or services/API) and having them in the user interface repository isn’t really representative of what the breadth of their scope;
  2. Making changes to the e2e tests are easier in a separate repository since we don’t have to deploy e2e PRs that don’t contain functional changes (we deploy every merge to our master branch immediately dozens of times per day)

The obvious downsides are:

  1. How do we make sure e2e tests know about incoming AB tests?
  2. How do we couple new changes to updates in the e2e test repository?

For incoming AB tests we make sure that our e2e tests know about the change by ensuring we create a matching PR in our e2e tests that override our AB tests during our test runs.

If someone updates the AB tests in Calypso they’re politely reminded to update the e2e tests:

Screen Shot 2018-09-19 at 3.31.47 pm
example prompt

For making sure e2e tests are up to date we automatically run two (of about 40 total) of the most critical e2e tests (in three browsers) when a PR is ready to be reviewed. These can fail and indicate a change is necessary to the e2e tests (or something is broken!)

There’s also a label we can add to any PR that runs the entire set of e2e tests against a PR running live and reports the result back to that PR:

Screen Shot 2018-09-19 at 3.38.29 pm
e2e Test Results against a Calypso PR

If changes are required to the e2e tests someone can create an e2e PR with the exact same branch name which will be used to run against the feature changes before they are merged. This means PRs can be coupled and tested together but merged separately.

To answer the second part of your question I understand it to be about conflicting changes? One of our key philosophies for work is “merge early and merge often” so we make sure that PRs are short-lived and merged quickly to minimize the chance of conflicts. These still do happen occasionally, we just deal with them as they come up.

Whilst there’s been some downsides to the separate repositories all-in-all the benefits continue to outweigh the downsides but we constantly assess and at any point in time we can easily merge them if need be.

npm ci

I recently discovered npm ci which you can use instead of npm install when running a node project on continuous integration (CI) system and want to install your npm dependencies. It does this in a more lightweight, more CI friendly way.

If you use npm test to run your tests, this can be shortened to npm t (much like npm i is npm install), and therefore you can run npm cit to install dependencies and run tests in CI.