Scheduling CircleCI Jobs

We use CircleCI to run our automated end-to-end (e2e) tests for WordPress.com.

We run our tests pretty frequently – not only against every individual change coming through – but we also run them in Production every time someone deploys (about 30 times per day) – as well as every 6 hours to cover weekends and quiet periods of deployments to make sure our hardware and other changes haven’t impacted our key customer flows.

Originally CircleCI didn’t support scheduled jobs so we set up our own infrastructure to schedule jobs to call the CircleCI API which executed the tests.

Fortunately version 2.0 of the CircleCI config now natively supports scheduling jobs which is exactly what we want to do. Since it also uses cron, the default scheduling format, it was very easy to create our jobs in CircleCI.

This is what our .circleci/config.yml looks like:

Before – no scheduling

version: 2
jobs:
  test:
    docker:
      - image: circleci/node:10.5.0-browsers
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
            - v1-npmcache-{{ checksum ".nvmrc" }}
            - v1-npmcache
      - run: npm ci
      - save_cache:
          key: v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
          paths:
            - "~/.npm"
      - run:
          name: Execute the e2e tests
          command: npm test

After – with scheduling

version: 2
jobs:
  test:
    docker:
      - image: circleci/node:10.5.0-browsers
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
            - v1-npmcache-{{ checksum ".nvmrc" }}
            - v1-npmcache
      - run: npm ci
      - save_cache:
          key: v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
          paths:
            - "~/.npm"
      - run:
          name: Execute the e2e tests
          command: npm test
workflows:
  version: 2
  commit:
    jobs:
      - test
  nightly:
    triggers:
      - schedule:
          cron: "0 0 * * *"
          filters:
            branches:
              only:
                - master
    jobs:
      - test

The thing I love about CircleCI is that config is checked in as code so it’s super easy to manage and track changes to the config over time. My sample project is available here.

AMA: Protractor for e2e Testing?

John X asks…

 I have dodged the AngularJS /Protractor bullet for many years now. May last foray, some 5 years ago, was a cluster to put it mildly! Cucumberjs/Angularjs/Protractor/chai/mocha/ the stack was in its infancy and failed miserably!! Many cycles were spent pulling fixes from our own repos instead of waiting for PRs to get done. This was in stark contrast to the ease and stability of the automation I wrote using Watir-Webdriver and eventually Watir.

I am now faced with automating the regression test cases for an Angularjs App.

Question: Do I finally jump back into the using stack that almost caused me to lose my mind, or is it possible to use Watir/Selenium to build out meaningful e2e UI automation for an angularjs app as we dawn on 2019?

My response…

It’s still my opinion in 2018 that writing e2e tests in Node using either Protractor or WebDriverJs is still more difficult than using Watir in Ruby.

Sure using async functions with await commands makes things easier than before (see examples in our project), when you would have first come across Protractor, but I still feel like there’s a lot of catching up to do to get to the stability and ease-of-use of Watir.

My decision would come down to whether others on your project are going to be comfortable maintaining tests in ruby – if they are I’d use Ruby and Watir, otherwise I’d revisit Protractor if they really want to use Node.

Testbash Australia 2018

I only speak at one conference a year and this year that conference will be the first ever Australian Testbash in Sydney on October 19, 2018:

TestBash_Australia_2018_Adverts_DOJO_EVENT_BANNER.png

My talk:

At WordPress.com we constantly deliver changes to our millions of customers – in the past month alone we released our React web client 563 times; over 18 releases every day. We don’t conduct any manual regression testing, and we only employ 5 software testers in a company of ~680 people with ~230 developers across . So, how do we ensure our customers get a consistently great user experience with all this rapid change?

Our automated end-to-end (e2e) tests give us confidence to release small frequent changes constantly to all our different applications and platforms knowing that our customers can continue to use our products to achieve their desires.

Alister will share how these automated tests are written and work, some of the benefits and challenges we’ve seen in implementing these tests, and what we have planned for future iterations.

Takeaways: How to effectively use automated end-to-end testing to ensure a consistent user experience in high frequency delivery environments

Grab a ticket before they sell out #

AMA: JavaScript in Test Automation

Max asks…

First of all thanks for sharing all your insights on this blog. I regularly try to come back to your blog and I helped me grow as a QA Engineer quite a lot.
I wanted to ask you on your opinion on a seemingly new generation of test tools for the web that are written in JavaScript to help deal with all the asynchronicity on modern front end frameworks. Our company is in the process of redesigning our website and it seems that it just gets harder and harder to deal with all the JavaScript in test automation. I recently started looking into some other tool like cypress.io, but as of now they still seem quite immature.
Could you help me and point me in the right direction?

My response…

I think there’s two parts to the JavaScript asynchronicity issue which are not necessarily related.

Modern JavaScript front-end web frameworks (like React and Angular) are designed to be asynchronous and fast and therefore this can cause issues when trying to write automated tests against them. I don’t believe writing automated tests in an asynchronous way actually makes the tests easier to write, or more maintainable or robust, but just a result of writing these tests in the same way the web frameworks work.

You can write synchronous tests (like using Watir/Ruby) against asynchronous web interfaces, you just need to use the waiting/polling mechanisms (or write/extend your own) – the same as you need to do in asynchronous test automation tools.

We choose WebDriverJS for automated end-to-end testing of our React application as it was the official WebDriver for JavaScript project and seems to be a good choice at the time. I somewhat regret that decision now as using a synchronous third party implementation like webdriver.io seems like an easier way to write and maintain tests.

I have tried to use cypress.io but the way it controls sites (through proxies) has (current) limitations like not working on iFrames and cross-domains which are deal-breakers for our end-to-end testing needs at present.

If you don’t need to write your end-to-end tests in JavaScript I’d say avoid it unless absolutely necessary and stick to another non-asynchronous programming language.

I’m glad I’ve been able to help you grow over the years.

AMA: hiding/changing IP addresses using Watir

Mudit Bhardwaj asks…

Hey! In order to test the security of a website, I’m trying to create hundreds of accounts. However, after a certain limit, there is always an error which prevents me from going further.
How can I hide/change my ip address during each iteration with watir?

My response…

I can understand why there would be this limitation in place as this activity seems suspicious from a systems perspective.

Watir can’t change the IP address. You can use an anonymous browser profile and/or delete cookies for different account creation runs but this probably won’t help.

I’d speak to your development/devops/systems team to whitelist your IP address(es) you are using for this purpose.

Puppeteer for Automated e2e Chrome Testing in Node

I recently noticed the new Google Chrome project Puppeteer:

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

As someone who only runs WebDriver tests in Google Chrome anyway, this looks like a promising project that bypasses WebDriver to have full programmatic control of Google Chrome including for automated end-to-end (e2e) tests.

The thing I really love about this is no Chromedriver dependency and how installing the library installs Chromium by default which can be controlled headlessly with zero config or any other dependencies.

You can even develop scripts using this playground.

I set up a (very basic) demo project that uses Mocha + Puppeteer and it runs on CircleCI with zero config. Awesome.