AMA: running e2e tests continually

Clark Hamilton asks…

What do you think about running e2e GUI tests continually and displaying the results on a real-time dashboard? We could build such a system but maybe there is an open-source solution we don’t know about. We’re using Mocha with webdriverio and found your articles useful. Appreciate your thoughts!

My response…

We currently do something like this for We have our e2e tests (now open source) which are run in three different circumstances:

  1. on every deploy to (tens of times per day); and
  2. every 6 hours from UTC 0:00 (to pick up issues like connectivity that happen independent of deployments); and
  3. every time a change to the e2e tests is merged into the master branch.

We do this using CircleCI which connects to Github to perform #3 automatically. With #1 and #2 we use the CircelCI API to trigger builds on the master branch during our deploy process and every 6 hours using a cron job.

As for a dashboard, we use the CircleCI results page which shows the history of each run, and we also have some slack channels set up which also get real time notifications of results, and any tests that fail (including screenshots). The way we do this is in our source code.

Since we’re a 100% distributed company with no offices we don’t have any build lights/monitors etc. for an office space, we just individually use our own channels to monitor these test results.

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

3 thoughts on “AMA: running e2e tests continually”

  1. I am interested to know what the time taken for these runs is roughly? (given how devs constantly want test suites at build to offer quicker and quicker feedback, but as qa i want confidence), an what strategies you might have for doing this like running subsets in a smoke test?

    Liked by 1 person

    1. The functional tests take about 30 mins, and the visual regression tests add another 15 mins.
      We don’t run them before deployment, we deploy then run our tests in production as we don’t want to slow things down by blocking deployments. If something goes bad, the developer who deployed their change has to roll it back.
      I am about to enable parallel test execution in our CI environment to speed this test time up and we may look at running these against each pull request as it is created in GitHub.


Comments are closed.