AMA: running e2e tests continually

Clark Hamilton asks…

What do you think about running e2e GUI tests continually and displaying the results on a real-time dashboard? We could build such a system but maybe there is an open-source solution we don’t know about. We’re using Mocha with webdriverio and found your articles useful. Appreciate your thoughts!

My response…

We currently do something like this for WordPress.com. We have our e2e tests (now open source) which are run in three different circumstances:

  1. on every deploy to WordPress.com (tens of times per day); and
  2. every 6 hours from UTC 0:00 (to pick up issues like connectivity that happen independent of deployments); and
  3. every time a change to the e2e tests is merged into the master branch.

We do this using CircleCI which connects to Github to perform #3 automatically. With #1 and #2 we use the CircelCI API to trigger builds on the master branch during our deploy process and every 6 hours using a cron job.

As for a dashboard, we use the CircleCI results page which shows the history of each run, and we also have some slack channels set up which also get real time notifications of results, and any tests that fail (including screenshots). The way we do this is in our source code.

Since we’re a 100% distributed company with no offices we don’t have any build lights/monitors etc. for an office space, we just individually use our own channels to monitor these test results.

Continuous Integration/Testing/Delivery/Deployment

Manoj Kumar K asked me via LinkedIn for some feedback on his article about the differences between Continuous Integration/Testing/Delivery and Deployment.

Here’s my response (left as a comment on the original article):

Continuous Integration is continually integrating several developer’s code, compiling, deploying and automatically testing that code (Continuous Testing), and that can be done on a branch, or in trunk/master.

When you have continual production readiness of every build, but you don’t deploy every build automatically, instead you choose to deploy say x times a week, that is Continuous Delivery.

When you automatically deploy every build to production after it’s been automatically integrated, built and tested, that is Continuous Deployment.

I’ve previously briefly explained the semantics here.

So Continuous Integration (CI) and Continuous Testing together allow Continuous Delivery. Continuous Delivery allows Continuous Deployment (should you choose to do this).

Martin Fowler explains this in more detail here.