Feature Toggles for Automated e2e Tests

Feature toggles aren’t just for production code. Feature toggles are also a powerful technique to change the behaviour of your automated end-to-end tests without changing code.

Continue reading “Feature Toggles for Automated e2e Tests”

The Importance of Doing Customer Support

One of the many, many things I love about working at Automattic is the being part of a company that sees huge value in everyone spending time doing customer support, and acts on it:

“When you join full-time, you’ll do customer support for WordPress.com for your first three weeks and spend a week in support annually, for evermore, regardless of your position. We believe an early and ongoing connection with the people who use our products is irreplaceable.”

Automattic’s Work With Us Page

Every single bit of that statement is true: I did three full weeks when I joined last September, and I’ve already done an additional week this year.

Continue reading “The Importance of Doing Customer Support”

AMA: running e2e tests continually

Clark Hamilton asks…

What do you think about running e2e GUI tests continually and displaying the results on a real-time dashboard? We could build such a system but maybe there is an open-source solution we don’t know about. We’re using Mocha with webdriverio and found your articles useful. Appreciate your thoughts!

My response…

We currently do something like this for WordPress.com. We have our e2e tests (now open source) which are run in three different circumstances:

  1. on every deploy to WordPress.com (tens of times per day); and
  2. every 6 hours from UTC 0:00 (to pick up issues like connectivity that happen independent of deployments); and
  3. every time a change to the e2e tests is merged into the master branch.

We do this using CircleCI which connects to Github to perform #3 automatically. With #1 and #2 we use the CircelCI API to trigger builds on the master branch during our deploy process and every 6 hours using a cron job.

As for a dashboard, we use the CircleCI results page which shows the history of each run, and we also have some slack channels set up which also get real time notifications of results, and any tests that fail (including screenshots). The way we do this is in our source code.

Since we’re a 100% distributed company with no offices we don’t have any build lights/monitors etc. for an office space, we just individually use our own channels to monitor these test results.

WordPress.com e2e Automated Tests Now Open Source

I am very pleased to announce that all of our e2e tests for the WordPress.com platform are open source as of this morning. This is following in the footsteps of the WordPress.com Calypso front-end which is also open source.

I am continually reminded of how fortunate I am to work at Automattic who takes pride in its commitment to Default to Open:

“The quality of our code and our product depend on the amount of feedback we get and on the amount of people who use them. If we’re developing behind closed doors, we are putting artificial limits to both.

We have done our best work in the open, let’s continue working this way.”

~ Matt Mullenweg, CEO Automattic

Our ongoing development of these tests is in the open. So please feel free to take a look through our e2e tests, their CI results, and fork them, provide us some feedback, or use some of our ideas. And pull requests are always welcome 😀

Running Automated Tests with A/B Testing

Like a lot of modern, data driven sites, WordPress.com uses A/B testing extensively to introduce new features. These tests may be as simple as a label change or as complex as changing the entire sign up flow, for example by offering a free trial.

Since I have been working on a set of automated end-to-end tests for WordPress.com, I have found A/B testing to be problematic for automated testing on this very fast moving codebase, namely:

  1. Automated tests need to be deterministic: having a randomised experiment as an A/B test means the first test run may get an entirely different sign up flow than a second test run which is very hard to automate; and
  2. Automated tests need to know which experiments are running otherwise they may encounter unexpected behaviour randomly.

What we need is two methods to deal with A/B tests when running automated tests:

  1. We need to be able to see which A/B tests are active and compare this to a known list of expected A/B tests – so that we don’t suddenly encounter some unexpected/random behaviour for some of our test runs
  2. We need to be able to set the desired behaviour to the control group so that are our tests are deterministic.

Different sites conduct A/B testing using different tools and approaches, WordPress.com uses HTML5 local storage to set which A/B tests are active and which group the user belongs to.

Luckily it’s easy to read and update local storage using WebDriver and JavaScript. This means our approach is to:

  1. Each time a page object is initialised, there is a call on the base page model that checks the A/B tests that are active using something like return window.localStorage.ABTests; and then compares this to the known list of A/B tests which are checked in as a config item. This fails the test if there’s a new A/B test introduced that isn’t in the list of known tests. This is better than not knowing about the A/B test and failing based upon some non-deterministic behaviour.
  2. When a new A/B test is introduced and we wish to ensure our automated tests always use the control group, we can set this using a similar method window.localStorage.setItem('ABTests','{"flow":"default"}'); and refresh the page.

Ideally it would be good to know and plan every A/B test for our automated e2e tests, but since this isn’t possible, checking against known A/B tests and ensuring control groups are set means our automated tests are at least more consistent and deterministic, and fail a lot faster and more consistently when a new A/B test has been introduced.

How do you deal with non-determinism with A/B tests?