WordPress.com e2e Automated Tests Now Open Source

I am very pleased to announce that all of our e2e tests for the WordPress.com platform are open source as of this morning. This is following in the footsteps of the WordPress.com Calypso front-end which is also open source.

I am continually reminded of how fortunate I am to work at Automattic who takes pride in its commitment to Default to Open:

“The quality of our code and our product depend on the amount of feedback we get and on the amount of people who use them. If we’re developing behind closed doors, we are putting artificial limits to both.

We have done our best work in the open, let’s continue working this way.”

~ Matt Mullenweg, CEO Automattic

Our ongoing development of these tests is in the open. So please feel free to take a look through our e2e tests, their CI results, and fork them, provide us some feedback, or use some of our ideas. And pull requests are always welcome 😀

Running Automated Tests with A/B Testing

Like a lot of modern, data driven sites, WordPress.com uses A/B testing extensively to introduce new features. These tests may be as simple as a label change or as complex as changing the entire sign up flow, for example by offering a free trial.

Since I have been working on a set of automated end-to-end tests for WordPress.com, I have found A/B testing to be problematic for automated testing on this very fast moving codebase, namely:

  1. Automated tests need to be deterministic: having a randomised experiment as an A/B test means the first test run may get an entirely different sign up flow than a second test run which is very hard to automate; and
  2. Automated tests need to know which experiments are running otherwise they may encounter unexpected behaviour randomly.

What we need is two methods to deal with A/B tests when running automated tests:

  1. We need to be able to see which A/B tests are active and compare this to a known list of expected A/B tests – so that we don’t suddenly encounter some unexpected/random behaviour for some of our test runs
  2. We need to be able to set the desired behaviour to the control group so that are our tests are deterministic.

Different sites conduct A/B testing using different tools and approaches, WordPress.com uses HTML5 local storage to set which A/B tests are active and which group the user belongs to.

Luckily it’s easy to read and update local storage using WebDriver and JavaScript. This means our approach is to:

  1. Each time a page object is initialised, there is a call on the base page model that checks the A/B tests that are active using something like return window.localStorage.ABTests; and then compares this to the known list of A/B tests which are checked in as a config item. This fails the test if there’s a new A/B test introduced that isn’t in the list of known tests. This is better than not knowing about the A/B test and failing based upon some non-deterministic behaviour.
  2. When a new A/B test is introduced and we wish to ensure our automated tests always use the control group, we can set this using a similar method window.localStorage.setItem('ABTests','{"flow":"default"}'); and refresh the page.

Ideally it would be good to know and plan every A/B test for our automated e2e tests, but since this isn’t possible, checking against known A/B tests and ensuring control groups are set means our automated tests are at least more consistent and deterministic, and fail a lot faster and more consistently when a new A/B test has been introduced.

How do you deal with non-determinism with A/B tests?

Comparison of JavaScript browser automation and test specification libraries

As part of my trial for my current role at Automattic, I was tasked with implementing some e2e acceptance tests using my choice of library/framework/language.

I very much recommend writing automated acceptance tests in the same language as your app, even though I have described some benefits of using a different language, and since WordPress is moving towards JavaScript from PHP, JavaScript seems the most suitable language for Automattic.

Continue reading “Comparison of JavaScript browser automation and test specification libraries”

Bush Walking (or how a wasp can destroy your iPhone)

Everyone at the Automattic Grand Meetup is required to give a 4 minute (or less) flash talk about any topic they like. I told a story about how I was stung by a wasp bush walking. This was my talk:

Bush Walking by Alister Scott (16)

Continue reading “Bush Walking (or how a wasp can destroy your iPhone)”

Automattic Grand Meetup

My first few weeks working at Automattic (WordPress.com) have been great. As I mentioned previously; I spent my first three weeks on customer support (like everyone else), and this week I am in Park City, Utah for our annual all company meetup (Automattic Grand Meetup).

Since Automattic is a 100% distributed company with about 400 staff across 36 countries, it’s important to have face-to-face contact a few times a year, and this is the biggest meetup where everyone from across the globe comes together. Other meetups throughout the year are for teams or projects and are much smaller.

Park City is a very pretty place, I’ve enjoyed my time here. Here’s some of my photos.