Finding a home for your content

I’ve been blogging on WordPress.com for over nine years and I’m very happy with my decision to become both a WordPress.com user back then, and a WordPress.com (Automattic) employee in 2015.

Continue reading “Finding a home for your content”

Testing in Production, Oops, No ‘Undo’

Earlier today I accidentally published a test blog post to this site (titled ‘d’) which meant that the ~450 email subscribers to this site would have received that empty blog post via email. I am really sorry about that: I’ll explain below how it happened, and why it shouldn’t happen again.

The way we do testing at WordPress.com is we test new designs and features against ‘production’ backend WordPress sites, which are typically test sites/blogs we have set up under our own WordPress.com user accounts, or test specific accounts we may also use. As with any testing that touches something in production, there’s some risks involved and this morning I accidentally published a test blog post to this WatirMelon site, which I avoid using for testing purposes.

Like many unfortunate events: the reason it happened was a combination of a few different things: I was testing a new mobile editing feature which happened to have an issue where you can’t see the site you are publishing to when you click publish – so I had no visual feedback that I was using this WatirMelon site. I was also using Chrome dev tools to inspect the DOM at the time, and I thought it was in select mode, when it actually wasn’t – the Chrome UI doesn’t differentiate these modes very well. Finally, clicking publish on WordPress.com is (currently) an irreversible action: as soon as I clicked it I realised I had stuffed up but had no way to undo my actions: the emails were sent, it was too late.

I have been advocating internally an option that allows a grace period to undo the publication of a post or page, similar to GMail’s life saving ‘undo send’ feature, for some time, and I recently raised this as a public WordPress.com enhancement request.

This is to address the ‘publish anxiety’ I have (and I imagine many others also have) knowing that clicking a single button is irreversible.

Allowing user actions to be undone, or emphasising ones that are irreversible, is part of Jakob Nielson’s classic 10 usability heuristics.

User control and freedom: users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

In the meantime, I wrote a GreaseMonkey/TamperoMonkey UserScript that offers an additional confirmation dialog box when clicking “Publish” on WordPress.com. This extra step should be enough to stop me accidentally clicking Publish like I did today and clogging your inboxes with junk.

My UserScript is available as a Gist if you’re interested in it.

And sorry again.

Running Automated Tests with A/B Testing

Like a lot of modern, data driven sites, WordPress.com uses A/B testing extensively to introduce new features. These tests may be as simple as a label change or as complex as changing the entire sign up flow, for example by offering a free trial.

Since I have been working on a set of automated end-to-end tests for WordPress.com, I have found A/B testing to be problematic for automated testing on this very fast moving codebase, namely:

  1. Automated tests need to be deterministic: having a randomised experiment as an A/B test means the first test run may get an entirely different sign up flow than a second test run which is very hard to automate; and
  2. Automated tests need to know which experiments are running otherwise they may encounter unexpected behaviour randomly.

What we need is two methods to deal with A/B tests when running automated tests:

  1. We need to be able to see which A/B tests are active and compare this to a known list of expected A/B tests – so that we don’t suddenly encounter some unexpected/random behaviour for some of our test runs
  2. We need to be able to set the desired behaviour to the control group so that are our tests are deterministic.

Different sites conduct A/B testing using different tools and approaches, WordPress.com uses HTML5 local storage to set which A/B tests are active and which group the user belongs to.

Luckily it’s easy to read and update local storage using WebDriver and JavaScript. This means our approach is to:

  1. Each time a page object is initialised, there is a call on the base page model that checks the A/B tests that are active using something like return window.localStorage.ABTests; and then compares this to the known list of A/B tests which are checked in as a config item. This fails the test if there’s a new A/B test introduced that isn’t in the list of known tests. This is better than not knowing about the A/B test and failing based upon some non-deterministic behaviour.
  2. When a new A/B test is introduced and we wish to ensure our automated tests always use the control group, we can set this using a similar method window.localStorage.setItem('ABTests','{"flow":"default"}'); and refresh the page.

Ideally it would be good to know and plan every A/B test for our automated e2e tests, but since this isn’t possible, checking against known A/B tests and ensuring control groups are set means our automated tests are at least more consistent and deterministic, and fail a lot faster and more consistently when a new A/B test has been introduced.

How do you deal with non-determinism with A/B tests?