Finding a home for your content

I’ve been blogging on WordPress.com for over nine years and I’m very happy with my decision to become both a WordPress.com user back then, and a WordPress.com (Automattic) employee in 2015.

Continue reading “Finding a home for your content”

Testing in Production, Oops, No ‘Undo’

Earlier today I accidentally published a test blog post to this site (titled ‘d’) which meant that the ~450 email subscribers to this site would have received that empty blog post via email. I am really sorry about that: I’ll explain below how it happened, and why it shouldn’t happen again.

The way we do testing at WordPress.com is we test new designs and features against ‘production’ backend WordPress sites, which are typically test sites/blogs we have set up under our own WordPress.com user accounts, or test specific accounts we may also use. As with any testing that touches something in production, there’s some risks involved and this morning I accidentally published a test blog post to this WatirMelon site, which I avoid using for testing purposes.

Like many unfortunate events: the reason it happened was a combination of a few different things: I was testing a new mobile editing feature which happened to have an issue where you can’t see the site you are publishing to when you click publish – so I had no visual feedback that I was using this WatirMelon site. I was also using Chrome dev tools to inspect the DOM at the time, and I thought it was in select mode, when it actually wasn’t – the Chrome UI doesn’t differentiate these modes very well. Finally, clicking publish on WordPress.com is (currently) an irreversible action: as soon as I clicked it I realised I had stuffed up but had no way to undo my actions: the emails were sent, it was too late.

I have been advocating internally an option that allows a grace period to undo the publication of a post or page, similar to GMail’s life saving ‘undo send’ feature, for some time, and I recently raised this as a public WordPress.com enhancement request.

This is to address the ‘publish anxiety’ I have (and I imagine many others also have) knowing that clicking a single button is irreversible.

Allowing user actions to be undone, or emphasising ones that are irreversible, is part of Jakob Nielson’s classic 10 usability heuristics.

User control and freedom: users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

In the meantime, I wrote a GreaseMonkey/TamperoMonkey UserScript that offers an additional confirmation dialog box when clicking “Publish” on WordPress.com. This extra step should be enough to stop me accidentally clicking Publish like I did today and clogging your inboxes with junk.

My UserScript is available as a Gist if you’re interested in it.

And sorry again.

Running Automated Tests with A/B Testing

Like a lot of modern, data driven sites, WordPress.com uses A/B testing extensively to introduce new features. These tests may be as simple as a label change or as complex as changing the entire sign up flow, for example by offering a free trial.

Since I have been working on a set of automated end-to-end tests for WordPress.com, I have found A/B testing to be problematic for automated testing on this very fast moving codebase, namely:

  1. Automated tests need to be deterministic: having a randomised experiment as an A/B test means the first test run may get an entirely different sign up flow than a second test run which is very hard to automate; and
  2. Automated tests need to know which experiments are running otherwise they may encounter unexpected behaviour randomly.

What we need is two methods to deal with A/B tests when running automated tests:

  1. We need to be able to see which A/B tests are active and compare this to a known list of expected A/B tests – so that we don’t suddenly encounter some unexpected/random behaviour for some of our test runs
  2. We need to be able to set the desired behaviour to the control group so that are our tests are deterministic.

Different sites conduct A/B testing using different tools and approaches, WordPress.com uses HTML5 local storage to set which A/B tests are active and which group the user belongs to.

Luckily it’s easy to read and update local storage using WebDriver and JavaScript. This means our approach is to:

  1. Each time a page object is initialised, there is a call on the base page model that checks the A/B tests that are active using something like return window.localStorage.ABTests; and then compares this to the known list of A/B tests which are checked in as a config item. This fails the test if there’s a new A/B test introduced that isn’t in the list of known tests. This is better than not knowing about the A/B test and failing based upon some non-deterministic behaviour.
  2. When a new A/B test is introduced and we wish to ensure our automated tests always use the control group, we can set this using a similar method window.localStorage.setItem('ABTests','{"flow":"default"}'); and refresh the page.

Ideally it would be good to know and plan every A/B test for our automated e2e tests, but since this isn’t possible, checking against known A/B tests and ensuring control groups are set means our automated tests are at least more consistent and deterministic, and fail a lot faster and more consistently when a new A/B test has been introduced.

How do you deal with non-determinism with A/B tests?

Why I won’t be using Github Gists on WordPress.com

WordPress.com recently announced the ability to embed Github Gists (snippets of code) into a blog post. Each Gist is essentially a git repository so you can do all sorts of neat stuff with it. Then it’s cool that you can embed these straight in your page right? Well, um, actually, no.

I started using Github Gists for all my blog posts recently, but since then I’ve noticed a couple of shortcomings. Firstly, these Gists don’t get embedded in RSS feeds and emails, so readers of your blogs via these means miss out on important content.

Secondly, even though a Gist can contain multiple files, you only insert the Gist by URL into your blog post so there’s no way to split the files up into different sections in your blog post, so you end up creating more Gists than you really need.

Lastly, and probably most importantly, you’re essentially creating an external dependency for your blog. All the code is stored on Github, which is great until it’s down, or you start using another service, or you change your username or something else. Then you have broken links and blog posts that no longer make sense.

Whilst I prefer the look of the Github Gists in my blog posts (they’re prettier IMO), I think I’ll stick to the plain old embedded [ sourcecode ] tags, at least that way my code will live on with my blog posts.

Watir.com

I have spent a bit of time over the last few days setting up Watir.com, hosted here on WordPress.

We were originally aiming to host our own version of Confluence and JIRA and use Confluence to serve the Watir.com homepage, but this ended up being a lot more complicated and expensive than originally planned.

The great thing about WordPress is, although it was originally a blogging platform, its functionality also works as a very neat CMS. Whilst wordpress.com has some limitations over wordpress.org, we can live with these limitations for now as we have a free (as in beer) hosted site that the world can see.

Check it out.

watir.com