AMA: bdd frameworks, api test tools & unit testing responsibility

Swapnil Waghmare asks…

Are BDD frameworks like Specflow, Cucumber better for E2E tests?

My response…

There’s two ways I can interpret this question (my additions are in bold).


Are BDD frameworks that use ‘Given/When/Then’ feature files like Specflow, Cucumber  better than frameworks that use ‘describe/it’ blocks like RSpec and Mocha for e2e tests?

As I previously explained, Automattic’s unit tests are written in Mocha, so that was a logical choice for writing e2e tests as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.


Are BDD frameworks like Specflow, Cucumber better for E2E tests than using them for automated integration, component or unit tests?

I think you could write integration tests or even unit tests in Given/When/Then format as most unit tests follow the same arrange/act/assert pattern anyhow which is exactly what Given/When/Then is.

Keep in mind there is overhead in maintaining all the step definitions and feature files for Cucumber/Specflow that give you non technical readability so if you don’t require that readability it is probably overkill. But a personal preference nonetheless.

Swapnil Waghmare also asks…

Which tools have you used for API testing, which ones would you recommend?

My response…

At the moment my main focus is on automated unit test and automated e2e test coverage in JavaScript.

I try to keep things simple, so in the past when I’ve written REST integration tests I’ve just called them using in built libraries, and have used Postman quite a lot for manual testing and debugging.

Swapnil Waghmare finally asks…

What is the reason behind writing E2E tests using JavaScript? Isn’t any Object oriented language a better choice for writing E2E tests? Should QA’s write Unit tests?

My response…


Firstly, JavaScript is actually object oriented, it’s just not class-based object oriented like Java, C# or Ruby. The newer versions of JavaScript, called ECMAScript, or ESScript are more class-based object oriented.

I’ve actually already answered why I chose JavaScript in a previous answer, so I’ll summarise that here and you can read the rest in that answer. built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

As for whether QA’s should write unit tests? No, I don’t think so, as I believe unit tests should drive software development, and writing code by writing unit tests is much easier than trying to add unit tests after by someone else, as the original code will not likely be very testable as it wasn’t written with testability in mind. One benefit of code written with unit tests at the time is that it will mostly be better code as the tests are consuming your API that you are developing.

Comparison of JavaScript browser automation and test specification libraries

As part of my trial for my current role at Automattic, I was tasked with implementing some e2e acceptance tests using my choice of library/framework/language.

I very much recommend writing automated acceptance tests in the same language as your app, even though I have described some benefits of using a different language, and since WordPress is moving towards JavaScript from PHP, JavaScript seems the most suitable language for Automattic.

Continue reading “Comparison of JavaScript browser automation and test specification libraries”

CukeUp! Australia 2015

The first ever CukeUp! Australia is being held in Sydney on November 19 and 20, 2015.

I have been selected to speak and my talk is titled ‘Establishing a Self-Sustaining Culture of Quality at Domino’s Digital’.

Just 12 months ago Domino’s had a dedicated manual testing team who performed testing during a dedicated testing phase at the end of each project. Not only did this substantially slow down projects, releases were big and introduced lots of risk despite having been independently tested. Fast forward to today, Domino’s Digital consists of multiple cross-functional teams who are wholly responsible and accountable for quality into and beyond production through regular releases: no testing team, no testing phases, no testing manager. Alister will share the journey of moving to a self-sustaining culture of quality and detail the cosmic benefits the business has received in increasing both quality and velocity across all digital delivery initiatives.

Early-bird tickets are available now. Hoping to see those from down under there.

Five automated acceptance test anti-patterns

Whilst being involved with lots of people writing automated acceptance tests using tools like SpecFlow and WebDriver I’ve seen some ‘anti-patterns’ emerge that can make these tests non-deterministic (flaky), very fragile to change and less efficient to run.

Here’s five ‘anti-patterns’ I’ve seen and what you can do instead.

Anti-pattern One: Not using page-objects

Page objects are just a design pattern to ensure automated UI tests use reusable, modular code. Not using them, eg, writing WebDriver code directly in step definitions, means any changes to your UI will require updates in lots of different places instead of the one ‘page’ class.


[When(@"I buy some '(.*)' tea")]
public void WhenIBuySomeTea(string typeOfTea)


[When(@"I buy some '(.*)' tea")]
public void WhenIBuySomeTea(string typeOfTea)

Complicated set up scenarios within the tests themselves

Whilst there’s a place for automated end-to-end scenarios (I call these user journies), I prefer most acceptance tests to jump straight to the point.


Scenario: Accept Visa and Mastercard for Australia
 Given I am on the home page for Australia
 And I choose the tea menu
 And I select some 'green tea'
 And I add the tea to my basket
 And I choose to checkout
 Then I should see 'visa' is accepted
 And I should see 'mastercard' is accepted


This usually requires adding some special functionality to your app, but the ability for testing to ‘jump’ to certain pages with data automatically set up makes automated tests much easier to read and maintain.

Scenario: Accept Visa and Mastercard for Australia
 Given I am the checkout page for Australia
 Then I should see 'visa' is accepted
 And I should see 'mastercard' is accepted

Using complicated x-path or CSS selectors

Using element identification selectors that have long chains from the DOM in them leads to fragile tests, as any change to that chain in the DOM will break your tests.


private static readonly By TeaTypeSelector =
                "#input-tea-type > div > div.TeaSearchRow > > div:nth-child(2) > label");


Identify by ‘id’ (unique) or ‘class’. If there’s multiple elements in a group, create a parent container and iterate through them.

private static readonly By TeaTypeSelector = By.Id("teaType");

Directly executing JavaScript

Since WebDriver can directly execute any arbitrary JavaScript, it can be tempting to bypass DOM manipulation and just run the JavaScript.


public void RemoveTea(string teaType)
  (driver as IJavaScriptExecutor).ExecuteScript(string.Format("viewModel.tea.types.removeTeaType(\"{0}\");", teaType));


It is much better to let the WebDriver control the browser elements which should fire the correct JavaScript events and call the JavaScript, as that way you avoid having your ‘test’ JavaScript in sync to your ‘real’ JavaScript.

public void RemoveTea(string teaType)

Embedding implementation detail in your features/scenarios

Acceptance test scenarios are meant to convey intention over implementation. If you start seeing things like URLs in your test scenarios you’re focusing on implementation.


 Scenario: Social media links displayed on checkout page
   Given I am the checkout page for Australia
   Then I should see a link to ''
   And I should see a link to ''


Hide implementation detail in the steps (or pages, or config) and make your scenarios about the test intention.

 Scenario: Social media links displayed on checkout page
   Given I am the checkout page for Australia
   Then I should see a link to the Beautiful Tea Twitter account
   And I should see a link to the Beautiful Tea Facebook page

I hope you’ve enjoyed these anti-patterns. Leave a comment below if you have any of your own.

You probably don’t need a specification framework

I think plain language specification frameworks like SpecFlow and Cucumber are great, but have a lot of overhead and are way overused.

If you don’t have non-technical folk collaborating with you on your specifications, try writing plain automated tests instead. This means using plain NUnit/MSTest over SpecFlow in C# or minitest over Cucumber in Ruby. You’ll avoid the overhead of maintaining a plain language specification framework and be able to focus on developing a great set of tests instead.

It’s easier than you think to add a plain language specification layer to a set of well structured plain tests. So only add the specification layer when you need it, because chances are you ain’t gonna.

Five page object anti-patterns

I’ve observed some page object anti-patterns which commonly arise when starting out designing end-to-end automated tests. Chris McMahon recently asked for some feedback on his initial test spike for Wikipedia, and some of these anti-patterns were present.

Anti-pattern one: frequently opening and closing browsers

I often see both RSpec and Cucumber tests frequently opening and closing browsers. This slows down test execution times, and should be avoided unless absolutely necessary.

You can clear cookies between tests if you’re worried about state.

To open and close the browser only once in Cucumber, specify this in your env.rb file:

browser =

Before do
  @browser = browser

at_exit do

To open and close the browser only once in RSpec:

browser =

RSpec.configure do |config|
  config.before(:each) { @browser = browser }
  config.after(:suite) { browser.close }

Anti-pattern two: hard coding URLs on page classes

Chances are you’ll at some point run your automated tests in different environments, even if it’s just to verify that production has been updated correctly. If you’ve hard coded URLs in page classes, this can be problematic.

Fortunately it’s easy to avoid, by creating a module that contains base URLs which can be accessed by page classes. These base URLs can be stored in YAML files which can be switched for different environments.

module Wikipedia
  BASE_URL = ''

class BogusPage
  include PageObject
  page_url "#{Wikipedia::BASE_URL}/wiki/Bogus_page"

Anti-pattern three: pages stored as instance variables in steps or rspec specs

I don’t like seeing pages stored as instance variables (those starting with an @) in Cucumber steps or RSpec specs, as it introduces state and thus more room for error.

If you’re using the page-object gem, there are two methods available to access pages directly without using instance variables: visit_page and on_page (also visit or on from 0.6.4+). Both of these can be used as blocks, so you can perform multiple actions within these methods.

visit LoginPage do |page|
  page.login_with('foo', 'badpass')
  page.text.should include "Login error"
  page.text.should include "Secure your account"

Anti-pattern four: checking the entire page contains some text somewhere

I often see people checking that the entire web page contains some expected text. Even if the text was at the very bottom of the page hidden in the footer the test would probably pass.

You should check the text is where it should be, using a container that it should belong to. Ideally a span or a div may exist that contains the exact text, but even if it’s in a slightly larger container it is still better than asserting it exists somewhere on the page.

class BogusPage
  include PageObject
  cell :main_text, :class => 'mbox-text'

visit_page BogusPage do |page|
  page.main_text.should include 'Wikipedia does not have an article with this exact name'
  page.main_text.should include 'Other reasons this message may be displayed'

Anti-pattern five: using RSpec for end-to-end tests

This one is contentious, and I am sure I’ll get lots of opinions to the contrary, but I believe that RSpec is best suited to unit/integration tests, and Cucumber is suited to end-to-end tests.

I find I create duplication when trying to do end-to-end tests in RSpec, which is where Cucumber step definitions come in. Trying to do unit tests in Cucumber seems like too much overhead, and in my opinion is more suited to RSpec.

Turnip: trying to solve Cucumber’s problems with ruby BDD

I’ve been playing around with Turnip: a library that was designed to solve Cucumber’s problems in four main areas:

  1. Having a separate test framework is annoying
  2. Mapping steps to regexps is hard
  3. Cucumber has a huge, messy codebase
  4. Steps are always global

I don’t really care about having a Cucumber test framework as I don’t often use RSpec, I actually find regexen quite powerful, I also don’t really care about Cucumber’s code base but I can see some merit in having scoped steps.

I converted my existing demo test suite across to use Turnip and this is what I found.

Turnip follows Spinach in trying to move away from using regular expressions to define steps. Fortunately, unlike Spinach, Turnip allows you to capture values using placeholders in your steps (this was an instant NO to using Spinach):

  step "I should see at least :exp_num_results results" do |exp_num_results|
    on :results do |page|
      page.number_search_results.should >= exp_num_results.to_i

Unfortunately there are a number of limitations to this approach:

  • You can’t capture more than a single word in a placeholder without using double quotes in your feature file – making them less readable; and
  • You can’t capture values like “100,000” as it doesn’t appear to support special characters in placeholders.

You can also do some magic using custom step placeholders, where you define regular expressions, but to me that kinda defeats the point of not using regexen in the first place.

The scoped steps are quite neat and ruby like, and I can see these being very useful for large projects where multiple people are writing the steps.

Running Turnip files is as simple as using the RSpec test runner, but unfortunately the output of pending steps is less useful than Cucumber in that there are no code snippets, and only the first undefined step is displayed as undefined (unlike the Cucumber which shows you every step that is undefined).


I like the approach that Turnip takes in being a simple way to write business like tests and run them using RSpec. Unfortunately that simplicity comes at a price, and losing the power of regexen in step definitions is not something I would like to give up hastily. I am hoping that the scoped steps someday make their way into the Cucumber code base.