AMA: bdd frameworks, api test tools & unit testing responsibility

Swapnil Waghmare asks…

Are BDD frameworks like Specflow, Cucumber better for E2E tests?

My response…

There’s two ways I can interpret this question (my additions are in bold).

Firstly:

Are BDD frameworks that use ‘Given/When/Then’ feature files like Specflow, Cucumber  better than frameworks that use ‘describe/it’ blocks like RSpec and Mocha for e2e tests?

As I previously explained, Automattic’s unit tests are written in Mocha, so that was a logical choice for writing e2e tests as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.

Secondly:

Are BDD frameworks like Specflow, Cucumber better for E2E tests than using them for automated integration, component or unit tests?

I think you could write integration tests or even unit tests in Given/When/Then format as most unit tests follow the same arrange/act/assert pattern anyhow which is exactly what Given/When/Then is.

Keep in mind there is overhead in maintaining all the step definitions and feature files for Cucumber/Specflow that give you non technical readability so if you don’t require that readability it is probably overkill. But a personal preference nonetheless.

Swapnil Waghmare also asks…

Which tools have you used for API testing, which ones would you recommend?

My response…

At the moment my main focus is on automated unit test and automated e2e test coverage in JavaScript.

I try to keep things simple, so in the past when I’ve written REST integration tests I’ve just called them using in built libraries, and have used Postman quite a lot for manual testing and debugging.

Swapnil Waghmare finally asks…

What is the reason behind writing E2E tests using JavaScript? Isn’t any Object oriented language a better choice for writing E2E tests? Should QA’s write Unit tests?

My response…

 

Firstly, JavaScript is actually object oriented, it’s just not class-based object oriented like Java, C# or Ruby. The newer versions of JavaScript, called ECMAScript, or ESScript are more class-based object oriented.

I’ve actually already answered why I chose JavaScript in a previous answer, so I’ll summarise that here and you can read the rest in that answer.

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for WordPress.com where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

As for whether QA’s should write unit tests? No, I don’t think so, as I believe unit tests should drive software development, and writing code by writing unit tests is much easier than trying to add unit tests after by someone else, as the original code will not likely be very testable as it wasn’t written with testability in mind. One benefit of code written with unit tests at the time is that it will mostly be better code as the tests are consuming your API that you are developing.

Five automated acceptance test anti-patterns

Whilst being involved with lots of people writing automated acceptance tests using tools like SpecFlow and WebDriver I’ve seen some ‘anti-patterns’ emerge that can make these tests non-deterministic (flaky), very fragile to change and less efficient to run.

Here’s five ‘anti-patterns’ I’ve seen and what you can do instead.

Anti-pattern One: Not using page-objects

Page objects are just a design pattern to ensure automated UI tests use reusable, modular code. Not using them, eg, writing WebDriver code directly in step definitions, means any changes to your UI will require updates in lots of different places instead of the one ‘page’ class.

Bad

[When(@"I buy some '(.*)' tea")]
public void WhenIBuySomeTea(string typeOfTea)
{
Driver.FindElement(By.Id("tea-"+typeOfTea)).Click();
Driver.FindElement(By.Id("buy")).Click();
}

Better

[When(@"I buy some '(.*)' tea")]
public void WhenIBuySomeTea(string typeOfTea)
{
     MenuPage.BuyTea(typeOfTea);
}

Complicated set up scenarios within the tests themselves

Whilst there’s a place for automated end-to-end scenarios (I call these user journies), I prefer most acceptance tests to jump straight to the point.

Bad

Scenario: Accept Visa and Mastercard for Australia
 Given I am on the home page for Australia
 And I choose the tea menu
 And I select some 'green tea'
 And I add the tea to my basket
 And I choose to checkout
 Then I should see 'visa' is accepted
 And I should see 'mastercard' is accepted

Better

This usually requires adding some special functionality to your app, but the ability for testing to ‘jump’ to certain pages with data automatically set up makes automated tests much easier to read and maintain.

Scenario: Accept Visa and Mastercard for Australia
 Given I am the checkout page for Australia
 Then I should see 'visa' is accepted
 And I should see 'mastercard' is accepted

Using complicated x-path or CSS selectors

Using element identification selectors that have long chains from the DOM in them leads to fragile tests, as any change to that chain in the DOM will break your tests.

Bad

private static readonly By TeaTypeSelector =
            By.CssSelector(
                "#input-tea-type > div > div.TeaSearchRow > div.TeaSearchCell.no > div:nth-child(2) > label");

Better

Identify by ‘id’ (unique) or ‘class’. If there’s multiple elements in a group, create a parent container and iterate through them.

private static readonly By TeaTypeSelector = By.Id("teaType");

Directly executing JavaScript

Since WebDriver can directly execute any arbitrary JavaScript, it can be tempting to bypass DOM manipulation and just run the JavaScript.

Bad

public void RemoveTea(string teaType)
{
  (driver as IJavaScriptExecutor).ExecuteScript(string.Format("viewModel.tea.types.removeTeaType(\"{0}\");", teaType));
  }

Better

It is much better to let the WebDriver control the browser elements which should fire the correct JavaScript events and call the JavaScript, as that way you avoid having your ‘test’ JavaScript in sync to your ‘real’ JavaScript.

public void RemoveTea(string teaType)
{
  driver.FindElement(By.Id("remove-"+teaType)).Click();
}

Embedding implementation detail in your features/scenarios

Acceptance test scenarios are meant to convey intention over implementation. If you start seeing things like URLs in your test scenarios you’re focusing on implementation.

Bad


 Scenario: Social media links displayed on checkout page
   Given I am the checkout page for Australia
   Then I should see a link to 'http://twitter.com/beautifultea'
   And I should see a link to 'https://facebook.com/beautifultea'
 

Better

Hide implementation detail in the steps (or pages, or config) and make your scenarios about the test intention.


 Scenario: Social media links displayed on checkout page
   Given I am the checkout page for Australia
   Then I should see a link to the Beautiful Tea Twitter account
   And I should see a link to the Beautiful Tea Facebook page
 

I hope you’ve enjoyed these anti-patterns. Leave a comment below if you have any of your own.

You probably don’t need a specification framework

I think plain language specification frameworks like SpecFlow and Cucumber are great, but have a lot of overhead and are way overused.

If you don’t have non-technical folk collaborating with you on your specifications, try writing plain automated tests instead. This means using plain NUnit/MSTest over SpecFlow in C# or minitest over Cucumber in Ruby. You’ll avoid the overhead of maintaining a plain language specification framework and be able to focus on developing a great set of tests instead.

It’s easier than you think to add a plain language specification layer to a set of well structured plain tests. So only add the specification layer when you need it, because chances are you ain’t gonna.

Don’t bury your hooks

A slightly technical post here.

If you’re using a BDD framework such as Specflow or JBehave, can I please ask that you don’t bury your hooks. These frameworks provide hooks, or events, you can use to repeat certain things. An example in Specflow is:

[BeforeScenario]
public static void GoToHomePage()
{
Driver.Navigate().GoToUrl(GoogleUrl);
}

You can put these hooks in any steps class, but please, put them in one place, all together, preferably in a class/file named hooks or the like.

I recently came across a code base where these hooks where spread across lots of different steps files, which makes it very confusing and hard to debug as you don’t know where all this code is being called from when you’re running your tests.

C#: Avoiding the WebDriverException: No response from server for url

When it comes to automated testing, there’s not much worse than intermittent failures, especially when they stem from the driver itself. The current version of the C# WebDriver bindings has such a failure, but I worked out a reasonable way to avoid it happening. Basically it involves creating a WebDriver extension method that I use instead of Driver.FindElement, which tries a number of times to find the element, ignoring the exception that is intermittently raised.

I hope you find this useful if you’re consuming WebDriver in C#.

using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;
namespace Extensions
{
    public static class WebDriverExtensions
    {
        public static SelectElement GetSelectElement(this IWebDriver driver, By by)
        {
            return new SelectElement(driver.GetElement(by));
        }
        public static IWebElement GetElement(this IWebDriver driver, By by)
        {
            for (int i = 1; i <= 5; i++ )
            {
                try
                {
                    return driver.FindElement(by);
                }
                catch (Exception e)
                {
                    Console.WriteLine("Exception was raised on locating element: " + e.Message);
                }
            }
            throw new ElementNotVisibleException(by.ToString());
        }
    }
}

SpecDriver now includes Etsy.com examples

A fellow ThoughtWorker, Paul Hammant, recently announced his intent for QuickTest Professional. And his method? Provide some new technology tutorials for newbies, in the form of a series of open source automated tests for craft site Etsy.com.

He asked me to assist by providing some SpecDriver examples, and since I enjoy doing this stuff I am more than happy to help out.

So, I’ve updated the SpecDriver repository on Github to include some Etsy.com feature files, and working tests. It was a lot of fun to write these tests, because Etsy.com is awesomely testable. Its nicely formatted code with proper ids and class names makes automating against it a real treat.

The benefit of doing this exercise is that it’s meaty enough for me to get some insight into how other people code real automated tests. For example, I’ve had a look into a couple of the code samples and I see a lot of XPath selectors. I personally really dislike XPath stuff, just like I dislike XML as I find it hard to read, and less intuitive than using straight identifiers. I am proud that there isn’t a single XPath used in my Etsy.com examples.

Another thing that stood out to me was how embedded some of the stories are. For example, in the JBehave example the features are located under //etsy-stories/src/main/resources/stories, which took me some time to locate at first, and every time I forget where they are. I much prefer having a features folder in root that makes it pretty obvious what’s in there.

One thing I am very interested in is seeing how people specify tests. I found two of Paul’s scenarios repetitious so I moved them into a single scenario outline. Moving them into a scenario outline made me think about some different ways to test, so I added a new scenario, which didn’t require much more code, but strengthened my feature.

For example, Paul’s two scenarios:


Scenario: Advanced Search for a hat
  Given I am searching on Etsy.com
  When I specify the Knitting sub category
  And I search for hat
  Then there are search results

Scenario: Advanced Search for a ring
  Given I am searching on Etsy.com
  When I specify the Jewelry sub category
  And I search for ring
  Then there are search results

became my:

Scenario Outline: Advanced Search for items in various categories that exist and don't exist
  Given I am searching on Etsy.com
    When I specify the <Sub Category> sub category
    And I search for <Search Term>
    Then I should see <Search Results> search results for <Search Term> 

Examples:
  | Sub Category | Search Term  | Search Results |
  | Knitting     | 'hat'        | some           |
  | Jewelry      | 'necklace'   | some           |
  | Jewelry      | 'specdriver' | no             | 

While I was at it, I noticed there was a spelling correction feature of the advanced search. I also thought it was a good idea to test that this was working correctly.


Scenario: Misspelling a word corrects search automatically
  Given I am searching on Etsy.com
  When I specify the Knitting sub category
  And I search for 'scalf'
  Then I should see some search results for 'scarf'
  And I should see that the search was for 'scarf' instead of 'scalf'

Now I’ve done this exercise using SpecFlow and WebDriver in C#, I’m keen to do it again using Cucumber & Watir (WebDriver) in Ruby, to compare the implementation. When I am done I will post it on here for you all to see. Enjoy.

SpecDriver: A simple, open-source, page object model framework for C# automated web testing

There doesn’t seem to be a lot of material available in the C# .NET automated testing space, so I thought I would create and share my own page object model centered framework: SpecDriver.

It uses SpecFlow to define features, scenarios and steps, and then WebDriver to actually drive the browser to automate these steps, with a page object model in between to ensure maximum maintainability of the solution.

I have previously documented the steps to getting SpecFlow up and running with Visual Studio C# Express which is free to use for both commercial and non-commercial reasons. You should follow these if you would like to play around with SpecDriver.

You can access all the source code on the github repository, and please feel free to fork/improve it as you see fit.

I will explain the various elements here and how they fit together.

Feature Files for Test Specifications

SpecFlow uses .feature files exactly the same as Cucumber, so it’s pretty easy to create these. I actually used my feature files from my Cucumber framework for this example; the benefits of writing these in a non-technical style!

Feature: Google Search
  As a casual internet user
  I want to find some information about watir, and do a quick conversion
  So that I can be knowledgeable being

Scenario: Search for Watir
  Given I am on the Google Home Page
  When I search for "Watir"
  Then I should see at least 100,000 results

Scenario: Do a unit conversion
  Given I am on the Google Home Page
  When I convert 10 cm to inches
  Then I should see the conversion result "10 centimetres = 3.93700787 inches"

Scenario: Do a search using data specified externally
  Given I am on the Google Home Page
  When I search for a ridiculously small number of results
  Then I should see at most 100 results

Step Definitions that call page objects

The step definitions are small, granular methods that call methods on page objects and do assertions against expected results.

namespace Project1.StepDefinitions
{
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using TechTalk.SpecFlow;
    using NUnit.Framework;
    using Project1.Pages;

    [Binding]
    public class GoogleSearchStepDefinitions : BaseStepDefinitions
    {

        [Given(@"I am on the Google Home Page")]
        public void GivenIAmOnTheGoogleHomePage()
        {
            GoogleHomePage = new GoogleHomePageModel(Driver);
            Assert.AreEqual("Google", GoogleHomePage.Title);
        }

        [When(@"I search for ""(.+)""")]
        public void WhenISearchForSomething(string searchTerm)
        {
            GoogleSearchResultsPage = GoogleHomePage.Search(searchTerm);
        }

        [When(@"I search for a ridiculously small number of results")]
        public void WhenISearchForARidiculouslySmallNumberOfResults()
        {
            GoogleSearchResultsPage = GoogleHomePage.Search("macrocryoglobulinemia marvel");
        }

        [When(@"I convert (.+)")]
        public void WhenIConvertSomething(string ConversionString)
        {
            GoogleSearchResultsPage = GoogleHomePage.Search("convert " + ConversionString);
        }

        [Then(@"I should see at most ([\d,]+) results")]
        public void ThenIShouldSeeAtMostNumberOfResults(string expMaxNumberResults)
        {
            expMaxNumberResults = expMaxNumberResults.Replace(",", "");
            Assert.LessOrEqual(Convert.ToInt32(GoogleSearchResultsPage.NumberOfResults), Convert.ToInt32(expMaxNumberResults));
        }

        [Then(@"I should see at least ([\d,]+) results")]
        public void ThenIShouldSeeAtLeastNumberOfResults(string expMinNumberResults)
        {
            expMinNumberResults = expMinNumberResults.Replace(",", "");
            Assert.GreaterOrEqual(Convert.ToInt32(GoogleSearchResultsPage.NumberOfResults), Convert.ToInt32(expMinNumberResults));
        }

        [Then(@"I should see the conversion result ""(.+)""")]
        public void ThenIShouldSeeTheConversionResult(string expectedConversionResult)
        {
            Assert.AreEqual(expectedConversionResult, GoogleSearchResultsPage.ConversionResult);
        }
    }
}

Page Object Model

Each page in the application under test is represented by a page class (that inherits from a base page class), and this page class has elements and methods associated with it. The pages are the things that actually use WebDriver to interact with browsers. You can see, like my ruby page object pattern, methods that change pages return an instance of that new page. The other thing to note is that there is a base page class that has a constructor, that requires an known element to instantiate the page. This is a way of knowing where you are in your application and constantly checking it is in the right place. This also ensures consistent syncronization, especially when pages contain dynamic content such as AJAX calls.

namespace Project1.Pages
{
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using OpenQA.Selenium;

    public class GoogleHomePageModel : BasePageModel
    {
        private static readonly By SearchEditSelector = By.Name("q");
        private static readonly By SearchButtonSelector = By.Name("btnG");

        public GoogleHomePageModel(IWebDriver driver)
            : base(driver, SearchEditSelector)
        {
        }

        private IWebElement SearchEditElement
        {
             get { return Driver.FindElement(SearchEditSelector); }
        }

        private IWebElement SearchButtonElement
        {
            get { return Driver.FindElement(SearchButtonSelector); }
        }

        public GoogleSearchResultsPageModel Search(string term)
        {
            this.SearchEditElement.Set(term);
            this.SearchEditElement.SendKeys(Keys.Escape);
            this.SearchButtonElement.Click();
            return new GoogleSearchResultsPageModel(Driver);
        }
    }
}

Putting it all together

I have two batch files that I use in Visual Studio 2010 C# Express, one to generate the SpecFlow NUnit tests, the other to execute them and provide a visual report. I add these as external tools in VS2010 and run them from the menu. Neat.