A Software Testing Blog by Alister Scott
João Monteiro asks…
I recently joined a small company where I am the only QA and the test automation suite was already written in a given/when/then style but rarely (if not at all) gets read by the rest of the team (product or developers). Any tips on how to mentor the team to adopt a BDD approach? Do you recommend any tools / framework to share the features in a centralised place easily accessible by the rest of the team an not just on the tests repository?
As part of my trial for my current role at Automattic, I was tasked with implementing some e2e acceptance tests using my choice of library/framework/language.
I very much recommend writing automated acceptance tests in the same language as your app, even though I have described some benefits of using a different language, and since WordPress is moving towards JavaScript from PHP, JavaScript seems the most suitable language for Automattic.
Continue reading “Comparison of JavaScript browser automation and test specification libraries”
Whilst being involved with lots of people writing automated acceptance tests using tools like SpecFlow and WebDriver I’ve seen some ‘anti-patterns’ emerge that can make these tests non-deterministic (flaky), very fragile to change and less efficient to run.
Here’s five ‘anti-patterns’ I’ve seen and what you can do instead.
Anti-pattern One: Not using page-objects
Page objects are just a design pattern to ensure automated UI tests use reusable, modular code. Not using them, eg, writing WebDriver code directly in step definitions, means any changes to your UI will require updates in lots of different places instead of the one ‘page’ class.
Bad
[sourcecode language=”csharp”]
[When(@"I buy some ‘(.*)’ tea")]
public void WhenIBuySomeTea(string typeOfTea)
{
Driver.FindElement(By.Id("tea-"+typeOfTea)).Click();
Driver.FindElement(By.Id("buy")).Click();
}
[/sourcecode]
Better
[sourcecode language=”csharp”]
[When(@"I buy some ‘(.*)’ tea")]
public void WhenIBuySomeTea(string typeOfTea)
{
MenuPage.BuyTea(typeOfTea);
}
[/sourcecode]
Complicated set up scenarios within the tests themselves
Whilst there’s a place for automated end-to-end scenarios (I call these user journies), I prefer most acceptance tests to jump straight to the point.
Bad
Scenario: Accept Visa and Mastercard for Australia Given I am on the home page for Australia And I choose the tea menu And I select some 'green tea' And I add the tea to my basket And I choose to checkout Then I should see 'visa' is accepted And I should see 'mastercard' is accepted
Better
This usually requires adding some special functionality to your app, but the ability for testing to ‘jump’ to certain pages with data automatically set up makes automated tests much easier to read and maintain.
Scenario: Accept Visa and Mastercard for Australia Given I am the checkout page for Australia Then I should see 'visa' is accepted And I should see 'mastercard' is accepted
Using complicated x-path or CSS selectors
Using element identification selectors that have long chains from the DOM in them leads to fragile tests, as any change to that chain in the DOM will break your tests.
Bad
[sourcecode language=”csharp”]
private static readonly By TeaTypeSelector =
By.CssSelector(
"#input-tea-type > div > div.TeaSearchRow > div.TeaSearchCell.no > div:nth-child(2) > label");
[/sourcecode]
Better
Identify by ‘id’ (unique) or ‘class’. If there’s multiple elements in a group, create a parent container and iterate through them.
[sourcecode language=”csharp”]
private static readonly By TeaTypeSelector = By.Id("teaType");
[/sourcecode]
Directly executing JavaScript
Since WebDriver can directly execute any arbitrary JavaScript, it can be tempting to bypass DOM manipulation and just run the JavaScript.
Bad
[sourcecode language=”csharp”]
public void RemoveTea(string teaType)
{
(driver as IJavaScriptExecutor).ExecuteScript(string.Format("viewModel.tea.types.removeTeaType(\"{0}\");", teaType));
}
[/sourcecode]
Better
It is much better to let the WebDriver control the browser elements which should fire the correct JavaScript events and call the JavaScript, as that way you avoid having your ‘test’ JavaScript in sync to your ‘real’ JavaScript.
[sourcecode language=”csharp”]
public void RemoveTea(string teaType)
{
driver.FindElement(By.Id("remove-"+teaType)).Click();
}
[/sourcecode]
Embedding implementation detail in your features/scenarios
Acceptance test scenarios are meant to convey intention over implementation. If you start seeing things like URLs in your test scenarios you’re focusing on implementation.
Bad
Scenario: Social media links displayed on checkout page
Given I am the checkout page for Australia
Then I should see a link to 'http://twitter.com/beautifultea'
And I should see a link to 'https://facebook.com/beautifultea'
Better
Hide implementation detail in the steps (or pages, or config) and make your scenarios about the test intention.
Scenario: Social media links displayed on checkout page
Given I am the checkout page for Australia
Then I should see a link to the Beautiful Tea Twitter account
And I should see a link to the Beautiful Tea Facebook page
I hope you’ve enjoyed these anti-patterns. Leave a comment below if you have any of your own.
I’ve been working on an iOS project and have been looking for a suitable automated functional test library to drive the iOS GUI. I must say that automated iOS testing feels like automated web testing did about 10 years ago: lots of different tools taking different approaches, and all approaches quite flaky!
Through research I came across quite a few different tools that support iOS automated functional testing but need to work out which one is best.
Whilst I often advocate writing functional tests in the same language as your codebase, in the case of iOS and Objective C, I think using a more lightweight language like Ruby has its own advantages (I don’t really like Objective C).
The one thing I really dislike with a lot of these iOS functional test tools is how they are married to Cucumber and encourage users to write tests like “I tap button x” and “I scroll through list y”. These types of tests are much harder to read as they express implementation over intention and should be avoided.
I will also be writing tests for an Android app so being able to use the same tool-set is a great advantage to me.
I also personally prefer an approach where I don’t need to modify the core behavior of my app to run tests against it (designing for testability is completely different and vital). Some approaches embed some sort of server to receive automation commands, which the user must then remove later because it uses undocumented Apple APIs which will lead to app store rejection. Being able to run tests against the app that you submit to the app store means you can be more confident you have tested it right.
Finally, the iOS app I am working on using embedded web views to display dynamic content from the server, which can be interacted with, and therefore it is vital that these can be interacted with. This feature is actually very rare in an iOS automation framework.
Here’s the list of iOS functional automation tools I am aware of and how they stack up:
I’ll let you guess which tool I selected until I write my next blog post about how to get started in that tool.
I quite frequently come across a scenario that looks something like this:
[sourcecode]
Scenario: Create some animals
Given I am a zoo keeper
When I create a giraffe
And I create a lion
And I create a pony
And I create a unicorn
Then I should have a zoo
[/sourcecode]
and step definitions that implement the When steps with a single step definition:
[sourcecode language=”ruby”]
When /^I create a (\D+)$/ do |animal|
case animal
when ‘lion’
create_a_lion()
when ‘giraffe’
create_a_giraffe()
when ‘pony’
create_a_pony()
else
raise ‘Unknown animal’
end
end
[/sourcecode]
I don’t like having case statements in steps for a number of reasons:
For example, you could change our steps to look something like this:
[sourcecode language=”ruby”]
When /^I create a lion$/ do
create_a_lion()
end
When /^I create a giraffe$/ do
create_a_giraffe()
end
When /^I create a pony/ do
create_a_pony()
end
[/sourcecode]
Even though this is three times as many step definitions, it is actually less code (9 lines compared to 12).
By using this approach it is obvious we can’t currently create a unicorn as RubyMine tells us before we even run our tests. And we don’t need to raise any exceptions myself.
Whilst a lot of people use case statements in steps to reduce the number of steps, it is actually counter intuitive as it means you have more code to do so, and the outcome is less usable when writing scenarios. So, please avoid putting case statements in your step definitions.