Feature toggles aren’t just for production code. Feature toggles are also a powerful technique to change the behaviour of your automated end-to-end tests without changing code.
If you saw my talk at GTAC last year, ‘your tests aren’t flaky‘, then you’re probably aware of my view on flaky tests actually being indicative of broader application/systems problems that we should address over making our tests less flaky.
But what if you’re in a situation where you work with a system where you can’t feasibly improve the reliability? Say you’ve got a domains page that should show you a list of available domains but since it’s using an external third-party service it sometimes just shows nothing?
Michael Karlovich asks..
What’s your design approach for incorporating internal product APIs into test automation? I don’t mean in order to explicitly test them, but more for leveraging them to stage data and set application states.
As explained previously, in my current role at Automattic I primarily work on end-to-end automated tests for WordPress.com. These tests run against live data (Production) no matter where our UI client (Calypso) is running (for example on localhost), so we don’t use APIs for staging data or setting application state.
In previous roles we utilised a REST API to create dynamic data for an internally used web application which we found useful/necessary for repeatable UI tests.
We also utilised test controllers to set web application state for a public website. These test controllers were very handy as they allowed you to visit something like
http://myteststore.com/testsetup/checkout which would set up an order for you with products in your session, and instantly display the checkout page, which would typically take 8 steps from the start of the process to this page.
This saved us lots of time and made our specific tests more deterministic as we could avoid the 8 or so ‘setup’ steps and use a single URL to access our page.
This approach had a couple of downsides in that this couldn’t ever be deployed to production, and it didn’t test realistic user flow which includes those ‘setup’ steps. There were two things we had to do to avoid the risk of using this approach; firstly ensure that these test controllers were never deployed to production though config, and secondly we had to ensure we had some end-to-end coverage so we were at least testing some real user flows.
Michael Karlovich asks…
Do you have any updated thoughts on rolling your own page objects with Watir? The original post is almost 4 years old but is still the basis (loosely) of every page object framework I’ve built since then.
Wow, I can’t believe that post is almost four years old. I have also have used this for the basis of every page object framework I have built since then.
I believe this is because some patterns are classic and therefore almost timeless, they can be applied over and over again to different contexts. There’s a huge amount of negativity towards best practices of late, but I could seriously say that page objects are a best practise for test automation of ui systems, which isn’t saying they will be exactly the same in every context, but there’s a common best-practice pattern there which you most likely should be using.
Page objects, as a pattern, typically:
- Inherit from a base page object/container which stores common actions like:
- instantiating the object looking for a known element that defines that page’s existence
- optionally allow a ‘visit’ to the page during instantiation using some defined URL/path
- provides actions and properties common to all pages, for example: waiting for the page, checking the page is displayed, getting the title/url of the page, and checking cookies and local storage items for that page;
- Define actions as methods which are ways of interacting with that page (such as logging in);
- Do not expose internals about the page outside the page – for example they typically don’t expose elements or element selectors which should only be used within actions/methods for that page which are exposed; and
- Can also be modelled as components for user interfaces that are built using components to give greater reusability of the same components across different pages.
The biggest benefit I have found from using page objects as a pattern is having more deterministic end-to-end tests since instantiating a page I know I am on that page, so my tests will fail more reliably with a better understanding of what went wrong.
Are there any other pattern attributes you would consider vital for page objects?
BONUS: Any tips on testing modules that rely on dynamically created SQL? Common sense suggests testing to the nearest clearly defined “business value” and eventually separating concerns/refactoring. Any weakly held strong options?
I still think you should write your tests in the same language as your app, so for a MS web stack I would lean towards SpecFlow/WebDriver (see SpecDriver for an example). I am not sure whether Watin is actively maintained or whether it supports browsers other than IE, but I know the C# WebDriver bindings are increasingly solid.
Bonus answer: I think your idea makes sense as there’s elements of context and unpredictability, so starting with one approach and letting it evolve over time through refactoring is often the best outcome.
Swapnil Waghmare asks…
Are BDD frameworks like Specflow, Cucumber better for E2E tests?
There’s two ways I can interpret this question (my additions are in bold).
Are BDD frameworks that use ‘Given/When/Then’ feature files like Specflow, Cucumber better than frameworks that use ‘describe/it’ blocks like RSpec and Mocha for e2e tests?
As I previously explained, Automattic’s unit tests are written in Mocha, so that was a logical choice for writing e2e tests as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.
There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.
Are BDD frameworks like Specflow, Cucumber better for E2E tests than using them for automated integration, component or unit tests?
I think you could write integration tests or even unit tests in Given/When/Then format as most unit tests follow the same arrange/act/assert pattern anyhow which is exactly what Given/When/Then is.
Keep in mind there is overhead in maintaining all the step definitions and feature files for Cucumber/Specflow that give you non technical readability so if you don’t require that readability it is probably overkill. But a personal preference nonetheless.
Swapnil Waghmare also asks…
Which tools have you used for API testing, which ones would you recommend?
I try to keep things simple, so in the past when I’ve written REST integration tests I’ve just called them using in built libraries, and have used Postman quite a lot for manual testing and debugging.
Swapnil Waghmare finally asks…
As for whether QA’s should write unit tests? No, I don’t think so, as I believe unit tests should drive software development, and writing code by writing unit tests is much easier than trying to add unit tests after by someone else, as the original code will not likely be very testable as it wasn’t written with testability in mind. One benefit of code written with unit tests at the time is that it will mostly be better code as the tests are consuming your API that you are developing.
Butch Mayhew asks…
I have noticed you blogging more about JS frameworks. How do these compare to Watir/Ruby? Would you recommend one over the other?
Whilst I see merit in both views: I still think having your automated acceptance tests in the same language as your application leads to better maintainability and adoptability.
But generally with the direction ES is going, writing page objects as classes is much nicer than using functions for everything as in ES5.