AMA: CodeceptJS support for Safari and IE?

Sahana Asks…

We area VOD startup and we have web app, mobile apps and TV apps. I am writing acceptance tests for web app now and chose codeceptjs framework since we have our website’s front end code in Javascript. We have dockerised the processes and docker images for codeceptjs webdriver IO is availble only for chrome and firefox browsers. How can I handle Safari , Internet Explorer browsers ? Looks like CodeceptJs does not support IE and Safari browsers. Do you have any suggestion?

My Response…

I’ve never personally found the return on investment of getting automated tests running across Internet Explorer and  Safari to be worthwhile as in my experience this took more effort than the bugs it found. So I personally stick to running our full e2e test suite in our most used browser (Chrome) and supplementing this with exploratory testing on all other browsers.

In saying that the reason you won’t be able to use Docker containers for these purposes is that they’re Linux and Internet Explorer requires Microsoft Windows and Safari requires Apple macOS to be able to run. To be able to use these for your existing automated tests you can sign up to a on-demand browser service like SauceLabs and use the remote WebDriver protocol to execute your tests.

AMA: Test Data Infrastructure

Anonymous asks…

Do you have set up (inexpensive) infrastructure to store data collected in your automated tests? We are currently using using selenium Java webdriver to automate our tests and IntelliJ as our IDE. We create data from scratch for each and every test case :(

My response…

I’m a little confused by the question and whether it’s about test data: data is that is needed by the automated tests, or test results data: insights into the results of our automated tests. So I’ll answer both 😀

Infrastructure to manage test data

Our tests run on specific test accounts and sites on production databases. Since our tests are end-to-end in fashion, we try to make our tests have as few dependencies as possible on existing data. Often an end-to-end scenario will involve creating, viewing, editing and deleting something. If we don’t do all of this by our UI we can use hooks that either use services or database jobs to clean up the data. I explained this in more detail previously.

Infrastructure to manage test results data

We use CircleCI for automated end-to-end tests. We have a number of projects that run different types of end-to-end tests from the same code repository for different purposes (canary tests, visual-diff tests, full regression tests for example).

We generate x-unit test results (from Mocha/Magellan) which CircleCI uses to provide insights into our test results such as this:

You can also drill down into slowest tests and most failed tests etc.

Since all our tests are open source you can view these build insights yourself!

We’re pretty happy with the insights we get from CircleCI at the moment so we don’t see a need to currently develop anything ourself.

WebDriverJs Select Lists in Chrome

Chromedriver/Chrome is pretty great at executing WebDriverJs scripts without taking away your focus (so you can execute them in the background whilst doing other things), the one exception I found was selecting items in a select list. I found it would do this:

Continue reading “WebDriverJs Select Lists in Chrome”

Feature Toggles for Automated e2e Tests

Feature toggles aren’t just for production code. Feature toggles are also a powerful technique to change the behaviour of your automated end-to-end tests without changing code.

Continue reading “Feature Toggles for Automated e2e Tests”

Prioritising Test Reliability over Perfection

If you saw my talk at GTAC last year, ‘your tests aren’t flaky‘, then you’re probably aware of my view on flaky tests actually being indicative of broader application/systems problems that we should address over making our tests less flaky.

But what if you’re in a situation where you work with a system where you can’t feasibly improve the reliability? Say you’ve got a domains page that should show you a list of available domains but since it’s using an external third-party service it sometimes just shows nothing?

Continue reading “Prioritising Test Reliability over Perfection”

AMA: product APIs for test automation

Michael Karlovich asks..

What’s your design approach for incorporating internal product APIs into test automation? I don’t mean in order to explicitly test them, but more for leveraging them to stage data and set application states.

My response…

As explained previously, in my current role at Automattic I primarily work on end-to-end automated tests for These tests run against live data (Production) no matter where our UI client (Calypso) is running (for example on localhost), so we don’t use APIs for staging data or setting application state.

In previous roles we utilised a REST API to create dynamic data for an internally used web application which we found useful/necessary for repeatable UI tests.

We also utilised test controllers to set web application state for a public website. These test controllers were very handy as they allowed you to visit something like which would set up an order for you with products in your session, and instantly display the checkout page, which would typically take 8 steps from the start of the process to this page.

This saved us lots of time and made our specific tests more deterministic as we could avoid the 8 or so ‘setup’ steps and use a single URL to access our page.

This approach had a couple of downsides in that this couldn’t ever be deployed to production, and it didn’t test realistic user flow which includes those ‘setup’ steps. There were two things we had to do to avoid the risk of using this approach; firstly ensure that these test controllers were never deployed to production though config, and secondly we had to ensure we had some end-to-end coverage so we were at least testing some real user flows.

AMA: R.Y.O. Page Objects 2.0

Michael Karlovich asks…

Do you have any updated thoughts on rolling your own page objects with Watir? The original post is almost 4 years old but is still the basis (loosely) of every page object framework I’ve built since then.

My response…

Wow, I can’t believe that post is almost four years old. I have also have used this for the basis of every page object framework I have built since then.

I recently had a look at our JavaScript (ES2015) code of page objects and despite ES2015 not having meta-programming support like ruby, our classes are remarkably similar to what I was proposing ~4 years ago.

I believe this is because some patterns are classic and therefore almost timeless, they can be applied over and over again to different contexts. There’s a huge amount of negativity towards best practices of late, but I could seriously say that page objects are a best practise for test automation of ui systems, which isn’t saying they will be exactly the same in every context, but there’s a common best-practice pattern there which you most likely should be using.

Page objects, as a pattern, typically:

  • Inherit from a base page object/container which stores common actions like:
    • instantiating the object looking for a known element that defines that page’s existence
    • optionally allow a ‘visit’ to the page during instantiation using some defined URL/path
    • provides actions and properties common to all pages, for example: waiting for the page, checking the page is displayed, getting the title/url of the page, and checking cookies and local storage items for that page;
  • Define actions as methods which are ways of interacting with that page (such as logging in);
  • Do not expose internals about the page outside the page – for example they typically don’t expose elements or element selectors which should only be used within actions/methods for that page which are exposed; and
  • Can also be modelled as components for user interfaces that are built using components to give greater reusability of the same components across different pages.

The biggest benefit I have found from using page objects as a pattern is having more deterministic end-to-end tests since instantiating a page I know I am on that page, so my tests will fail more reliably with a better understanding of what went wrong.

Are there any other pattern attributes you would consider vital for page objects?