AMA: Moving automated tests from Java to JavaScript

Anonymous asks…

I am currently using a BDD framework with Cucumber, Selenium and Java for automating a web application. I used page factory to store the objects and using them in java methods I wanted to replace the java piece of code with javaScript like mocha or webdriverio. could you share your thoughts on this? can I still use page factory to maintain objects and use them in js files

My response…

What’s the reasoning for moving to JavaScript from Java? Despite having common names, there’s very little otherwise in common (Car is to Carpet as Java is to JavaScript.)

I wouldn’t move for moving sake since I see no benefit in writing BDD style web tests in JavaScript, if anything, e2e automated tests are much harder to write in JavaScript/Node because everything is asynchronous and so you have to deal with promises etc. which is much harder to do than just using Java (or Ruby).

Aside: I still dream of writing e2e tests in Ruby: it’s just so pleasant. But our new user interface is written extensively in JavaScript (React) so it makes sense from a sustainability point of view to use JS over Ruby.

 

Why you should use CSS selectors for your WebDriver tests

I didn’t used to be a fan of CSS selectors for automated web tests, but I changed my mind.

The reason I didn’t use to be a fan of CSS selectors is that historically they weren’t really encouraged by Watir, since the Watir API was designed to find elements by type and attribute, so the Watir API would look something like:

browser.div(:class => 'highlighted')

where the same CSS selector would look like:

div.highlighted

Since WebDriver doesn’t use the same element type/attribute API and just uses findElement with a By selector, CSS selectors make the most sense since they’re powerful and self-contained.

The the best thing about using CSS selectors, in my opinion, is the Chrome Dev Tools allows you to search the DOM using a CSS selector (and XPath selectors, but please don’t use XPath), using Command/Control & F:

chrome css selectors
Using CSS selectors to find elements in Chrome Dev Tools

So you can ‘test’ your CSS in a live browser window before deciding to use it in your WebDriver test.

The downside of using CSS selectors are they’re a bit less self explanatory than explicitly using by.className or by.id.

But CSS selectors are pretty powerful: especially pseudo selectors like nth-of-type and I’ve found the only thing you can’t really do in CSS is select by text value, which you probably shouldn’t be doing anyway as text values are more likely to change (since they’re copy often changed by your business) and can be localised in which case your tests won’t run across different cultures.

The most powerful usage of CSS selectors is where you add your own data attributes to elements in your application and use these to select elements: straightforward, efficient and less brittle than other approaches. For example:

a[data-e2e-value="free"]

How do you identify elements in your WebDriver automated tests?

AMA: Trunk Guardian Service?

Sue asks…

I read a LinkedIn blog post from 2015 by Keqiu Hu from LinkedIn about flaky UI tests. He explains how they fixed their flaky UI tests for the LinkedIn app. Among other things they implemented what they called the “Trunk Guardian service” which runs automated UI tests on the last known good build twice and if the test passes on the first run but fails on the second it is marked as ‘flaky’ and disabled and the owner is notified to fix it or get rid of it. I wondered what your thoughts were on such a “Trunk Guardian service” – if the culture / process was in place to solve the other issues that create flaky tests, could such a thing be worth the effort to implement? Article: Test Stability – How We Make UI Tests Stable

My response…

Continue reading “AMA: Trunk Guardian Service?”

AMA: IE11 Button Clicking in Selenium

Anthony asks…

I have coded to click buttons on IE11/Win7 but the latest version of Selenium IE doesn’t click the buttons correctly most of times. Most of times, it clicks one button below. I thought it might be loading time so added some waiting but still click one button below or two buttons below sometimes. I googled this and found several posting saying Selenium IE doesn’t click buttons well. Now I have moved it to FF but I am still wondering why IE is not accurate. I know a lot of Selenium test developers in the field but they are having the same issue or they know a workaround. What do you think of this issue on IE11? Are you aware of this issue? FYI, the buttons are not regular HTML tag. The menu system with clickable tag is created by javascript. Thank you!

My Response…

We actually don’t run any tests in Internet Explorer any more since these weren’t finding any browser specific bugs (we do exploratory testing in Internet Explorer instead).

But, I have heard of problems generally with the IEDriver tool. If you’re working on a JavaScript generated app I think the best thing for you to do would instead of using a native click in Selenium is instead execute a JavaScript click event. The exact syntax will depend on which language you’re using Selenium in, but it should look something like this:

this.driver.executeScript( 'return arguments[0].click();', webElement );

I hope this solution helps!

AMA: Iterative vs Incremental

Anonymous asks…

What is the difference between iterative and incremental models?

My response:

Fortunately I have written an entire post on this exact topic here.

My conclusion was:

We can’t build anything without iterating to some degree: no code is written perfectly the second that it is typed or committed. Even if it looks like a company is incrementally building their software: they’re iteratively building it inside.

We can’t release anything without incrementing to some degree: no matter how small a release is, it’s still an incremental change over the last release. Some increments are bigger because they’ve already been internally iterated upon more, some are smaller as they’re less developed and will evolve over time.

So, we develop software iteratively and release incrementally in various sizes over time.

AMA: How do I do data migration testing?

Nathan asks…

Data Migration testing from one application to another application. Which way to test best and easy? The new application should be in Salesforce.

My Response…

This is quite a generic question but I’ll try to answer it the best I can. I usually look at data migrations as three separate steps:

Extract data from the old system
Transform the data to fit the new system
Load the data into the new system

I would test that each step has worked correctly by verifying the data starting in the deepest parts of the system (database tables), moving up into APIs and finally into any user interfaces. I know some CRMs such as Salesforce don’t allow access to database tables so sometimes you can only use APIs or user interfaces to ‘spot check’ data.

I hope this helps you Nathan.

AMA: CodeceptJS support for Safari and IE?

Sahana Asks…

We area VOD startup and we have web app, mobile apps and TV apps. I am writing acceptance tests for web app now and chose codeceptjs framework since we have our website’s front end code in Javascript. We have dockerised the processes and docker images for codeceptjs webdriver IO is availble only for chrome and firefox browsers. How can I handle Safari , Internet Explorer browsers ? Looks like CodeceptJs does not support IE and Safari browsers. Do you have any suggestion?

My Response…

I’ve never personally found the return on investment of getting automated tests running across Internet Explorer and  Safari to be worthwhile as in my experience this took more effort than the bugs it found. So I personally stick to running our full e2e test suite in our most used browser (Chrome) and supplementing this with exploratory testing on all other browsers.

In saying that the reason you won’t be able to use Docker containers for these purposes is that they’re Linux and Internet Explorer requires Microsoft Windows and Safari requires Apple macOS to be able to run. To be able to use these for your existing automated tests you can sign up to a on-demand browser service like SauceLabs and use the remote WebDriver protocol to execute your tests.

AMA: What sets exceptional QA testers apart?

Dayana asks…

I wondered if you could tell me what sets exceptional QA testers apart? Not just personality or work ethic traits, but specific skills and programming knowledge that will be very valuable to a team?

My response…

I think exceptional QA testers, as explained recently, aren’t people who are exceptional at just one thing, eg. testing, but good at lots of things.

So an exceptional QA tester, in my opinion, will typically have (at least good) skills in the following things:

  1. Skills in human exploratory testing: an exceptional QA tester has the ability to effectively find the most important bugs fast. Whilst this skill can be developed, I have found it’s mostly a mindset.
  2. Skills in developing automated tests: an exceptional QA tester will have programming skills needed to develop automated tests and I would recommend these to typically match the programming language(s) that programmers in your organization use. For example, skills in automated testing in .NET if your company primarily uses Microsoft .NET. Although, someone with strong programming skills in one language (eg. ruby) should be able to transfer these skills to another language (eg. python).
  3. Knowledge/Experience in your business domain: an exceptional QA tester will fully understand your business domain and keep this context in mind whilst testing a product and raising issues. An exceptional tester is always testing your system – just as I am testing WordPress.com publishing this post.
  4. An empathetic mindset: we design and develop software for real people and real life. An exceptional QA tester will test with this in mind.

iOS11 Screen Recording (for bug reports)

iOS11 is out and it supports native in-device screen recording! I’m a huge fan of attaching screen recordings to bug reports (they capture flow!), whether that be animated gifs or videos, and until now to do this on iOS required plugging your iDevice into your Mac via USB and using QuickTime to record the screen. Alas no longer, it’s now easy to do so via control centre!

Here’s a screen recording (no audio), recorded on my iPhone no less, that shows you how 😊

Protip: force touch the screen recording button to enable/disable microphone for narration

Protip #2: use the in built video editor to trim the start and end

AMA: Test Data Infrastructure

Anonymous asks…

Do you have set up (inexpensive) infrastructure to store data collected in your automated tests? We are currently using using selenium Java webdriver to automate our tests and IntelliJ as our IDE. We create data from scratch for each and every test case :(

My response…

I’m a little confused by the question and whether it’s about test data: data is that is needed by the automated tests, or test results data: insights into the results of our automated tests. So I’ll answer both 😀

Infrastructure to manage test data

Our tests run on specific test accounts and sites on production databases. Since our tests are end-to-end in fashion, we try to make our tests have as few dependencies as possible on existing data. Often an end-to-end scenario will involve creating, viewing, editing and deleting something. If we don’t do all of this by our UI we can use hooks that either use services or database jobs to clean up the data. I explained this in more detail previously.

Infrastructure to manage test results data

We use CircleCI for automated end-to-end tests. We have a number of projects that run different types of end-to-end tests from the same code repository for different purposes (canary tests, visual-diff tests, full regression tests for example).

We generate x-unit test results (from Mocha/Magellan) which CircleCI uses to provide insights into our test results such as this:

You can also drill down into slowest tests and most failed tests etc.

Since all our tests are open source you can view these build insights yourself!

We’re pretty happy with the insights we get from CircleCI at the moment so we don’t see a need to currently develop anything ourself.

AMA: Testing an Angular app

Ben asks…

I’ve been working with angular a while now but I have to admit, the testing side throws me. Every time I start to tackle it, I find myself distracted from the test I want to write by all of the things I need to mock. Sometimes it feels like I need to build an entire mock framework to test one feature. Maybe I’m doing it wrong. Any tips appreciated, whether practical or even just the right headspace for approaching it :)

My response…

I haven’t worked with angular applications directly but I’ve worked on React applications and I’m guessing the approach to unit testing these will be similar.

I would start as simple as possible with the smallest test that would possibly work. If you find that you need an entire mock framework to test a feature it sounds like your components may need to be broken down further into smaller components as you have too many dependencies that need to be mocked. If you have smaller components that only require a single dependency then these components should be easier to test as the dependencies will be easier to mock.

With smaller components I would also encourage you to be able to pass in dependencies. This allows you to easily mock/override dependencies for testing purposes – known as dependency injection. I wrote about this using a simple JavaScript example previously which you may want to read.

If retrofitting unit tests into your app is still too difficult you could try to automated some key end-to-end scenarios using Protractor, but I’d discourage you from going overboard since these can quickly get out of hand. You may still benefit from having a few of these tests even if you get unit testing of small components working well to ensure the small components work well together.

I hope this helps you Ben 😊

AMA: BA roles in Automattic?

Iroy asks…

Automattic uses interesting and fun names for different roles (QA being excellence wrangler). Are there Business Analyst roles in Automattic? If so, what is it called?

My Response…

At Automattic we differentiate between a Job Title and a Role. My (current) job title is indeed Excellence Wrangler and my role is Code Wrangling. Anyone is free to change their job title to anything they like at any time so we have some fun ones, whereas the roles are pretty static.

Continue reading “AMA: BA roles in Automattic?”

The blurry line between test and development

One of the themes I talked about during my presentation in Wellington was the blurry line between test and development in a distributed environment like Automattic.

I was recently having trouble with a complex method in our WordPress.com e2e test page objects, so I used my skills as a developer and wrote a change to our user interface which adds a data attribute to the HTML element.

This meant our page object method immediately went from this:

Continue reading “The blurry line between test and development”

Staying focused with one conference presentation per year

I recently saw Basecamp made their employee handbook available online1 and this statement stood out to me in their moonlighting guide:

Not OK

2. Go out on a regular speaking circuit tour which is going to require multiple days of travel multiple times a year. That’s too disruptive to our own work schedule and to your fellow teammates.

For this exact reason, without knowing, I created my own personal rule2 of only committing to doing one conference presentation per year.

Continue reading “Staying focused with one conference presentation per year”

Test for Real Life

“Most of us are anxious pretty much all the time – but frequently imagine that other people aren’t. It’s time to admit the truth. Anxiety is just a basic fact about being human.”

~ Alain de Botton

We are all human, we are all worried and anxious pretty much all the time, people just don’t tell you that they are. We wear masks and we hide it well.

But why do we test like we’re not anxious or worried? Why don’t we test for real life?

Continue reading “Test for Real Life”

Save password prompts in Chrome 57 with WebDriver

When running Selenium WebDriver scripts against the latest version of Chrome (57) it shows a save password prompt that hasn’t appeared previously whilst using Chromedriver, as far as I know.

chrome 57 save password prompt Continue reading “Save password prompts in Chrome 57 with WebDriver”

WebDriverJs Select Lists in Chrome

Chromedriver/Chrome is pretty great at executing WebDriverJs scripts without taking away your focus (so you can execute them in the background whilst doing other things), the one exception I found was selecting items in a select list. I found it would do this:

Continue reading “WebDriverJs Select Lists in Chrome”