Testbash Australia 2018

I only speak at one conference a year and this year that conference will be the first ever Australian Testbash in Sydney on October 19, 2018:

TestBash_Australia_2018_Adverts_DOJO_EVENT_BANNER.png

My talk:

At WordPress.com we constantly deliver changes to our millions of customers – in the past month alone we released our React web client 563 times; over 18 releases every day. We don’t conduct any manual regression testing, and we only employ 5 software testers in a company of ~680 people with ~230 developers across . So, how do we ensure our customers get a consistently great user experience with all this rapid change?

Our automated end-to-end (e2e) tests give us confidence to release small frequent changes constantly to all our different applications and platforms knowing that our customers can continue to use our products to achieve their desires.

Alister will share how these automated tests are written and work, some of the benefits and challenges we’ve seen in implementing these tests, and what we have planned for future iterations.

Takeaways: How to effectively use automated end-to-end testing to ensure a consistent user experience in high frequency delivery environments

Grab a ticket before they sell out #

AMA: JavaScript in Test Automation

Max asks…

First of all thanks for sharing all your insights on this blog. I regularly try to come back to your blog and I helped me grow as a QA Engineer quite a lot.
I wanted to ask you on your opinion on a seemingly new generation of test tools for the web that are written in JavaScript to help deal with all the asynchronicity on modern front end frameworks. Our company is in the process of redesigning our website and it seems that it just gets harder and harder to deal with all the JavaScript in test automation. I recently started looking into some other tool like cypress.io, but as of now they still seem quite immature.
Could you help me and point me in the right direction?

My response…

I think there’s two parts to the JavaScript asynchronicity issue which are not necessarily related.

Modern JavaScript front-end web frameworks (like React and Angular) are designed to be asynchronous and fast and therefore this can cause issues when trying to write automated tests against them. I don’t believe writing automated tests in an asynchronous way actually makes the tests easier to write, or more maintainable or robust, but just a result of writing these tests in the same way the web frameworks work.

You can write synchronous tests (like using Watir/Ruby) against asynchronous web interfaces, you just need to use the waiting/polling mechanisms (or write/extend your own) – the same as you need to do in asynchronous test automation tools.

We choose WebDriverJS for automated end-to-end testing of our React application as it was the official WebDriver for JavaScript project and seems to be a good choice at the time. I somewhat regret that decision now as using a synchronous third party implementation like webdriver.io seems like an easier way to write and maintain tests.

I have tried to use cypress.io but the way it controls sites (through proxies) has (current) limitations like not working on iFrames and cross-domains which are deal-breakers for our end-to-end testing needs at present.

If you don’t need to write your end-to-end tests in JavaScript I’d say avoid it unless absolutely necessary and stick to another non-asynchronous programming language.

I’m glad I’ve been able to help you grow over the years.

AMA: hiding/changing IP addresses using Watir

Mudit Bhardwaj asks…

Hey! In order to test the security of a website, I’m trying to create hundreds of accounts. However, after a certain limit, there is always an error which prevents me from going further.
How can I hide/change my ip address during each iteration with watir?

My response…

I can understand why there would be this limitation in place as this activity seems suspicious from a systems perspective.

Watir can’t change the IP address. You can use an anonymous browser profile and/or delete cookies for different account creation runs but this probably won’t help.

I’d speak to your development/devops/systems team to whitelist your IP address(es) you are using for this purpose.

Puppeteer for Automated e2e Chrome Testing in Node

I recently noticed the new Google Chrome project Puppeteer:

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

As someone who only runs WebDriver tests in Google Chrome anyway, this looks like a promising project that bypasses WebDriver to have full programmatic control of Google Chrome including for automated end-to-end (e2e) tests.

The thing I really love about this is no Chromedriver dependency and how installing the library installs Chromium by default which can be controlled headlessly with zero config or any other dependencies.

You can even develop scripts using this playground.

I set up a (very basic) demo project that uses Mocha + Puppeteer and it runs on CircleCI with zero config. Awesome.

Five Books I Enjoyed in 2017

Happy New Year! 🎉 One thing I love about the end of the year is going through all the lists I have made during the year and counting up my stats and reflecting on things. I read (to completion) 37 books (and abandoned a further 4) in 2017. Not quite as many as 2016 when I finished 48 books but I had some difficult circumstances with family illness in 2017.

Here’s five of my favourite books I read in 2017 (with Good Reads links):

9781471138867_hrThe Art of Stillness: Adventures in Going Nowhere by Pico Iyer. This was the first TED book I read and having read quite a few more since it was probably my favourite.  I read this at a part of the year where I was over-committing to too many things (travel, conferences, events) and it was a apt reminder to slow down and appreciate things. This book led me to my mindfulness meditation practice which I took up in the later part of the year (and have continued to do since then).

how-to-fail-at-almost-everything-and-still-win-big-coverHow to Fail at Almost Everything and Still Win Big: Kind of the Story of My Life by Scott Adams. I’m not a huge fan of Dilbert, but I’m a big Scott Adams fan after reading this book. The ideas I really like are the importance of systems over goals (a good blog post about that here), and that passion follows success (passion doesn’t lead to success: you can be passionate about something and really suck at it). Also the importance of diversification. An enjoyable well written read.

29430779._SR1200,630_Payoff: The Hidden Logic That Shapes Our Motivations by Dan Ariely. The second TED book I read this year (I love the format!) and the best book about motivation I’ve read so far. I’ve tried to read Drive by Dan Pink a few times but haven’t been motivated to finish it. The thing I learnt from this book is that we lose meaning and motivation in life by outsourcing the hard work/small things (like cleaning, gardening, maintenance) – we accomplish more but get less.

subtle-art-coverThe Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life by Mark Manson. This book is a bit weird. It’s hard to read, because it jumps all over the place, but there’s so much good content it’s still worth reading. I also wish the book had a really clear list of things I could take away from the book and do, rather than understanding the general message which is a bit bleak. Still a good book and enjoyable read after all that.

27985224Deep Work: Rules for Focused Success in a Distracted World by Cal Newport. One of the last books I read this year it is particularly relevant to me working for a 100% distributed company in an asynchronous communication environment it’s easy to get distracted by constant chatter and noise and not focus enough on deep work. The thing about this book is that if you can master deep work in our current world it’s another skill that you can use to be very successful as it’s a huge competitive advantage.

*** BONUS ***

9781760630775Not work related but I read a lot of fiction particularly thrillers. Michael Connelly’s latest Harry Bosch book Two Kinds of Truth was just fantastic: hard to describe how good it was.

 

 

 

AMA: Clicking a non-visible element in Watir

Mike asks…

Is there any way to make Watir click a link/button that is not visible? I wanted to switch from Capybara/CapybaraWebKit (which allows clicking non-visible elements) but I am stuck since Watir always times out on the click attempt.

My response…

The easiest way to do this is just to execute a click in JavaScript on the object itself.

In Watir, this looks something like:

[code language=”ruby”]
browser.execute_script( "return arguments[0].click();", browser.link(:id => ‘blah’)
[/code]

Hope this helps!

AMA: Misc

Amruth asks…

Hi Scott, I am Amruth and I am from INDIA. I would like to automate my project by using Watir with Ruby. Plz brief me about Watir and Watir-Webdriver.

My response…

I suggest you visit http://watir.com/guides/ and follow the guides. Have fun.


Victor Hugo dos Santos asks…

How we can run our tests parallel when we use Cucumber-JVM?

My response…

I haven’t done this personally, but these instructions look helpful.


Viktoriya Musiy asks…

Just to add: I’ve tried your instruction https://watirmelon.blog/2010/12/09/how-to-set-up-cucumber-and-watir-on-osx/. Unfortunately it does not work on my macbook. I get the error message: You don’t have write permissions for the /usr/bin directory. Viktoriyas-MBP:~ viktoriyamusiy$ sudo gem install rspec –no-rdoc –no-ri Password: ERROR: While executing gem … (Gem::FilePermissionError) You don’t have write permissions for the /usr/bin directory. Viktoriyas-MBP:~ viktoriyamusiy$

My response..

That post is over seven years old so I am not surprised it no longer works. Please try watir.com for up to date instructions.


Niopvief asks…

Food of fried tomatoes like absolutely everything. Guests usually require recipe are surprised mastery hostess. Even guys with pleasure eat fried tomato in breadcrumbs. Tomatoes very good combined are combined with many products. Garlic adds sharpness. Cheese brings ruddy crust. Italian herbs turn roasted tomatoes into a dish from a restaurant. Omelette with tomatoes – easy and nutritious dish for breakfast. Make including sandwiches with fried tomatoes. Cooking tomatoes simply. For cooking use butter. In olive oil – fewer calories. It is also combined mixed with fried cubes tomatoes.

My response…

I am not a big tomato fan but my wife likes them. Thanks for the tips – Merry Christmas

AMA: automated unit vs component tests

Jason asks…

Hi Alister, Love your blog and the content. I have matured my knowledge in test automation, and without even meaning to, created a very similar test automation pyramid you derived. From it, though, i have a difficult time when trying to educate the development team the nuances between their unit level automated tests and component automated tests. How would you go about differentiating between the two? Thanks for your time! – JH

My response…

My understanding is that unit and component testing are similar but differ in their focus. For example, say I was building a table, it would consist of many parts or units:

1 x tabletop
4 x leg brackets
4 x table legs
8 x bolts

Unit testing would be testing each individual part (or unit) to make sure it is good quality but ignoring anything it connects to or requires.

Component testing would be broader in that whether the table legs work with the leg brackets as leg components, and whether the bolts with work the tabletop. I would call this component testing.

Finally testing the table fits together as whole I would call system testing, how it looks in a room or what it’s like to use: end-to-end or user acceptance testing.

When I was developing a Minesweeper game I wrote unit tests for the smallest units (eg. cells) and then component tests for groupings of cells (fields) and system tests for the game itself (interacting with fields).

The reason to do component testing is that it’s more realistic than unit testing so it’s likely to find problem where units interact. The downsides is it’s takes more time to execute and can be harder to isolate problems when they occur.

I hope this helps.

→ How Canaries Help Us Merge Good Pull Requests

I recently published an article on the WordPress.com Developer’s Blog about how we run automated canary tests on pull requests to give us confidence to release frequent changes without breaking things. Feel free to check it out.

AMA: Difference between explicit and fluent wait

Anonymous asks…

What is the difference between Explicit wait and Fluent wait?

My response…

I hadn’t heard of fluent waiting before, only explicit and implicit waiting.

From my post about Waiting in C# WebDriver:

Implicit Waiting

Implicit, or implied waiting involves setting a configuration timeout on the driver object where it will automatically wait up to this amount of time before throwing a NoSuchElementException.

The benefit of implicit waiting is that you don’t need to write code in multiple places to wait as it does it automatically for you.

The downsides to implicit waiting include unnecessary waiting when doing negative existence assertions and having tests that are slow to fail when a true failure occurs (opposite of ‘fail fast’).

Explicit Waiting

Explicit waiting involves putting explicit waiting code in your tests in areas where you know that it will take some time for an element to appear/disappear or change.

The most basic form of explicit waiting is putting a sleep statement in your WebDriver code. This should be avoided at all costs as it will always sleep and easily blow out test execution times.

WebDriver provides a WebDriverWait class which allows you to wait for an element in your code.

As for fluent waits, according to this page it’s a type of explicit wait with more limited conditions on it. I don’t believe WebDriverJs supports fluent waits.

AMA: Dealing with Docker

Howdy! First, thanks to Alister for asking me to guest-post on his blog. I’m always excited to talk about Docker and its potential to solve all the world’s problems 😉.  He asked me to take on the following question from his AMA:

Alister, What are your thoughts on how containerization should fit into a great development and testing workflow? Have you got behind using Docker in your day to day? Thanks!

One of the oldest problems in software development and testing is that a developer writes code on their desktop, where everything works flawlessly, but when it’s shipped to the test or production environments it mysteriously breaks. Maybe their desktop was running a different version of a specific library, or they had unique file permissions enabled. When used correctly, Docker eliminates the “works on my machine” concern. By packaging the runtime environment configuration along with the source code you ensure that the application executes the same in every instance. And just as important, changes to that configuration are logged and can be easily reverted.

Another concern is how to test specific behavior that only gets executed when your application is running in the production environment. By putting all of your application and test servers in individual containers, you can easily connect them on their own private network and just tell the application that it’s running in production. Obviously this is application unique, and building a copy of the production environment presents its own challenges, but at the core it’s definitely doable within a Docker infrastructure. The important thing is that a network of containers is isolated, so you can do things like set machine hostnames to exactly match their production counterparts without worrying about conflicts.

It all just boils down to consistency…if you can ensure that your developer is writing code against the same configuration as your test environment, which is the same configuration is production – everybody wins.

The second part of the question is a little trickier – Have you got behind using Docker in your day to day?

In some ways the answer is yes. The main application that we test is the Calypso front-end to WordPress.com, which itself is built and runs inside Docker. Our core end-to-end tests also run in a custom Docker container on CircleCI 2.0, so we can define exactly what version of NodeJS and Chrome we’re using to test with. However, some of our other test sets (such as certain WooCommerce and Jetpack tests) still run using the default CircleCI container. And as far as I know nobody on our team actually uses that container for developing tests locally, we typically just run directly on our laptops. The CI server is the first place that actually executes via Docker.

The other piece that’s missing for a full Dockerization of the our test setup is that our Canary tests run against the custom https://calypso.live setup (https://github.com/Automattic/calypso-live-branches) rather than directly building/running Calypso side by side in a container. It’s something I’d like to pursue updating at some point, but in the interim the existing setup works great…and most importantly it’s already built and working, allowing us to focus on other things.

So the long story short here is that containerization is a great technology, and has a ton of potential for solving problems in the dev/test world. We’re just scratching the surface of that potential at Automattic, but even the limited use we’re giving it right now is beneficial and I plan on continuing to dig deeper.

AMA: Moving automated tests from Java to JavaScript

Anonymous asks…

I am currently using a BDD framework with Cucumber, Selenium and Java for automating a web application. I used page factory to store the objects and using them in java methods I wanted to replace the java piece of code with javaScript like mocha or webdriverio. could you share your thoughts on this? can I still use page factory to maintain objects and use them in js files

My response…

What’s the reasoning for moving to JavaScript from Java? Despite having common names, there’s very little otherwise in common (Car is to Carpet as Java is to JavaScript.)

I wouldn’t move for moving sake since I see no benefit in writing BDD style web tests in JavaScript, if anything, e2e automated tests are much harder to write in JavaScript/Node because everything is asynchronous and so you have to deal with promises etc. which is much harder to do than just using Java (or Ruby).

Aside: I still dream of writing e2e tests in Ruby: it’s just so pleasant. But our new user interface is written extensively in JavaScript (React) so it makes sense from a sustainability point of view to use JS over Ruby.

 

Why you should use CSS selectors for your WebDriver tests

I didn’t used to be a fan of CSS selectors for automated web tests, but I changed my mind.

The reason I didn’t use to be a fan of CSS selectors is that historically they weren’t really encouraged by Watir, since the Watir API was designed to find elements by type and attribute, so the Watir API would look something like:

[code language=”ruby” light=”true”]
browser.div(:class => ‘highlighted’)
[/code]

where the same CSS selector would look like:

[code language=”css” light=”true”]
div.highlighted
[/code]

Since WebDriver doesn’t use the same element type/attribute API and just uses findElement with a By selector, CSS selectors make the most sense since they’re powerful and self-contained.

The the best thing about using CSS selectors, in my opinion, is the Chrome Dev Tools allows you to search the DOM using a CSS selector (and XPath selectors, but please don’t use XPath), using Command/Control & F:

chrome css selectors
Using CSS selectors to find elements in Chrome Dev Tools

So you can ‘test’ your CSS in a live browser window before deciding to use it in your WebDriver test.

The downside of using CSS selectors are they’re a bit less self explanatory than explicitly using by.className or by.id.

But CSS selectors are pretty powerful: especially pseudo selectors like nth-of-type and I’ve found the only thing you can’t really do in CSS is select by text value, which you probably shouldn’t be doing anyway as text values are more likely to change (since they’re copy often changed by your business) and can be localised in which case your tests won’t run across different cultures.

The most powerful usage of CSS selectors is where you add your own data attributes to elements in your application and use these to select elements: straightforward, efficient and less brittle than other approaches. For example:

[code language=”css” light=”true”]
a[data-e2e-value="free"]
[/code]

How do you identify elements in your WebDriver automated tests?

AMA: Trunk Guardian Service?

Sue asks…

I read a LinkedIn blog post from 2015 by Keqiu Hu from LinkedIn about flaky UI tests. He explains how they fixed their flaky UI tests for the LinkedIn app. Among other things they implemented what they called the “Trunk Guardian service” which runs automated UI tests on the last known good build twice and if the test passes on the first run but fails on the second it is marked as ‘flaky’ and disabled and the owner is notified to fix it or get rid of it. I wondered what your thoughts were on such a “Trunk Guardian service” – if the culture / process was in place to solve the other issues that create flaky tests, could such a thing be worth the effort to implement? Article: Test Stability – How We Make UI Tests Stable

My response…

Continue reading “AMA: Trunk Guardian Service?”

AMA: IE11 Button Clicking in Selenium

Anthony asks…

I have coded to click buttons on IE11/Win7 but the latest version of Selenium IE doesn’t click the buttons correctly most of times. Most of times, it clicks one button below. I thought it might be loading time so added some waiting but still click one button below or two buttons below sometimes. I googled this and found several posting saying Selenium IE doesn’t click buttons well. Now I have moved it to FF but I am still wondering why IE is not accurate. I know a lot of Selenium test developers in the field but they are having the same issue or they know a workaround. What do you think of this issue on IE11? Are you aware of this issue? FYI, the buttons are not regular HTML tag. The menu system with clickable tag is created by javascript. Thank you!

My Response…

We actually don’t run any tests in Internet Explorer any more since these weren’t finding any browser specific bugs (we do exploratory testing in Internet Explorer instead).

But, I have heard of problems generally with the IEDriver tool. If you’re working on a JavaScript generated app I think the best thing for you to do would instead of using a native click in Selenium is instead execute a JavaScript click event. The exact syntax will depend on which language you’re using Selenium in, but it should look something like this:

[code]this.driver.executeScript( ‘return arguments[0].click();’, webElement );[/code]

I hope this solution helps!

AMA: Iterative vs Incremental

Anonymous asks…

What is the difference between iterative and incremental models?

My response:

Fortunately I have written an entire post on this exact topic here.

My conclusion was:

We can’t build anything without iterating to some degree: no code is written perfectly the second that it is typed or committed. Even if it looks like a company is incrementally building their software: they’re iteratively building it inside.

We can’t release anything without incrementing to some degree: no matter how small a release is, it’s still an incremental change over the last release. Some increments are bigger because they’ve already been internally iterated upon more, some are smaller as they’re less developed and will evolve over time.

So, we develop software iteratively and release incrementally in various sizes over time.

AMA: How do I do data migration testing?

Nathan asks…

Data Migration testing from one application to another application. Which way to test best and easy? The new application should be in Salesforce.

My Response…

This is quite a generic question but I’ll try to answer it the best I can. I usually look at data migrations as three separate steps:

Extract data from the old system
Transform the data to fit the new system
Load the data into the new system

I would test that each step has worked correctly by verifying the data starting in the deepest parts of the system (database tables), moving up into APIs and finally into any user interfaces. I know some CRMs such as Salesforce don’t allow access to database tables so sometimes you can only use APIs or user interfaces to ‘spot check’ data.

I hope this helps you Nathan.

AMA: CodeceptJS support for Safari and IE?

Sahana Asks…

We area VOD startup and we have web app, mobile apps and TV apps. I am writing acceptance tests for web app now and chose codeceptjs framework since we have our website’s front end code in Javascript. We have dockerised the processes and docker images for codeceptjs webdriver IO is availble only for chrome and firefox browsers. How can I handle Safari , Internet Explorer browsers ? Looks like CodeceptJs does not support IE and Safari browsers. Do you have any suggestion?

My Response…

I’ve never personally found the return on investment of getting automated tests running across Internet Explorer and  Safari to be worthwhile as in my experience this took more effort than the bugs it found. So I personally stick to running our full e2e test suite in our most used browser (Chrome) and supplementing this with exploratory testing on all other browsers.

In saying that the reason you won’t be able to use Docker containers for these purposes is that they’re Linux and Internet Explorer requires Microsoft Windows and Safari requires Apple macOS to be able to run. To be able to use these for your existing automated tests you can sign up to a on-demand browser service like SauceLabs and use the remote WebDriver protocol to execute your tests.

AMA: What sets exceptional QA testers apart?

Dayana asks…

I wondered if you could tell me what sets exceptional QA testers apart? Not just personality or work ethic traits, but specific skills and programming knowledge that will be very valuable to a team?

My response…

I think exceptional QA testers, as explained recently, aren’t people who are exceptional at just one thing, eg. testing, but good at lots of things.

So an exceptional QA tester, in my opinion, will typically have (at least good) skills in the following things:

  1. Skills in human exploratory testing: an exceptional QA tester has the ability to effectively find the most important bugs fast. Whilst this skill can be developed, I have found it’s mostly a mindset.
  2. Skills in developing automated tests: an exceptional QA tester will have programming skills needed to develop automated tests and I would recommend these to typically match the programming language(s) that programmers in your organization use. For example, skills in automated testing in .NET if your company primarily uses Microsoft .NET. Although, someone with strong programming skills in one language (eg. ruby) should be able to transfer these skills to another language (eg. python).
  3. Knowledge/Experience in your business domain: an exceptional QA tester will fully understand your business domain and keep this context in mind whilst testing a product and raising issues. An exceptional tester is always testing your system – just as I am testing WordPress.com publishing this post.
  4. An empathetic mindset: we design and develop software for real people and real life. An exceptional QA tester will test with this in mind.

iOS11 Screen Recording (for bug reports)

iOS11 is out and it supports native in-device screen recording! I’m a huge fan of attaching screen recordings to bug reports (they capture flow!), whether that be animated gifs or videos, and until now to do this on iOS required plugging your iDevice into your Mac via USB and using QuickTime to record the screen. Alas no longer, it’s now easy to do so via control centre!

Here’s a screen recording (no audio), recorded on my iPhone no less, that shows you how 😊

Protip: force touch the screen recording button to enable/disable microphone for narration

Protip #2: use the in built video editor to trim the start and end