This code works perfectly when executing against Chrome or Firefox, but when it came to executing against IE11 I would see the following (rather unhelpful) error:
Command duration or timeout: 69 milliseconds
I believe the expression “jack of all trades, master of none” is a misnomer, as I’ve mentioned previously. Being good at two or more complimentary skills is better than being excellent at just one, in my opinion.
But what about being excellent at one skill, and still being good at two or more? Why can’t we be both?
Jason Yip describes a T-shaped person and the benefits that having t-shaped people on teams brings:
A T-shaped person is capable in many things and expert in, at least, one.
As opposed to an expert in one thing (I-shaped) or a “jack of all trades, master of none” generalist, a “t-shaped person” is an expert in at least one thing but also somewhat capable in many other things. An alternate phrase for “t-shaped” is “generalizing specialist”.
Ideally we’d like to have a team of t-shaped testers in Flow Patrol at Automattic. But how do we get to this end goal?
I recently embarked on an exercise to measure and benchmark our skills and do just this with our team. Here’s the steps we took.
Step One – Devise Desired Team Skills
The first thing we did was come up with a list of skills that we have in the team and would like to have in the team. These can be ‘hard’ skills like a specific programming languages and ‘soft’ skills like triaging bugs. In a standard co-located team this would be as easy as conducting a brainstorming session and using affinity grouping to discover these skills. In a distributed environment I wrote a blog post to my team’s channel and had individual members comment with a list of skills they thought appropriate, and then I did the grouping and came up with a draft list of skills and groups.
Step Two – Self-assess against a team skills matrix
Once I had a final list of skills and groups (see below for full list), I put together a matrix (in a Google Spreadsheet) that listed team members on the x-axis, and the skills on the y-axis, and came up with a skill level rating. Our internal systems use a three level scale (Newbie, Comfortable, Expert) which we didn’t think was broad enough so we decided upon five levels:
I hadn’t seen Jason yip’s visual representation at that point in time, otherwise I may have used something like that, which has five similar levels:
Step Three – Publish results and cross-skill
Once we had the self assessments done we could then publish the data within our organisation and use the benchmark to cross-skill people in the team. In a co-located environment this could involve pair programming, in a distributed one it could involve mentoring and reviewing other team member’s work.
Have you done a skills matrix for your team? How did you do it? What did you discover?
Full List of Skills and Skill Groups for Flow Patrol at Automattic
First of all thanks for sharing all your insights on this blog. I regularly try to come back to your blog and I helped me grow as a QA Engineer quite a lot.
Could you help me and point me in the right direction?
You can write synchronous tests (like using Watir/Ruby) against asynchronous web interfaces, you just need to use the waiting/polling mechanisms (or write/extend your own) – the same as you need to do in asynchronous test automation tools.
I have tried to use cypress.io but the way it controls sites (through proxies) has (current) limitations like not working on iFrames and cross-domains which are deal-breakers for our end-to-end testing needs at present.
I’m glad I’ve been able to help you grow over the years.
Hey! In order to test the security of a website, I’m trying to create hundreds of accounts. However, after a certain limit, there is always an error which prevents me from going further.
How can I hide/change my ip address during each iteration with watir?
I can understand why there would be this limitation in place as this activity seems suspicious from a systems perspective.
Watir can’t change the IP address. You can use an anonymous browser profile and/or delete cookies for different account creation runs but this probably won’t help.
I’d speak to your development/devops/systems team to whitelist your IP address(es) you are using for this purpose.
I recently noticed the new Google Chrome project Puppeteer:
Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.
As someone who only runs WebDriver tests in Google Chrome anyway, this looks like a promising project that bypasses WebDriver to have full programmatic control of Google Chrome including for automated end-to-end (e2e) tests.
The thing I really love about this is no Chromedriver dependency and how installing the library installs Chromium by default which can be controlled headlessly with zero config or any other dependencies.
Is there any way to make Watir click a link/button that is not visible? I wanted to switch from Capybara/CapybaraWebKit (which allows clicking non-visible elements) but I am stuck since Watir always times out on the click attempt.
Hi Alister, Love your blog and the content. I have matured my knowledge in test automation, and without even meaning to, created a very similar test automation pyramid you derived. From it, though, i have a difficult time when trying to educate the development team the nuances between their unit level automated tests and component automated tests. How would you go about differentiating between the two? Thanks for your time! – JH
My understanding is that unit and component testing are similar but differ in their focus. For example, say I was building a table, it would consist of many parts or units:
1 x tabletop
4 x leg brackets
4 x table legs
8 x bolts
Unit testing would be testing each individual part (or unit) to make sure it is good quality but ignoring anything it connects to or requires.
Component testing would be broader in that whether the table legs work with the leg brackets as leg components, and whether the bolts with work the tabletop. I would call this component testing.
Finally testing the table fits together as whole I would call system testing, how it looks in a room or what it’s like to use: end-to-end or user acceptance testing.
When I was developing a Minesweeper game I wrote unit tests for the smallest units (eg. cells) and then component tests for groupings of cells (fields) and system tests for the game itself (interacting with fields).
The reason to do component testing is that it’s more realistic than unit testing so it’s likely to find problem where units interact. The downsides is it’s takes more time to execute and can be harder to isolate problems when they occur.
I recently published an article on the WordPress.com Developer’s Blog about how we run automated canary tests on pull requests to give us confidence to release frequent changes without breaking things. Feel free to check it out.
Howdy! First, thanks to Alister for asking me to guest-post on his blog. I’m always excited to talk about Docker and its potential to solve all the world’s problems 😉. He asked me to take on the following question from his AMA:
Alister, What are your thoughts on how containerization should fit into a great development and testing workflow? Have you got behind using Docker in your day to day? Thanks!
One of the oldest problems in software development and testing is that a developer writes code on their desktop, where everything works flawlessly, but when it’s shipped to the test or production environments it mysteriously breaks. Maybe their desktop was running a different version of a specific library, or they had unique file permissions enabled. When used correctly, Docker eliminates the “works on my machine” concern. By packaging the runtime environment configuration along with the source code you ensure that the application executes the same in every instance. And just as important, changes to that configuration are logged and can be easily reverted.
Another concern is how to test specific behavior that only gets executed when your application is running in the production environment. By putting all of your application and test servers in individual containers, you can easily connect them on their own private network and just tell the application that it’s running in production. Obviously this is application unique, and building a copy of the production environment presents its own challenges, but at the core it’s definitely doable within a Docker infrastructure. The important thing is that a network of containers is isolated, so you can do things like set machine hostnames to exactly match their production counterparts without worrying about conflicts.
It all just boils down to consistency…if you can ensure that your developer is writing code against the same configuration as your test environment, which is the same configuration is production – everybody wins.
The second part of the question is a little trickier – Have you got behind using Docker in your day to day?
In some ways the answer is yes. The main application that we test is the Calypso front-end to WordPress.com, which itself is built and runs inside Docker. Our core end-to-end tests also run in a custom Docker container on CircleCI 2.0, so we can define exactly what version of NodeJS and Chrome we’re using to test with. However, some of our other test sets (such as certain WooCommerce and Jetpack tests) still run using the default CircleCI container. And as far as I know nobody on our team actually uses that container for developing tests locally, we typically just run directly on our laptops. The CI server is the first place that actually executes via Docker.
The other piece that’s missing for a full Dockerization of the our test setup is that our Canary tests run against the custom https://calypso.live setup (https://github.com/Automattic/calypso-live-branches) rather than directly building/running Calypso side by side in a container. It’s something I’d like to pursue updating at some point, but in the interim the existing setup works great…and most importantly it’s already built and working, allowing us to focus on other things.
So the long story short here is that containerization is a great technology, and has a ton of potential for solving problems in the dev/test world. We’re just scratching the surface of that potential at Automattic, but even the limited use we’re giving it right now is beneficial and I plan on continuing to dig deeper.
I didn’t used to be a fan of CSS selectors for automated web tests, but I changed my mind.
The reason I didn’t use to be a fan of CSS selectors is that historically they weren’t really encouraged by Watir, since the Watir API was designed to find elements by type and attribute, so the Watir API would look something like:
browser.div(:class => 'highlighted')
where the same CSS selector would look like:
Since WebDriver doesn’t use the same element type/attribute API and just uses findElement with a By selector, CSS selectors make the most sense since they’re powerful and self-contained.
The the best thing about using CSS selectors, in my opinion, is the Chrome Dev Tools allows you to search the DOM using a CSS selector (and XPath selectors, but please don’t use XPath), using Command/Control & F:
So you can ‘test’ your CSS in a live browser window before deciding to use it in your WebDriver test.
The downside of using CSS selectors are they’re a bit less self explanatory than explicitly using by.className or by.id.
But CSS selectors are pretty powerful: especially pseudo selectors like nth-of-type and I’ve found the only thing you can’t really do in CSS is select by text value, which you probably shouldn’t be doing anyway as text values are more likely to change (since they’re copy often changed by your business) and can be localised in which case your tests won’t run across different cultures.
The most powerful usage of CSS selectors is where you add your own data attributes to elements in your application and use these to select elements: straightforward, efficient and less brittle than other approaches. For example:
How do you identify elements in your WebDriver automated tests?
I read a LinkedIn blog post from 2015 by Keqiu Hu from LinkedIn about flaky UI tests. He explains how they fixed their flaky UI tests for the LinkedIn app. Among other things they implemented what they called the “Trunk Guardian service” which runs automated UI tests on the last known good build twice and if the test passes on the first run but fails on the second it is marked as ‘flaky’ and disabled and the owner is notified to fix it or get rid of it. I wondered what your thoughts were on such a “Trunk Guardian service” – if the culture / process was in place to solve the other issues that create flaky tests, could such a thing be worth the effort to implement? Article: Test Stability – How We Make UI Tests Stable
We actually don’t run any tests in Internet Explorer any more since these weren’t finding any browser specific bugs (we do exploratory testing in Internet Explorer instead).
I’ve never personally found the return on investment of getting automated tests running across Internet Explorer and Safari to be worthwhile as in my experience this took more effort than the bugs it found. So I personally stick to running our full e2e test suite in our most used browser (Chrome) and supplementing this with exploratory testing on all other browsers.
In saying that the reason you won’t be able to use Docker containers for these purposes is that they’re Linux and Internet Explorer requires Microsoft Windows and Safari requires Apple macOS to be able to run. To be able to use these for your existing automated tests you can sign up to a on-demand browser service like SauceLabs and use the remote WebDriver protocol to execute your tests.
I wondered if you could tell me what sets exceptional QA testers apart? Not just personality or work ethic traits, but specific skills and programming knowledge that will be very valuable to a team?
I think exceptional QA testers, as explained recently, aren’t people who are exceptional at just one thing, eg. testing, but good at lots of things.
So an exceptional QA tester, in my opinion, will typically have (at least good) skills in the following things:
Skills in human exploratory testing: an exceptional QA tester has the ability to effectively find the most important bugs fast. Whilst this skill can be developed, I have found it’s mostly a mindset.
Skills in developing automated tests: an exceptional QA tester will have programming skills needed to develop automated tests and I would recommend these to typically match the programming language(s) that programmers in your organization use. For example, skills in automated testing in .NET if your company primarily uses Microsoft .NET. Although, someone with strong programming skills in one language (eg. ruby) should be able to transfer these skills to another language (eg. python).
Knowledge/Experience in your business domain: an exceptional QA tester will fully understand your business domain and keep this context in mind whilst testing a product and raising issues. An exceptional tester is always testing your system – just as I am testing WordPress.com publishing this post.
An empathetic mindset: we design and develop software for real people and real life. An exceptional QA tester will test with this in mind.
Do you have set up (inexpensive) infrastructure to store data collected in your automated tests? We are currently using using selenium Java webdriver to automate our tests and IntelliJ as our IDE. We create data from scratch for each and every test case :(
I’m a little confused by the question and whether it’s about test data: data is that is needed by the automated tests, or test results data: insights into the results of our automated tests. So I’ll answer both 😀
Infrastructure to manage test data
Our tests run on specific test accounts and sites on production databases. Since our tests are end-to-end in fashion, we try to make our tests have as few dependencies as possible on existing data. Often an end-to-end scenario will involve creating, viewing, editing and deleting something. If we don’t do all of this by our UI we can use hooks that either use services or database jobs to clean up the data. I explained this in more detail previously.
Infrastructure to manage test results data
We use CircleCI for automated end-to-end tests. We have a number of projects that run different types of end-to-end tests from the same code repository for different purposes (canary tests, visual-diff tests, full regression tests for example).
We generate x-unit test results (from Mocha/Magellan) which CircleCI uses to provide insights into our test results such as this:
You can also drill down into slowest tests and most failed tests etc.
I’ve been working with angular a while now but I have to admit, the testing side throws me. Every time I start to tackle it, I find myself distracted from the test I want to write by all of the things I need to mock. Sometimes it feels like I need to build an entire mock framework to test one feature. Maybe I’m doing it wrong. Any tips appreciated, whether practical or even just the right headspace for approaching it :)
I haven’t worked with angular applications directly but I’ve worked on React applications and I’m guessing the approach to unit testing these will be similar.
I would start as simple as possible with the smallest test that would possibly work. If you find that you need an entire mock framework to test a feature it sounds like your components may need to be broken down further into smaller components as you have too many dependencies that need to be mocked. If you have smaller components that only require a single dependency then these components should be easier to test as the dependencies will be easier to mock.
If retrofitting unit tests into your app is still too difficult you could try to automated some key end-to-end scenarios using Protractor, but I’d discourage you from going overboard since these can quickly get out of hand. You may still benefit from having a few of these tests even if you get unit testing of small components working well to ensure the small components work well together.