Notes from GTAC 2014 (aka The Flaky Test Conference)

I was lucky enough the attend the Google Test Automation Conference (GTAC) at Google Kirkland in Washington last week. As usual, it was a very well run conference with an interesting mix of talks and attendees.

Whilst there wasn’t an official theme this year, I personally saw two themes emerge throughout the two days: dealing with flaky tests and running automated tests on real mobile devices.

Flaky Tests

There wasn’t too many talks that didn’t mention flaky automated tests (known as ‘flakes’) at some point. Whilst there seemed to be some suggestions for dealing with flaky tests (like Facebook running new tests x times to see if they fail and classify them as flaky and assign to the owner to fix), there didn’t seem to be a lot of solutions for avoiding the creation of flaky tests in the first place which I would have liked to see.

Real Mobile Devices

The obsession of running mobile automated tests on real devices continued from last year’s conference with talks about mobile devices as a service. I personally think we’d be better spending the time and effort on making more realistic mobile emulators that we can scale rather than continuing the real device test obsession.

Key Takeaway

My key takeaway was even highly innovative companies like Google, Facebook and Netflix still struggle balancing software quality and velocity. In Australia, these companies don’t have a strong presence here, and often the IT management of smaller companies here like to say things like “Google does x” or “Facebook does y”. The problem with this is they only know these companies from the outside. Ankit Mehta’s slides at the beginning of his keynote captured this perfectly and hence were my favorite slides of the conference:

Google from the outside (like a jet plane) Google from the inside (lots of people pushing a broken down car)Jonathan Rasmusson also recently wrote about something similar on his blog:

“When we see or admire what one company is doing, we are only seeing it through the lens of the end result. And not the underlying foundations that were setup to get them there.”

Well done to the GTAC committee and Google for putting on another excellent conference. Here’s some photos I took of Seattle/Kirkland during my time there.

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

2 thoughts on “Notes from GTAC 2014 (aka The Flaky Test Conference)”

  1. Alister

    Good article. For someone who has been reading and enjoying all of your articles and blogs for almost 4years now but yet to speak my mind, I felt like today is my day. The trend I see in the automation testing world the last couple of years is for most companies to come out and declare their muscle in terms of how many hundreds or thousands of test have they added to their suite. Then there is time and effort they spent on trying to maintain them like you mentioned about declaring tests that are flaky and ones that keep breaking the build cycle etc. I am yet to see companies focus on trying to design scalable tests that are easy to maintain and showcasing the power of their tests written in terms of how many bugs it caught and how much money it saved them during their entire product life cycle and how good their test design was for covering end to end scenarios for their application. For the quality assurance world automation tests focus is on v and v but the industry is getting obsessed with writing the best possible automation program(not tests). Great point on how perception changes when one is only hearing the outcome.


Comments are closed.