Notes from GTAC 2014 (aka The Flaky Test Conference)

I was lucky enough the attend the Google Test Automation Conference (GTAC) at Google Kirkland in Washington last week. As usual, it was a very well run conference with an interesting mix of talks and attendees.

Whilst there wasn’t an official theme this year, I personally saw two themes emerge throughout the two days: dealing with flaky tests and running automated tests on real mobile devices.

Flaky Tests

There wasn’t too many talks that didn’t mention flaky automated tests (known as ‘flakes’) at some point. Whilst there seemed to be some suggestions for dealing with flaky tests (like Facebook running new tests x times to see if they fail and classify them as flaky and assign to the owner to fix), there didn’t seem to be a lot of solutions for avoiding the creation of flaky tests in the first place which I would have liked to see.

Real Mobile Devices

The obsession of running mobile automated tests on real devices continued from last year’s conference with talks about mobile devices as a service. I personally think we’d be better spending the time and effort on making more realistic mobile emulators that we can scale rather than continuing the real device test obsession.

Key Takeaway

My key takeaway was even highly innovative companies like Google, Facebook and Netflix still struggle balancing software quality and velocity. In Australia, these companies don’t have a strong presence here, and often the IT management of smaller companies here like to say things like “Google does x” or “Facebook does y”. The problem with this is they only know these companies from the outside. Ankit Mehta’s slides at the beginning of his keynote captured this perfectly and hence were my favorite slides of the conference:

Google from the outside (like a jet plane) Google from the inside (lots of people pushing a broken down car)Jonathan Rasmusson also recently wrote about something similar on his blog:

“When we see or admire what one company is doing, we are only seeing it through the lens of the end result. And not the underlying foundations that were setup to get them there.”

Well done to the GTAC committee and Google for putting on another excellent conference. Here’s some photos I took of Seattle/Kirkland during my time there.