The main theme for today’s talks was Android UI automation with various approaches demonstrated.
Thomas Knych, Stefan Ramsauer and Valera Zakharov from Google gave a highly entertaining presentation about Android testing at scale. This was one of my favorite talks of the conference. They highlighted that insistence on automated testing using real devices is inefficient and problematic, and that you should first run a majority of tests on emulators which finds a majority of the bugs. This is something I have been saying for a long time and it was refreshing to hear it from a Google Android team. Ways to speed up Android emulators include using snapshots for fast restores, as well as using x86 accelerated AVDs. Interestingly, the Google Android team ran 82 million Android automated tests using emulators in March alone (there are approx 2.5 million seconds in March) with only 0.15% of tests being categorized as flaky. This is partly due to using a Google only automated testing tool for Android called Espresso. Another key takeaway was if you are using physical devices then don’t glue them to a wall or whiteboard. The devices get hot, melt the glue and get damaged as they hit the floor.
Guang Zhu (朱光) and Adam Momtaz also from Google talked about some historical approaches to Android automation (instrumental, image recognition and hierarchy viewer) and how to use features in newer Android API versions (16+) to automate tests reliably.
Jonathan Lipps from Sauce Labs demonstrated the very impressive tool Appium which enables iOS and Android automation using WebDriver bindings allowing you to use your language of choice with the promise to write once and run across the two platforms. This isn’t exactly true as the selectors will be different but these can be defined in a module so your test code is readable. Jonathan explained the philosophy behind the tool and even demonstrated a quick demo running against the new FirefoxOS to demonstrate its flexibility. Some of the limitations mentioned were you can only run one iOS emulator per physical Apple Mac which limits continuous integration scalability. It was overall a very impressive polished tool.
Eduardo Bravo from the Google+ team gave an interesting lightning talk about hands-on experience in testing Google+ apps across Android and iOS. They use KIF for iOS testing. Eduardo was quote worthy with such gems as “flaky tests are worse than no tests” and “don’t give devs a reason not to write tests“. The hermetic theme was recurrent with the ongoing endeavor to reduce flakiness by using hermetic environments with known canned responses to make tests deterministic. A very enjoyable talk.
Valera Zakharov from the Google Android dev team discussed an internal tool Espresso which makes Android tests much more efficient and reliable, and with less boilerplate code. My only complaint: don’t demo an awesome tool that isn’t open source and available for others to use.
Michael Klepikov from Google talked about using the upcoming ChromeDriver 2 server to access performance metrics from the Chrome Developer Tools. He demonstrated some fancy looking results generated by webpagetest.org. I don’t believe you need ChromeDriver 2 to do this though, the W3C navigation timing spec provides performance metrics right now.
Yvette Nameth and Brendan Dhein from the Google Maps team discussed the challenge of testing large Google Maps datasets, demonstrating a risk based approach: eg. Ensuring the Eiffel Tower is accurate is important, but the accuracy of your Gran’s farm is not.
Celal Ziftci and Vivek Ramavajjala from the University of San Diego presented their findings of work at Google to automatically find culprits in failing builds. This was a highly interesting talk about creating a tool to analyze multiple change sets in a build and work out which is most suspicious using a couple of heuristics: number of files changed and distance from root. The tool originally took 6 hours to perform an analysis but they reduced this to 2-3 minutes using extensive caching. The tool they developed allows extensible heuristics to allow additional intelligence such as keyword analysis.
Katerina Goseva-Popstojanova talked about academic analysis of software product line quality. She highlighted that open source software projects are the Promised Land for academia in that the code is fully accessible and can be used for academic analysis and research.
Claudio Criscione from Google discussed Cross Site Scripting (XSS) vulnerabilities and some automated solutions to checking for these.
During the afternoon I went for a tour of the Google New York City office here in Chelsea. All I can say is wow. The view from the 11th floor roof top balcony was very nice too (see pics below).
A very enjoyable and smooth conference and well done to all involved organizing it.