AMA: mobile automation tests

ccy asks…

Do you have plan to do some work in the area of mobile automation test?

My response…

I must say I haven’t had a lot of recent experience with mobile UI automation tools. A couple of years back we started using Appium but abandoned this effort as the tests were so fragile we could never get green builds, so we focused on writing unit tests for the specific mobile code, testing webviews from WebDriver, and doing human end to end testing on real devices. This worked well at the time.

One of my colleagues is implementing some e2e automated tests for the WordPress.com Android and iOS native apps, I will keep an eye on this and share my thoughts, or ask him to share his as we go.

AMA: Mobile simulators/emulators or real devices for testing?

Paul asks…

“Do you prefer testing on simulators/emulators or real devices when testing mobile apps, how do you decide?”

My response…

I don’t currently do a huge amount of mobile app testing but my general guideline is I try to do any mobile app automated testing on simulators/emulators and any mobile app manual/exploratory testing mobile on real devices.

The reasoning is emulators/simulators are more efficient and scale easily to use for automated tests when running in a CI environment. I understand that there are services that offer real devices on demand to run tests on but I can’t see the point for automated testing.

I do however think you’re more likely to find bugs on real devices as you’re using all the physical sensors/inputs like touch and electrical interference/weak signals. I have found that testing on real devices out in the field, whether that’s catching a bus/train, or taking your device to a shopping centre/library etc. is often a great way to find bugs.

Eight lessons learned in mobile app testing

I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned:

1 – Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the opposite. Whilst Appium shows some promise as a cross-platform UI testing tool for mobile apps, in my opinion, the maintenance overhead is still high as the tests are quite flaky. I recommend more thorough automated unit and integration testing, possibly supplemented with WebView tests (see below) and some exploratory manual app testing of the UI.

2 – Base your mobile OS testing on projected usage, not current usage. Mobile hardware and OS upgrade cycles are shorter than desktop hardware, and if you look at trends and test against those, chances are you will have more realistic usage based testing.

This chart from mixpanel shows iOS7 adoption across a very large number of devices. You can see iOS6 is trending downwards as iOS7 is broadly adopted, so you could put less focus on testing iOS6.

ios7 projected usage

3 – Test on real devices: I prefer to test on real devices like phones and tablets so that I can get a realistic feel for how the app runs. There’s also some things that can’t be tested on emulators/simulators like push notifications which require a device id, and Apple VoiceOver for accessibility testing which isn’t available in the simulators.

I have found that iPod Touch’s make excellent test iOS devices as they’re a lot cheaper than iPhones, and you can buy them refurbished on the Apple store, although if you want to test in the wild (see below), you’ll need a cellular iOS device. If you’re after an iOS6 specific test device you’re only option is to buy a 4th generation iPod touch refurbished as all new Apple iOS devices now come with iOS7 pre-installed.

It’s easy to find cheap Android pre-paid phones for testing, and you can still buy Android 2.3 phones with lower-res screens which is good for older version compatibility testing.

4 – Enable wireless distribution/installation of your test application locally: to enable quick installation of updates to your app on both iOS and Android.

For wireless iOS distribution this involves have a build machine (which has to be a Mac) compile an IPA file which is copied to a web-server, along with an XML manifest (PLIST) file that describes the app. You then have a simple web page with a itms-services link to the manifest, which can be opened in Safari on iOS, prompting the user to automatically install. If you sign your iOS app using an enterprise account it will run on an unlimited number of iOS devices without registering them as test devices. Full details are here.

For wireless Android distribution it is much easier. Simply have your build server compile an APK file and copy it to a web-server along with a page linking to the APK directly. Any Android phone set to allow installation of apps outside the Play Store will download the file and prompt to install it. Voila.

5 – Test in the wild: I like testing during my bus/train ride home to see how my app handles with intermittent cellular network coverage, especially in underground tunnels. If your backend services aren’t publically available you can use a VPN client on your phone to allow you to test against local infrastructure.

The one thing you need to be aware of is other people on your bus/train stickybeaking if your app is commercially sensitive.

6 – Test WebView/bridged app HTML content independently of your app: we use a service that returns HTML5 content that is displayed and executed using a JavaScript bridge in both iOS and Android native apps. This content can be tested independently of the native apps which is both faster and easier than testing via the apps themselves.

7 – Fake back-end dependencies for quicker UI testing. Back-end dependencies of your application such as a database can be faked so you can quickly test your app’s UI is functioning as designed initially without having to worry about setting up an entire application stack for your testing.

8 – Test for accessibility on real devices: using iOS VoiceOver & Android TalkBack. These are both excellent screen readers which use very similar gestures to control the operating system and apps on your phone. The most common accessibility issues I discover are missing accessibility labels for UI controls and embedded WebViews not being marked accessible to the screen reader tools.

Summary

There we have it: eight tips for testing mobile apps. What lessons have you learned in the mobile testing space?