Eight lessons learned in mobile app testing

I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned:

1 – Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the opposite. Whilst Appium shows some promise as a cross-platform UI testing tool for mobile apps, in my opinion, the maintenance overhead is still high as the tests are quite flaky. I recommend more thorough automated unit and integration testing, possibly supplemented with WebView tests (see below) and some exploratory manual app testing of the UI.

2 – Base your mobile OS testing on projected usage, not current usage. Mobile hardware and OS upgrade cycles are shorter than desktop hardware, and if you look at trends and test against those, chances are you will have more realistic usage based testing.

This chart from mixpanel shows iOS7 adoption across a very large number of devices. You can see iOS6 is trending downwards as iOS7 is broadly adopted, so you could put less focus on testing iOS6.

ios7 projected usage

3 – Test on real devices: I prefer to test on real devices like phones and tablets so that I can get a realistic feel for how the app runs. There’s also some things that can’t be tested on emulators/simulators like push notifications which require a device id, and Apple VoiceOver for accessibility testing which isn’t available in the simulators.

I have found that iPod Touch’s make excellent test iOS devices as they’re a lot cheaper than iPhones, and you can buy them refurbished on the Apple store, although if you want to test in the wild (see below), you’ll need a cellular iOS device. If you’re after an iOS6 specific test device you’re only option is to buy a 4th generation iPod touch refurbished as all new Apple iOS devices now come with iOS7 pre-installed.

It’s easy to find cheap Android pre-paid phones for testing, and you can still buy Android 2.3 phones with lower-res screens which is good for older version compatibility testing.

4 – Enable wireless distribution/installation of your test application locally: to enable quick installation of updates to your app on both iOS and Android.

For wireless iOS distribution this involves have a build machine (which has to be a Mac) compile an IPA file which is copied to a web-server, along with an XML manifest (PLIST) file that describes the app. You then have a simple web page with a itms-services link to the manifest, which can be opened in Safari on iOS, prompting the user to automatically install. If you sign your iOS app using an enterprise account it will run on an unlimited number of iOS devices without registering them as test devices. Full details are here.

For wireless Android distribution it is much easier. Simply have your build server compile an APK file and copy it to a web-server along with a page linking to the APK directly. Any Android phone set to allow installation of apps outside the Play Store will download the file and prompt to install it. Voila.

5 – Test in the wild: I like testing during my bus/train ride home to see how my app handles with intermittent cellular network coverage, especially in underground tunnels. If your backend services aren’t publically available you can use a VPN client on your phone to allow you to test against local infrastructure.

The one thing you need to be aware of is other people on your bus/train stickybeaking if your app is commercially sensitive.

6 – Test WebView/bridged app HTML content independently of your app: we use a service that returns HTML5 content that is displayed and executed using a JavaScript bridge in both iOS and Android native apps. This content can be tested independently of the native apps which is both faster and easier than testing via the apps themselves.

7 – Fake back-end dependencies for quicker UI testing. Back-end dependencies of your application such as a database can be faked so you can quickly test your app’s UI is functioning as designed initially without having to worry about setting up an entire application stack for your testing.

8 – Test for accessibility on real devices: using iOS VoiceOver & Android TalkBack. These are both excellent screen readers which use very similar gestures to control the operating system and apps on your phone. The most common accessibility issues I discover are missing accessibility labels for UI controls and embedded WebViews not being marked accessible to the screen reader tools.

Summary

There we have it: eight tips for testing mobile apps. What lessons have you learned in the mobile testing space?

Using appium in Ruby for iOS automated functional testing

As previously explained, I recently started on an iOS project and have spent a bit of time comparing iOS automation tools and chose Appium as the superior tool.

The things I really like about Appium is that it is language/framework agnostic as it uses the WebDriver standard WIRE protocol, it doesn’t require any modifications to your app, supports testing web views (also known as hybrid apps) and it supports Android since we are concurrently developing an Android application (it also supports OSX and Firefox OS but we aren’t developing for those, yet). There isn’t another iOS automated testing tool, that I know of, that ticks that many boxes for me.

Getting Started

The first thing to do is download the appium.app package from the appium website. I had an issue with the latest version (0.11.2) launching the server which can be resolved by opening the preferences and checking “Override existing sessions”.

You run the server from inside the appium.app which takes your commands and relays them to the iOS simulator. There’s also a very neat ‘inspector’ tool which shows you all the information you need to know about your app and how to identify elements.

Note: there’s currently a problem with XCode 5.0.1 (the latest version as I write) which means Instruments/UIAutomation won’t work at all. You’ll need to downgrade (uninstall/reinstall) to XCode 5.0 to get appium to work at all.

Two Ruby Approaches

This confused me a little to start, but there’s actually two vastly different ways to use appium in ruby.

1) Use the standard selenium-webdriver gem

If you’re used to using WebDriver, like me, this will be the most straightforward approach (this is the approach I have taken). Appium extends the API to add different gestures by calling execute_script from the driver, so all other commands stay the same (for example, find_element).

2) Use the appium_lib library

There is a Ruby gem appium_lib that has a different API to the selenium-webdriver gem to control appium. I don’t see any massive benefits to this approach besides having an API that is more specific to app testing.

Using Selenium-WebDriver to start appium in ruby

Launching an appium app is as simple as defining some capabilities with a path to your .app file you have generated using XCode (this gets put into a deep folder so you can write the location to a file and read it from that file).

capabilities = {
'browserName' => 'iOS',
'platform' => 'Mac',
'version' => '6.1',
'app' => appPath
}
driver = Selenium::WebDriver.for :remote,
desired_capabilities: capabilities,
url: "http://127.0.0.1:4723/wd/hub"

Locating elements

Once you’ve launched your app, you’ll be able to use the appium inspector to see element attributes you can use in appium. Name is a common attribute, and if you find that it’s not being shown, you can add a property AccessibilityIdentifier in your Objective C view code which will flow throw to appium. This makes for much more robust tests than relying on labels or xpath expressions.

driver.find_element(:name, "ourMap").displayed?

Enabling location services for appium testing

This got me stuck for a while as there’s quite a bit of conflicting information about appium on how to handle the location services dialog. Whilst you should be able to interact with it as a normal dialog in the latest version of appium, I would rather not see it at all, so I wrote a method to copy a plist file with location services enabled in it to the simulator at the beginning of the test run. It’s quite simple (you can manually copy the clients.plist after manually enabling location services):

def copy_location_services_authentication_to_sim
source = "#{File.expand_path(File.dirname(__FILE__))}/clients.plist"
destination = "#{File.expand_path('~')}/Library/Application Support/iPhone Simulator/7.0/Library/Caches/locationd"
FileUtils.cp_r(source, destination, :remove_destination => true)
end

Waiting during appium tests

This is exactly the same as selenium-webdriver. There’s an implicit wait, or you can explicitly wait like such:

driver.manage.timeouts.implicit_wait = 10
wait = Selenium::WebDriver::Wait.new :timeout => 30
wait.until {driver.find_element(:name, 'monkeys').displayed? }

Mobile gestures

The obvious difference between a desktop web browser and a mobile app is gestures. Appium adds gestures to WebDriver using execute_script. I recommend using the percentage method (0.5 etc) instead of pixel method as it is more resilient to UI change.

For example:

driver.execute_script 'mobile: tap', :x => 0.5, :y => 0.5

or

b = driver.find_element :name, 'Sign In'
driver.execute_script 'mobile: tap', :element => b.ref

Testing Embedded Web Views

The native and web views seamlessly combine so you can use the same find_element method to find either. The appium.app inspector displays the appropriate attributes.

Note: I can’t seem to be able to execute a gesture (eg. swipe) over a Web View. I don’t know whether this is a bug or a limitation of Appium.

Summary

I have found that using the familiar selenium-webdriver gem with appium has been very powerful and efficient. Being able to open an interactive prompt (pry or irb) and explore your app using the selenium-webdriver library and the appium.app inspector is very powerful as you can script on the fly. Whilst appium still seems relatively immature, it seems a very promising approach to iOS automation.

Now to get watir-webdriver to work with appium.