AMA: JavaScript & Mobile App Automation Tools

Justin Watts asks…

Have you looked at Chimp.js ? Any thoughts? I was very closely following your posts on picking a new framework for Automattic and I was quite surprised when you chose an async tool and stepped away from Cucumber. As a ruby dev looking to get comfortable with testing web-apps in JavaScript, I am torn on what library to dig into.

My response…

I had a quick look at Chimp.js when I was evaluating tools, it looks quite impressive, but quite opinionated and integrated, so I find these types of libraries are good to start with, but quickly become frustrating when you want to start doing advanced things (like the custom Slack reporter we just wrote that pings slack when tests fail with screenshots). I am curious about how Chimp.js is synchronous when the underlying WebDriver.IO tool uses promises?

I think the asynchronous thing is less of a big deal than most people make out. Once you get used to using promises whenever querying a value from the browser, then you just do that. I have set up my own es6 page objects so these take away most of this complication anyway.

The one synchronous tool I did seriously contemplate was webdriver-sync but since it wraps the underlying WebDriver Java driver, it was quite heavyweight as you need Java, the node-java bridge, and I couldn’t get this working on CircleCI which is what Automattic’s uses as its CI platform.

As for stepping away from Cucumber. Automattic’s unit tests are written in Mocha, so that was a logical choice as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.

Justin Watts also asks…

Appium seems to be getting more unstable as time goes on. Early Grey caught my eye. Do you have any thoughts / outlooks on the mobile app testing landscape?

My response…

I must say I haven’t had a lot of recent experience with mobile UI automation tools. A couple of years back we started using Appium but abandoned this effort as the tests were so fragile we could never get green builds, so we focused on writing unit tests for the specific mobile code, testing webviews from webdriver, and doing human end to end testing on real devices. This worked well, and unless I worked on complex native apps I would probably end up doing something similar again should this arise.



Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

2 thoughts on “AMA: JavaScript & Mobile App Automation Tools”

Comments are closed.