Using Node to build an automated local mobile test lab

As a small team focusing on delivering high quality mobile applications we found ourselves in need of an automated system to run our UI (user interface) tests on native iOS and Android devices.

Background

To understand the ideal solution, you first need to be a bit familiar with our CI (Continuous Integration) workflow (see our motivation here).

Each bug fix, new feature, or code change is developed on a new branch.  Once the code is complete; the developer creates a pull request against the main code branch.  At that point all sorts of ‘web magic’ happens: unit tests are run, applications are compiled, test coverage reports are run, notifications are sent out, etc…

Then, after all the automation is complete our QA team (aka me) downloads the code, manually runs the UI tests on a few devices, manually verifies the change, reviews the code changes, and finally (if all is well) merges the changes into our main branch.  My goal was to remove the ‘manually runs the UI tests on a few devices’ portion and move that to the ‘web magic’ portion.

The Current Offerings

There are other ‘test cloud’ their solutions but not one that meets all the needs identified above.  For instance the feedback loop of some test cloud solutions from ‘test requested to results’ can vary greatly.  Some test cloud solutions may be limited by requiring tests to be written using specific languages such as, for instance, Ruby (for Calabash).  Still others require the test code to specifically detail each and every device to be tested.  Moreover, none of the test cloud solutions provide the ability to run on custom devices, report to a good ‘single source of truth’ (e.g., where would you check to see if everything passed??), and each service requires significant tuning overhead to meet testing needs.

Xamarin Test Cloud

Xamarin’s solution is flat-out awesome… if you can afford it.  Their basic offering STARTS at $1000/month with less than priority execution (meaning that the feedback loop from test requested to results can vary greatly).  Further, Xamarin test cloud requires your tests to be written using Calabash, while robust in certain situations, it no longer meets our needs for mobile test tools.

Appthwack

Probably the most reasonably priced offering out there right now. Their test backend also allows tests to be written with Appium (what we use).  However, testing on Appthwack requires some modifications to the test code and also requires the test to specify each and every device to test.

TestDroid Cloud & TestDroid Enterprise

TestDroid Cloud rivals Appthwack with pricing and doesn’t require much test modification.  Although the tests do need to be configured to authenticate to their API.

TestDroid Enterprise allows you to ‘byod’ (bring your own devices) and run unlimited tests on those devices.  This is an attractive offer, except the $99/month charge per device that uses this service.

In short, current test cloud solutions are ill-suited for making smaller incremental changes.  What is needed is a robust, automated system to efficiently run UI (user interface) or other tests on new code on mobile devices executing on one or more mobile platforms / operating systems.

My Solution – ‘Mobile Test Cloud’

Mobile test cloud is a NodeJS app that runs appium tests (written in any language) on a mac (for iOS).

The test cloud kicks off when a Pull Request is created or updated on Github.  It then downloads the repository from Github and installs the tests (if necessary).  Once everything is setup, Mobile Test Cloud waits on the CI environment to finish running unit tests and building the mobile application. Next, it searches for and detects any devices connected and starts testing.  The tests may be run serially across the devices and results may be recorded and saved to a cloud based storage service.  When everything is finished the final results are published to Amazon’s S3 service and the GitHub PR is updated with a Pass or Fail.  The diagram below sets out the test/data flow of the solution.

How Mobile Test Cloud is Different

Single Source of Truth

This, in my opinion, is the most value addition after incorporating Mobile Test Cloud to my project.  Every result: Code Coverage, Unit Tests, & UI Tests are all viewable in the pull request.  Having everything in one location with links to detailed results creates a tighter feedback loop with fewer errors.

BYOD – Bring your own Device

Mobile Test Cloud is BYOD by design.  For Bandwidth, this is incredibly important. We’re testing against custom versions of Android, validating incoming/outgoing calls, sending/receiving SMS and MMS.  All of which are more difficult to test with the existing services.  However, for a webapp where the intended functionality of the app doesn’t change from device-to-device, BYOD may not be the best long term solution.

Minimal to no test change

Depending on how the Appium tests were originally written, there is little to no change to the test code.  Mobile Test Cloud only requires that the mobile application path is not specified within the tests.

Future Revisions

Mobile Test Cloud is an ongoing project. There are a few features/tricks that would benefit the system.

Parallel Tests: Due to limitations with iOS, this is only possible on Android.  Enabling the ability to run tests really only requires the Appium tests to be configured to accept the Appium port.

Build App within Mobile Test Cloud: This would speed up the feedback loop by starting the UI tests at the same time as the Unit tests. However, compiling the mobile applications is prone to error and was out of the initial scope

Results website: Instead of uploading the results to an S3 bucket, it would be better to host a simple website with the results of each run. This would allow developers to check the history of tests without having to find each pull request’s results.

Test queueing: Currently, Mobile Test Cloud can only run one test at a time. A simple change would allow the app to queue up tests and process them in order received.

Conclusion

Mobile Test Cloud set out to be a simple low cost alternative to the existing test cloud services.  However, it grew to be a much more customizable and flexible solution that could be used in tandem with the test services for a high quality application.  The ability to run your app on custom hardware configurations is incredibly powerful for companies like Republic Wireless where their flagship application requires a Republic phone.  Further, adding a device to the automated test process where standard backgroud apps like Facebook, Twitter, Chrome, etc.. take up processing power and memory help your tests simulate real world user experience.