Mobile Automation Testing – What To Consider
Automation testing is routinely included in many software development projects; however it’s still an emerging practice in the development of mobile applications.
There are myriad mobile automation testing tools available on the market, but what are the considerations when deciding which path to choose?
Highlighted below are a few of the important questions which should be asked:
- Should you rely on device emulation, or physical devices? - Ideally, real devices should be used to ensure that your test results are as accurate as possible. If you have budget/time constraints, emulation may also be an option.
- Have you considered the need to test multiple device types/versions? Different operating system versions? It’s important to understand your user population when deciding what devices to use in testing. Given the vast array of devices available (mobile, tablet etc), it’s generally unfeasible to test all device/operating system combinations; therefore a risk based approach should be adopted.
There is information available online to assist you to understand the breakdown of devices by operating system, such as Android Dashboards, iOS Version Stats (courtesy of 14 Oranges), and Blackberry App World. Rather than understanding which devices are selling right now, it’s important to consider the blend of devices which are currently in the wild.
- Do you need to test multiple operating systems? Android vs iOS vs others? It’s undesirable to need to use one tool for Android, one for iOS, etc – you would be wise to choose a tool which supports multiple device types.
- Which mobile design patterns are you using, and is any potential tool capable of testing them? native vs hybrid vs web apps? There are a number of different mobile design patterns in use. Are all of those supported by the tool? What level of assistance is available from the vendor should a gap be uncovered?
- Have you thought about the impact of network conditions, and from different physical locales? Your customers could be using your applications from many different locations, using differing network carriers and under different network conditions. All of these factors could impact the responsiveness of the application and therefore the user experience. The network in a given geographic location might vary widely in terms of speed and latency, dependant on the time of day, and what other users are doing at that time. Applications are best designed with this in mind, in order to reduce the impact of varying network conditions.
- Are you able to collect metrics (such as CPU utilisation, memory, network utilisation) from the device? This information is useful when trying to understand the reason for a poor user experience. In a real life situation, users have multiple applications running at the same time, so the resources available to your application may be more limited than the specifications of the device.
- What skills will your test engineers need to possess? Is extensive scripting knowledge required? How much up-skilling will be required? Scripts should be able to be created swiftly, while allowing for more advanced scripting requirements.
- Are you using Continuous Delivery within your organisation? Is the tool able to integrate into this process? Mobile apps are often delivered by teams using Agile methodology. Given this, it is highly desirable to be able to integrate mobile automation testing with Continuous Delivery processes; this way, the cost of regression testing for each build/release cycle is reduced.
I hope this has given you some food for thought. I haven’t even touched on the area of mobile performance yet, and will be blogging on this topic soon.
Ditch the Performance Test Plan – Risk Based Performance Testing
Most projects I’ve worked have required my team to create a long, formal performance test plan. We then spend weeks meticulously filling out every section of the templated document, only to find out during document sign-off that hardly anyone has… Read more
Plan for Success…..Plan for Performance
Whether you are a developer, performance tester, architect or project manager you’ve likely been on at least one IT project where testing found serious performance, capacity, scalability or stability issue at the eleventh hour. If you’ve been in… Read more
Sherlock Holmes, Kanban and the Art of Performance Remediation
I have recently completed reading the collected stories of Sherlock Holmes by Sir Arthur Conan Doyle, and was struck by the resonance between three memorable quotes, and the processes I have used during performance remediation activities in the last… Read more
Build Pipeline – Software quality through repeatability
What are the hallmarks of a quality software development team? What attributes stand out from software which gives it the feeling of quality? These questions and many more, lead to the seemingly elusive quality answer. Certainly, answers to… Read more
Performance Unit Test; a development concern
As a developer, I feel performance is too often neglected by the development team. I’ve been on a few projects that have been really compromised by the performance aspects of a JEE system, specifically because of the lack of performance testing… Read more
When to avoid the Container during Unit Testing
Over the course of my career, I have been eagerly reading, interviewing colleagues and experimenting with my own unit testing methodologies. It was only recently, that a fellow odecee colleague and I sat in a room and attempted to map out what we… Read more
Is Dependency Injection only useful for Unit Testing?
The rise in popularity of Dependency Injection over the last three years has been an interesting observation. “Lightweight containers” like Spring and Pico have really intrigued and excited the masses of Java Developers, so much so, that Spring… Read more
Unit testing; how far do you push the envelope?
Over the years, l’ve read lots of commentary, white papers, best practice papers, books on the topic of TDD. I’ve heard the rant of many TDD evangelists who preach about how total code coverage brings you closer to code quality perfection and how… Read more