Mobile Application Testing
With the advent of smartphones, a mobile phone has evolved from being just a communication device to a highly sophisticated business utility equipment. With smartphones no longer remaining a conspicuous consumption, they have become an efficient tool for handling finances, managing mails & agendas, and executing other business functions. Statistics highlight that 1.6 billion smartphones are sold at a rate of 1.8 to 2 million devices a day since 2007, and 1.5 billion smartphones will be shipped in 2016 – a 5.7 percent increase from 2015 – and will reach 1.92 billion by 2020. Mobile applications are the sole reason for the rise in popularity of smartphones. The ease and convenience of mobile applications usage has created a huge dependency on it. Over the years, the number and variety of consumer and enterprise mobile applications have grown phenomenally. Considering the abundance of mobile applications and the low cost of purchasing one, uninstalling a non-responsive application is just a touch away. This puts a lot of pressure on organizations developing mobile applications. They must adhere to stringent quality control measures to ensure they do not lose out on the business significance. The development of the mobile application is a very complex process and hence the possibility of bugs and errors cannot be ruled out. Testing of mobile applications becomes a pre-launch mandate for organizations to ensure that they stay ahead of the competition. Another aspect highlighting the importance of mobile testing is the variety of applications available for the public. Hence an error-prone 'app' is very unlikely to be forgiven by people and soon the app would be on its way out (of business).
Testing is an activity of verifying whether a system or a product meets customers’ requirements. While functional testing verifies the intended functionality of the system/product, non-functional testing, however, verifies how the system operates and the parameters which are not related to specific function or action. Mobile application testing verifies installation, compatibility, responsiveness, consistency, and layout of the application in addition to the functional and non-functional aspect of it. Wikipedia defines the term Mobile Application Testing as, ‘a process by which application software developed for handheld mobile devices is tested for its functionality, usability, and consistency. Mobile applications either come pre-installed or can be installed from mobile software distribution platforms.’’ Mobile application testing is not a conventional desktop testing and has to be device and platform agnostic. Due to varied OS platforms and browsers, numerous compatibility issues can be identified in addition to the functional and non-functional issues. So there is a need for focused application testing on mobile devices. Mobile applications are classified into Native, Web and Hybrid, the testing strategy for each would differ as per the nature of the application. Native applications are built for the specific device and its operating system, for example, an application built for the android device may work only on android devices. Mobile testing for native app ensures smooth download and installation of the app. It will also cover backward compatibility
Web applications are for different platforms and are accessible across the internet without being explicitly installed on any specific device or platform. The web application can be accessed from anywhere so the testing strategy would not require downloading and installation of the application. Hybrid applications are a combination of both native and web applications. Most crucial and complex testing would be testing of hybrid applications, which can be executed anywhere and on any device. Checking compatibility on each platform with intended functionality is cumbersome. Thus in all to have a robust testing strategy for the mobile application is important. Developing a testing strategy in end user’s perspective is very important. While using the application the common challenges faced by mobile application users are: The application is unresponsive for a long time The application crashes The application consumes significant bandwidth The impending security concerns.
Challenges faced while testing mobile applications
The multimodal platforms, operating systems, and device types create a plethora of challenges for a mobile application. Some common challenges include: Range of Mobile Devices There are innumerable types of mobile devices available in the market. Consumers have multiple selection choices – ranging from handsets to smartphones, tabs, iPads, wearable tech, etc. This creates a huge diversity of environments that a mobile app faces. Mobile testing in this wide range of devices and environments is challenging, especially when the size of the device plays a major role with over 130 resolutions. Device Fragmentation & Various OS Platforms As per the customer’s requirement, mobile apps can be installed on any operating systems like iOS, Android, Blackberry, Windows, etc. Real-world testing on these many OS is challenging, as mobile apps behave differently on each OS. A mobile application can also be deployed on any environment or device that encompasses multiple versions of an operating system such as iOS 8, iOS 7.X, BB 10.X, Windows 8.X, Windows 7.X, 2. Mobile App Types Mobile apps are not just limited to being native, it has evolved to being web and hybrid app. Mobile testing of each type of app is different. The end-users always require the mobile app to be supported on all kind of environments. Numerous Test Interfaces Exhaustive testing on a real device is not possible all the time with numerous test interfaces. Emulators and Simulators are important tools and they are used to verify general functionality and perform regular regression testing. Since the testing in a virtual environment will not give the real-world test results, so their scope is very limited. With the huge quantum of mobile applications competing to capture the biggest market share, there is a greater need for a robust testing methodology. Thus to say, ‘Mobile Testing is important’ is an understatement!
A holistic test strategy is adopted encompassing emulators, real devices and Cloud for reducing cost and manual effort. Having the right mix of emulators, real devices and Cloud apps, is a unique testing strategy for maximum testing coverage. It covers four main aspects while developing mobile application testing strategy:
Special focus is given to critical cases like poor network, low signal strength, unexpected impacts on memory, low battery power, specific constraints as per device, and recovery after fatal error.
Real-world testing is required with different screen sizes to check the downloading and installation process for the native and hybrid app, checking the responsive screen with a change in resolutions, configurations, and technical compatibility. It covers functional and usability testing, and validation of all tests in a testing lab before the app goes live. Emulators Requirement keeps changing till the time app ships. So it’s not feasible to repeat real-world testing after every requirement change or implementation. Emulators are used to identifying bugs and functional problems. To consider the impact of security and the network, testing is moved to the real device. Unlike conventional scenarios, it is important to anticipate the unexpected behavior of the mobile app in certain situations and check the performance and other conditions end-users may often run into. The testing team considers the following key considerations while testing using emulators:
- Bad Network
- Low signal Strength
- Unexpected impacts on memory
- Low battery power
- Specific constraints as per the device
- Recovery after fatal error
Many times the budgeting is low with strict timelines, with high costing per test or a large number of test cases. In such scenarios, the testing team adopts Cloud testing solution. It has unlimited storage, quick availability of the infrastructure with scalability, flexibility and availability of a distributed testing environment to reduce the execution time of testing of large applications and this leads to cost-effective solutions.
In addition to conventional testing, it is important to test the performance and security of mobile apps. This is quite different from performance testing of web or desktop application. With more and more users moving to smartphones the margin of performance is getting slimmer and hence performance testing becomes an integral part of mobile app testing. Using emulators, the same test can be run on thousands of devices to check the performance of the application, since it also reduces the testing cost and time. This cloud testing solution can also be used for web-based and hybrid apps. It is difficult to test the performance of a mobile app in the real world using actual devices. Depending on customer-specific needs, performance testing on real devices can be achieved.
With people executing huge monetary transactions over their smartphones, hackers also start targeting mobile apps. To avoid the penetration of hackers to mobile apps - security testing is essential. Following techniques are used for mobile security testing:
- Penetration testing:
This test is done with the intention of finding security weakness by penetrating dummy data into the system
- Password cracking:
This is the process of recovering password from data which is stored in the system
- SQL injection:
Inserting malicious SQL statements into an entry field for execution
- Cross-site scripting:
Inserting malicious script.
A solution has been devised that has tremendously reduced the manual testing efforts with approximately 70-80% ROI. A Regression, Smoke and Functional mobile automation suite has been delivered. An automated test suite is discovered from the manual test cases, and a strategy is devised to automate required test cases. Around 70% to 90% manual test cases will be automated to ensure a maximized ROI. Depending on the requirement and application under test, a customized automation framework is created that is device-specific. The reporting and quality metrics are also customized as per the requirement, and updated on the client’s Knowledge Base along with other process improvements.
Devising a testing strategy goes a long way in minimizing bugs and increasing efficiency of the mobile application. The testing strategy has to ensure 100% test coverage with the best-fit tools and simulators. With new technologies introducing new mobile ecosystems, it is also necessary to manage a proper Knowledge Base. Though mobile application testing has innumerable challenges, the key is to devise an end-to-end testing strategy that blends manual and automated testing. The testing methodology should also evolve with the evolving mobile ecosystems