Remember how Cersei Lannister from Game of Thrones reacted to dragons? She ordered her advisers to find a weapon against them. The weapon was found. They were huge scorpion crossbows. However, the queen wanted to be sure that such a weapon was a real threat to the dragon. She needed to do some sort of testing. It would seem that this is impossible to do because at that time not a single dragon flew near King's Landing. However, a solution was found: the skulls of long-dead dragons from the cellars under the castle.
QA engineers are familiar with the issue of shortage or even lack of all the necessary devices for testing. Virtual devices such as emulators and simulators solve this issue quite successfully. Together with Vadim Delendik, CTO at Zebrunner, we find out in which cases emulators and simulators can replace real devices.
A quality product cannot be developed without effective testing. Numerous mobile device manufacturers regularly release new models, and firmware versions are updated. At the end of 2021, there were 14.91 billion mobile devices in the world. By the end of 2022, this figure will grow to almost 16 billion. In 2025, there will already be 18.22 billion devices. This is Statista data and forecasts. Obviously, buying and maintaining up to date all possible models of devices is very expensive and takes a lot of time. Therefore, many teams turn to emulators and simulators.
Emulators and simulators are virtual machines for testing different features at different stages of product development. Virtual devices are easily available and generally free.
How is an emulator different from a simulator
Vadim Delendik says, “Emulators emulate and simulators simulate :). If we test applications non sensitive to hardware we can use both approaches. Anyway, this is virtualization”.
However, if you go into details, you can find some differences.
The emulator imitates a real device work, including its functions and hardware. You test battery performance, random access memory (RAM), camera and keyboard, processor, messaging, security-related functions, and web-browsing. This machine reproduces almost all the functions of applications running on Android and iOS. The emulator window seems like a real smartphone on a PC.
The simulator copies a real device behaviour and configuration. Nevertheless, a simulator doesn’t imitate a device’s hardware. You can’t test the operation of a microphone or speaker with the simulator, this is possible only with the emulator or a real device.
Use them when you want to test something properly, but don’t have money for buying real devices.
“Virtual devices are cheaper and much more scalable. Sometimes you don’t need real devices at all because of the specifics of the project. Emulators and simulators are especially useful when we need to perform a lot of device-specific tests. We can use an emulator or a simulator to speed up the testing phase”, Vadim Delendik says.
“When you have a simple application, with no IoT integrations, then emulators and simulators are the best solutions for you”, the CTO of Zebrunner adds.
QA teams perform different tests during the product development phase. Some of them can be carried out on emulators and simulators without compromising the quality of the product.
Sanity testing. You check the operation of a specific function according to the requirements stated in the specification. This testing type is used to determine the health of a particular part of the application after changes have been made to it or the environment.
Smoke testing. The minimum set of tests to detect obvious defects at an early stage of development.
Installation testing. You check the process of installing and uninstalling the application. These tests evaluate the process for speed and user-friendliness, as well as to make sure that updates do not contain errors.
Performance testing. You test application performance under certain load conditions such as application speed test, poor network coverage, and low memory.
Compatibility testing. Checking the compatibility of a mobile application with different operating systems and their versions.
Load testing. Checks the operation of the application in a situation where it is used by many users at the same time. This test checks how a large load affects the application, its speed, and its performance.
Localization testing. You check the quality of the app's adaptation to a specific geographic region. A test may include verification of a specific language, compliance with local laws, or time zone requirements.
Real devices give you an idea of how the application works in real conditions. This is something that the most advanced emulators and simulators cannot provide.
In what cases is there no replacement for real devices?
“You must test on real devices when you need to check real integrations. For example, the iOS simulator does not have Bluetooth at all, but you need to check the fitness watch. So this case is for real device testing. Also sometimes clients ask to test their app exclusively on real devices”, Vadim noted.
If you look in more detail, here is a list of the benefits that testing on real devices provides:
- You can test screen resolution and brightness in various lighting scenarios.
- The performance of a real device is higher than the performance of a virtual one. Thus, you understand the real speed of the application and its real capabilities.
- It's easier for you to do compatibility testing.
- It's easier to spot real bugs in the application.
- You can test push notifications, geolocation, and built-in device sensors such as orientation and Wi-Fi.
- Testing for battery drain issues that occur in your application.
- Testing for interruption by incoming SMS, notifications from other applications, and incoming calls.
Everything looks great, but testing on real devices is rather expensive. In addition to the cost of buying hundreds of devices, you need to consider the cost of their maintenance. You also need to consider the difficulty of buying devices that are popular in other countries but are not available for sale in the country where your QA team is located. What is the solution?
“The simulators and emulators are a compromise. The ideal solution is a mix. We run 75% of regression tests on simulators and emulators and perform 25% of test cases on real devices. As a result, we get savings, and reduce the cost of devices or the usage of device farms, while there is no threat to product quality”, Vadim Delendik advises.
A mobile device farm provides devices through the cloud. So you only pay for the time you use them. This is easier than buying devices by yourself and creating your own device farm, which also needs constant support.
There are many offers on the market for device farms, let's see what are the advantages of Zebrunner.
“Our customers do not need to spend money on support and maintenance, which are very expensive now. Moreover, our farm is good because we integrate it for both automated and manual testing. This is a big advantage as it allows us to utilize private devices almost 24/7. One more advantage is that you can assemble an on-premise farm yourself.
Zebrunner positions itself as a proprietary farm with the ability to integrate IoT. We have custom devices, and all of them are 100% private.
5G technology is becoming more and more widespread. Various devices like smart kettles, snickers, bags, etc. are synchronized and communicate with each other. In this case, it is difficult to do without the device farms service. There are many more kettle manufacturers than telephones! This means that more devices need to be tested. Device farms allow you to check this connection and the operation of devices according to various parameters.
We can integrate any gadgets-devices-things from the IoT category (sneakers, T-shirts, watches, pedometers, and whatever). Zebrunner private device farm simplifies IoT testing significantly”, CTO of Zebrunner says.
