AI/ML: why is it a must for your test automation?

The first association about artificial intelligence or machine learning can be an intelligent supercomputer or even a robot, why not. However, so far in practice, everything is somewhat different, somewhat simpler and more practical. AI and ML are key trends in many areas. Software test automation is no exception. In this article, we look into what artificial intelligence and machine learning are in testing, how these algorithms work in this case, what advantages this technology has, and how to apply AI and ML in your test automation.

Artificial intelligence (AI) and machine learning (ML) are often used interchangeably or as synonyms. This is the wrong approach. Let's first understand the definitions of artificial intelligence and machine learning.

Artificial intelligence is a technology that allows computers to learn from experience, adapt to given parameters and perform tasks that were previously only possible for humans. The first research in the field of AI was carried out in the 50s of the last century, at the same time the term “artificial intelligence” arose.

AI automates repetitive learning and searching processes through the use of data. AI is not a stand-alone application, but a technology that is being integrated into existing products. For example, the technology of intelligent assistants like Google assistant, Siri, Alexa were added to existing products: Google search engine, Apple devices, Amazon platform. AI in general can improve various existing technologies and products.

Machine learning is a branch of artificial intelligence as a technology. ML is a field of knowledge that explores algorithms. The algorithms, in turn, are trained on the data to find patterns. Machine learning uses neural network methods and statistics to reveal hidden useful information in data.

Simply put, machine learning is a pattern recognition technology. It works with those templates that are determined by algorithms, and predicts future trends. ML can process huge amounts of complex information, find patterns, classify data, create predictions, and warn of possible risks or discrepancies.

The basic principle of machine learning is that machines take data and learn based on them. Machine learning systems are successfully used in tasks such as the face, speech, object recognition, translation, etc. 

As you can see, artificial intelligence and machine learning are closely connected. In test automation, we are used to considering not so much their differences as interactions.

There are several basic machine learning algorithms for test automation. Let's see what they are.

Supervised learning. The machine receives training datasets with correct answers. The set includes data with information about the number of commits, tests and their results, code coverage, releases, the coverage of features, and fixed bugs. Also, this machine learning algorithm is often used to measure potential risks.

Unsupervised learning. The machine makes decisions by working with demarcated inputs. This algorithm helps to identify the possibility of an error, as well as to understand if the data in the same cluster are similar to each other, and simplify the data.

‍Reinforced learning. In this case, quality assurance engineers train the neural network on the reward and punishment mechanism. In case of an error, the work of the function is corrected so that the next time it chooses the true option. In this case, the more you teach the neural network, the fewer mistakes it makes.

Much has been written about the benefits of AI/ML. However, distinguishing real functionality from marketing exaggerations is not so easy. It's high time to consider the real benefits of using AI/ML technology in automated testing.

Simplified test script creation

AI/ML is able to create complex test scenarios and this makes the job of QA engineers easier. Moreover, this technology can convert manual test cases into automated ones.

AI also creates reusable components to run data-driven tests, and integrate CI/CD systems with both private and public grids. As a result, testing becomes more reliable and requires less effort to maintain, as test coverage grows.

Reduced test automation cost

High-quality test automation tools aim to reduce the time to run and process tests, fix bugs, and provide analytics and reports. AI/ML technology helps to prevent probable errors, automatically determines the reasons of failures, and makes it possible to identify their root causes. The reduction in costs, in this case, comes at the expense of saving the time that QA and development teams would spend to independently find the reasons for test failures.

Accelerated product release

Test automation with built-in AI/ML technology works in parallel with the QA team. AI/ML accelerates the execution of automated tests, and optimizes the entire testing process. In turn, QA engineers can focus on those types of testing that cannot be automated.

Speeded up recognition of failure reasons

This technology helps classify completed tests and speeds up the verification process. AI/ML is used for smart analysis, which, based on the results of the tests, indicates the most probable cause of failure. The model can be trained and thus improves its accuracy to the maximum possible. You can also see the most common test failure reasons and take steps to ensure that similar test issues do not occur again in the future.

Do not think that AI/ML technology takes testing teams into fantasy worlds. There are also issues here.

Lack of data leads to false results. AI/ML technology works based on the data it has. If for some reason there is not enough data, all the work on the implementation of artificial intelligence will go down the drain.

Changes in the application that are difficult to adapt. During development, the product is constantly evolving, and this is a serious challenge for AI/ML. For this technology to be useful, it must adapt to the changes that have been made. In practice, this is difficult to ensure.

The accuracy of the algorithm needs to be checked. To understand how accurately AI/ML works, involve experts in this field. They will measure the accuracy of the technology and suggest what needs to be done to make the technology work as efficiently as possible. Without overcoming this challenge, the expediency of using AI/ML becomes low.

We figured out the theory, it's time for practice. Vadim Delendik, CTO at Zebrunner, talks about the function of AI/ML technology at the Zebrunner Testing Platform and what test automation problems it can solve.

– What is the function of Zebrunner AI/ML?

– Here, AI/ML is a collective intelligence that is trained by a group of competent specialists through a voting mechanism. This allows you to train the model to assign categories to all failures on its own. This approach helps to find a bug as quickly as possible, as well as immediately understand who is responsible for each of the detected failures. Moreover, the AI/ML model streamlines work with automated tests.

The AI/ML technology in Zebrunner is focused on finding potential regression bugs, when something stopped working after adding a new feature.

Our AI/ML model learns easily and very quickly. As a result, the work of automation engineers is simplified, they do not need to spend time trying to understand what is the cause of specific failures and what to fix in the first place. AI/ML allows you to immediately determine the reason of the failure and quickly fix it.

Zebrunner does not provide a pre-trained model. The project team itself trains AI/ML. Thus, the model works solely based on the customer’s data and requirements.

Another feature is that Zebrunner AI/ML technology shows potential bugs in the first position. The technology provides results based on qualitative indicators, not quantitative ones.

– How long does it take to train an AI/ML algorithm?

– The number of results plays a key role here, not time. Each project has its own specifics. Our model is reconfigured and retrained after every 7-10 votes. The more voting happens, the more accurately our AI/ML works. If the score goes to thousands of votes, the accuracy becomes almost perfect.

As for voting, Zebrunner has a feature where you can limit access to categorization and provide it only to experienced people.

– What is the use of artificial intelligence technology and machine learning in automation testing?

– The life-cycle of a product involves its changes. The problems that are relevant at the first stages, are replaced by other challenges at the next stages. A person who is used to working at one stage of product development cannot always figure out what steps to take when the project has entered a new phase. AI/ML comes to the rescue in this case. A model based on statistically significant data gives a prediction: what can become a potential problem, and what, on the contrary, should not be feared. More or less adequate AI/ML predictions can be generated at every stage of a product's life. At the same time, the technology easily adapts to the needs and requirements of the team.

As you can see, AI/ML technology can improve the work of QA teams, make the use of test automation tools more efficient, and speed up the release. However, artificial intelligence may not be essential in all test automation scenarios. We have prepared top areas where AI/ML will be useful.

Visual testing to verify UI

Visual testing involves testing the UI from the user's point of view. This should not be confused with the usual functionality check through the UI.

The purpose of visual testing is to make sure that the user interface matches the design, and that each UI element has the correct color, shape, position, and size. This testing also checks if the UI elements overlap each other.

Visual inspection tools based on machine learning find differences in the current UI and approved design that would be hard for human testers to find.

Classification of executed tests

Test automation tools allow you to run hundreds or even thousands of threads at the same time. But after a short time, you get the results and, naturally, a list of failed tests. Depending on their number, it will take hours, if not days, to find the reasons for all failures. However, you still need to fix all the problems and restart the tests again to make sure that everything works as it should. Of course, such a situation can delay the release time, which threatens losses.

AI/ML technology comes in handy here. It categorizes completed tests, automatically assigns the most likely causes to bugs, and provides statistics on the most common failures and their reasons.

Test coverage analysis

Even after a small change in the application, you need to run tests to check that everything works as expected. Running the entire test suite is sometimes impractical, but that is often what needs to be done. AI/ML in this case allows you to determine the necessary tests and thus save time.

Moreover, artificial intelligence technology lets you analyze the current test coverage and mark areas with low coverage, as well as areas at risk.

Automated test writing by spidering

Machine learning is often used to write automated tests. In the first stage, the technology scans the product, collects data on their functionality, downloads the HTML codes of all pages, and measures their loading time. As a result, a set of data is formed. Based on it, you can train the algorithm on how the application should work. In the next stage, the AI/ML technology will compare the current state of the application with the templates it has. Any deviations found will be flagged as potential problems.

Keep in mind that control by a person with knowledge of the subject area is necessary here. But even taking this into account, time is saved significantly.

So, a brief summary of what we have. Artificial intelligence and machine learning are no longer new words in test automation, but a kind of necessity. AI/ML can speed up release times, provide better analytics, and help you find the root causes of failures and fix them. AI and machine learning are used in test automation, as well as in test writing, test coverage assessment, and UI testing. The technology has its drawbacks, but a competent QA team is capable of coping with them.

For sure, the test automation platform is important here. The quality of future results depends on how well the AI/ML model is implemented there, and how easily it learns and adapts to the customer’s requirements. You can try out how AI/ML works by requesting a trial version of Zebrunner Testing Platform.