Understanding the cost of Selenium: is Selenium Grid worth the investment?

Due to the modern concept of accelerated testing, whereby tests are performed early, frequently, and continuously, and the quality of the product is evaluated at every stage of the SDLC, it becomes very difficult to do without Selenium Grid. But what kind of 'beast' is this creature and why are there a number of services that charge money for this, if it seems to be free? Let's figure it out!

Selenium Grid is a powerful testing tool aimed to provide an easy way to run automated tests across multiple physical or virtual machines. This type of testing is called distributed test execution.

Selenium Grid is mainly used by Automation QA engineers for several reasons:

• to execute automated tests in parallel on multiple browsers and their versions as well as operating systems
• to ensure that the app is compatible with numerous browser/OS variants
• to reduce the overall test run time significantly

Selenium grid allows parallel testing, scaling, cross-platform testing, and load balancing, thus making automated testing faster, more efficient, and easy to analyze and compare the results.

For example, if you have a Suite of 100 tests, but you configured Grid to support 6 different machines (virtual machines or separate physical machines) to execute those tests, your test suite will complete (about)  six times faster than it would if you would run your tests sequentially on a single machine. Selenium Grid can save a great deal of time, especially when it comes to large or long-running test suites that perform validation on large amounts of data and can take hours to run. 

According to Agile software development practice when the test feedback must be delivered as soon as possible, Selenium Grid can be a critical point in Continuous Delivery success.

The two major elements of Selenium Grid:

• Hub. This is the central point of the Selenium Grid setup architecture. It manages the network of test machines. Hub accepts requests from the WebDriver client (specifications about the platform and browser) and routes them to Node. There can only be one Hub in a grid.

• Node. Nodes are test machines that run the tests previously loaded to the hub. They can be launched on numerous machines with various platforms and browsers. The node does not have to be running on the same platform as the hub. There can be 1 or multiple nodes in the grid.

For example, you would like to execute tests on Windows 11 and Chrome browser with version 102. Hub will look for a node that matches the criteria to run the test on that machine. 

The infrastructure for setting up a Selenium Grid installation for parallel execution consists of the following steps:

1. Download the Selenium Server Jar

2. Configure the Hub

3. Set Up the Nodes

Selenium Grid is an absolutely free tool. It is integrated into the open source distribution of Selenium, and it's quite easy to get the first demo and begin working with it. The Instructions step-by-step on how to get started with Selenium Grid as well as files for downloads are available on official website.

At first glance, everything seems pretty simple and easy to try with Selenium Grid.  Creating your own Selenium Grid is a starting point for many companies. However, even Selenium itself on its official site claims that 'is not a silver bullet'. It solves a subset of common delegation and distribution problems, but will for example not manage your infrastructure, and might not suit your specific needs'. 

Let's try to find out what problems the software teams that tried to set up Selenium Grid on their own faced, and Why a free tool often involves attracting non-free resources and sometimes results in an unsuccessful expensive undertaking.

1. One of the main barriers to using your own testing grid is maintaining (eg configuring servers, maintaining desktops) and scaling the infrastructure in accordance with growing needs. Such efforts tend to be complex and resource-intensive. 

2. Another major problem is the maintenance of the numerous operating systems/browser combinations. With the frequent releases of new browser versions and the further development of Selenium and drivers, a software team must keep Selenium Grid up and running. Even a slight discrepancy between browser/driver/Selenium can result in unpredictable browser functionality − leading to false positive or negative tests.

3. Homegrown solutions lack smart analytics features (logs, videos, screenshots) since they use the out-of-the-box Selenium Grid which is available on https://www.selenium.dev/. Thus, teams usually face a problem that Selenium Grid is not suitable for easy troubleshooting, debugging, and reporting.

4. Cost is a vital factor in the process of choosing a Selenium Grid solution. Although Selenium is free and does not assume any license payments, it requires an investment to use Selenium (salaries of your team resources, data center costs, cost per hardware, etc.).

5. Time. Building your own Selenium Grid normally takes several person-years. Think about whether you have this time if customers are waiting for a quality application now.

6. Expertise. In addition, building and maintaining such a solution typically requires several experienced engineers. In case of a change within the team, it may be challenging to fill these positions promptly. All this can stall the test automation and CI/CD efforts altogether. 

A robust and Scalable Selenium grid infrastructure is essential for efficient continuous testing and DevOps setup. Building and maintaining a mature, enterprise-grade Selenium Grid infrastructure is a challenging task that expects considerable investment (in terms of engineering resources, hardware, software including virtualization, etc). Selenium Grid is known to be difficult to configure, unstable, and complex to deploy and/or manage on a CI pipeline, you will need plenty of time and deep Selenium Grid know-how. The lack of built-in functions for easy debugging and troubleshooting is one more reason to think about Cloud Selenium Grid solutions.

Let's take a look at the main benefits you get with a Cloud Selenium Grid:

1. Easy start. If you would like to execute a huge number of tests and get results right now, Cloud Providers let you start immediately. After the registration, you get the credentials and URL, set it in your tests, and begin testing. Many companies offer to try solutions for free, so you can appreciate the speed and ease of use.

2. Hardware maintenance. With the help of Cloud Selenium Grid solutions, you can make automated testing of web applications independent of physical infrastructure. Cloud solutions with access to a Selenium Grid of thousand of browsers and real devices remove the need to buy, set up, maintain, and update in-house device labs or personal Selenium Grids. 

3. Software maintenance. Cloud Solutions ensure that the OS / browser / driver / Selenium combinations are always compatible with each other. All the browsers are fully functioning and there is no need to worry that they can lead to unreliable tests. 

4. Fast scaling & Reliability. The cloud testing solution is designed to scale. When an enterprise needs to expand, an enterprise cloud solution can be flexibly modified as needed. You can scale infrastructure up and down according to your needs. Selenium Cloud also provides high availability which eliminates the risk of downtime. 

5. High execution speed. Forget that your tests are running for more than 8 hours. There are solutions that enable rapid test execution (1000+ tests in less than an hour). So your team can run tests fast, detect errors earlier, and release with confidence.

6. Smart real-time analytics. Cloud Solutions provides such essential additional features as comprehensive analytics for easy debugging and troubleshooting (screenshots, published logs, interactive video recordings, test history line, etc.) as well as transparent reporting. 

7. Productivity. The process of building an in-house Selenium Grid takes you away from your main job. With Cloud Solutions, companies can focus on their staff activities within their core business. 

8. Cost reduction. Although Cloud solutions require license fees, you are exempt from paying annual salaries of your team resources, annual data center costs, cost per device used for testing activities, etc. Moreover, there are solutions in the market that allows you to pay only for the actual infrastructure time used without a monthly plan commitment. 

To get test feedback and release a quality product as soon as possible is crucial nowadays. The client cannot wait until you can set everything up and check how well it works. Cloud Selenium grid provides a service that can solve your problems in the shortest possible time in a cost-efficient way.