Selenium vs Cypress vs Playwright: comprehensive comparison

While Selenium has been in the market for quite some time and offers many benefits, Cypress is a newer tool that is even more efficient in some aspects! Playwright is primarily used for end-to-end testing and also has its own advantages. Today, Zebrunner experts compare Selenium, Cypress, and Playwright to uncover all their pros and cons.

The history of the Selenium project began in 2004. 

Selenium is an open-source testing framework used for automated testing of web applications across various platforms and browsers. With Selenium, you can automate applications across major browsers using various programming languages like Java, JavaScript, C#, Ruby, PHP, and Python.

Testers use Selenium's framework to run test cases concurrently across multiple browser instances, speeding up the testing process. Thanks to parallel execution, the testing phase in your CI/CD pipeline can conclude more quickly, thereby accelerating the overall development process.

Selenium consists of several different tools. Let's explore the tool suite in detail.

Selenium WebDriver. This library enables API calls from code, executing commands on the chosen browser. Testers use Selenium locators to find elements, creating test cases and interacting with them using WebDriver APIs. Direct browser interaction makes WebDriver faster than Selenium Remote Control (RC), with separate drivers for different browsers interpreting your script.

Selenium IDE. This tool allows users to record, edit, and replay test steps. As an add-on for Firefox, Chrome, and Edge, the IDE simplifies the creation and execution of automated tests without any coding requirements. With Selenium IDE, one can record browser actions such as clicking elements and filling forms, subsequently replaying them to assess web application functionality. Additionally, IDE facilitates script editing, assertion inclusion to verify application behavior, and script export to formats like HTML, Java, Python, and C#.

Selenium Grid. This tool lets tests run on multiple machines and browsers simultaneously. Selenium Grid is essential for large test suites needing quick execution by running tests on different machines, operating systems, and browsers. It consists of a central hub and multiple nodes, distributing test execution across various environments. Users configure a hub to manage test execution on connected nodes, each representing different operating systems, browsers, and versions. Nodes execute tests as directed by the hub, sending back results for aggregation. 

Overall, Selenium Grid’s operational mechanism involves:

1. Setting up a hub as the main communication point for test-running nodes.
2. Configuring nodes, which are machines running tests on various browsers or operating systems, to connect to the hub.
3. Writing tests and using client libraries like Selenium WebDriver to send commands to the hub, which then sends them to the appropriate nodes for execution.
4. Using Selenium Grid significantly improves test execution efficiency by allowing parallel execution across multiple machines and browsers.

Founded in 2014, Cypress.io revolutionized web testing by integrating it directly into the browser for real-time, agile testing. Initially created as an internal tool, it grew into a widely adopted platform, used by companies like Slack, Netflix, and Disney. Cypress offers a free, open-source app and a subscription-based Cypress Cloud, providing comprehensive testing tools and actionable insights.

Cypress is a front-end testing tool for modern web applications, using JavaScript and specializing in end-to-end testing with the Mocha framework. In recent years, Cypress has become a robust automation solution and a viable alternative to Selenium.

One of Cypress's notable features is its streamlined setup process. Most necessary components for crafting test cases are bundled within Cypress, eliminating the need for multiple dependencies. Operating within the browser, Cypress executes test suites swiftly, outpacing frameworks like Selenium. The Cypress package incorporates an integrated Chai assertion library that supports the Mocha framework, making it easy to implement test cases in a Behavior-Driven Development style.

Cypress automation tool addresses several key pain points encountered by developers and QA engineers when testing modern applications:

Effortless installation. Cypress can be installed via npm or through its desktop application, eliminating the need for users to install additional libraries, dependencies, or drivers.

Real-time reloads. With Cypress, test cases don't require manual reloading. Any modifications lead to the automatic launch and execution of the test.

Accelerated test case development. Cypress expedites test case creation; users can commence writing tests within minutes. By executing tests directly within the browser, Cypress provides results closely resembling user experiences.

Automatic waiting. Cypress UI automation eliminates the need for explicit wait times or sleep commands in tests. It automatically waits for commands and assertions to complete before proceeding to the next step.

Comprehensive reporting. Cypress automatically captures video footage of the entire test suite and generates screenshots upon test failures when running headlessly. This feature proves invaluable in diagnosing test failures efficiently.

Informative dashboard. Leveraging Cypress Cloud, users can execute test cases within CI/CD providers and record test results, providing an instructive overview of test performance.

Playwright is an open-source automation E2E testing framework for web applications, developed by Microsoft and launched on January 31, 2020. It has already gained popularity among programmers and web developers.

Playwright is suitable for cross-browser and cross-platform testing. It supports all modern rendering engines including Chromium, WebKit, and Firefox. With Playwright, you can test on Windows, Linux, and macOS, locally or on CI, headless or headed. Playwright API is available in TypeScript, JavaScript, Python, .NET, and Java.

Playwright ensures reliability by waiting for actionable elements before executing actions. Moreover, features like test retries, execution trace capture, as well as video and screenshot capabilities are available to minimize flakiness.

Each test executed with Playwright operates within its own unique browser context, akin to a new browser profile. This ensures complete test isolation without any additional overhead.

Selenium vs Cypress vs Playwright comparison

‍

Setting up test automation tools can vary significantly in complexity. Each tool — Selenium, Cypress, and Playwright — has its own setup requirements and features that cater to different testing needs. We'll explore the steps needed to get each tool up and running, helping you decide which one suits your project best.

Selenium

‍Installing and configuring Selenium can be challenging for an inexperienced tester. 

Setting up Selenium typically involves a series of steps, especially when configuring a local Selenium Grid. These steps are universally applicable across various programming languages supported by Selenium, including Python. It's important to note that the installation of Selenium Grid and Server isn't necessary for cloud-based Selenium Grids like Zebrunner Selenium Grid. After registering, Zebrunner gives you a grid URL and credentials. Set these in your tests to start testing in the cloud right away.

Cypress

Installing Cypress is a breeze, thanks to its user-friendly setup process. One of the standout features of Cypress is its organized folder structure, which streamlines test case development. By adhering to this predefined structure, you can effortlessly build the framework for your project.

To initiate the Cypress setup, simply follow these steps:

Step #1. Pre-requisites: Ensure that Node is already installed on your system.

Step #2. Generate package.json: Start by generating a package.json file using the command npm init.

Step #3. Install Cypress: Execute the command npm install cypress --save-dev to install the latest version of Cypress. As of the time of writing, the latest version is 12.2.0.

Step #4. Verify Installation: Once Cypress is installed, your package.json file should reflect the changes.

With the setup complete, launch Cypress to access all default .spec files and begin testing promptly.

Zebrunner CyServer offers fast scaling, allowing you to run an unlimited number of threads and execute thousands in parallel as needed. Additionally, you can execute tests with Zebrunner CyServer without compromising data security. If you prefer not to run tests in the cloud, you can opt for a plan to use the product within your infrastructure. Furthermore, the easily connected reporting platform lets you manage all testing activities in one place. 

Playwright

When it comes to installation and configuration, Playwright stands out for its ease and simplicity. Here's a rundown of how to get started with Playwright:

Firstly, ensure you have NodeJS installed on your system.

Next, equip yourself with a code editor. For the purposes of this guide, we'll be using Visual Studio Code.

There are two main methods for installing Playwright:

1. Using the VS Code Extension:

1.1. Begin by creating a folder, let's call it "Install_Playwright".

1.2. Open this folder in Visual Studio Code.

1.3. Search for the Playwright extension in the VS Code marketplace and install it.

1.4. Once installed, press command+shift+P (for Mac users) and click on "OK" to confirm.

1.5. The Playwright installation process will commence automatically.

2. Using the Init Command:

2.1. Start by creating a new folder, perhaps named "Playwright_New_Script".

2.2. Open this folder in Visual Studio Code.

2.3. From the root directory of your project, run the command npm init playwright@latest.

2.4. This command initiates the installation of Playwright. (As of the time of writing, the latest version is 1.29).

2.5. During installation, you'll be prompted to choose between TypeScript or JavaScript for your project.

2.6. After a successful installation, you'll find a default .spec file ready for use.

The Test Runner, as its name implies, serves as a crucial tool for executing tests and exporting their results. Acting as a library, it navigates through the source code directory, identifying and executing the test files to verify the presence of bugs and errors. Moreover, the Test Runner ensures the proper execution of these files, providing feedback on whether they meet the benchmark standards as programmed. Capable of running functional tests, the Test Runner generates results tailored to your specific setup.

Here, it is important to distinguish between software for executing automated tests, a test run orchestrator, and an environment for execution.

Selenium

For Selenium test execution, we can use:

• Selenium WebDriver. The core of Selenium, WebDriver enables users to automate various web browser actions like clicking elements, filling forms, and navigating pages.
• TestNG. A popular testing framework often paired with Selenium, TestNG enables the creation of comprehensive test cases, definition of test suites, and support for parallel test execution.
• JUnit. Another well-known testing framework commonly used alongside Selenium, JUnit offers similar capabilities to TestNG, allowing for test case creation, test suite definition, and parallel test execution.

As run orchestrators, we can apply:

• Jenkins. As a continuous integration tool, Jenkins plays a pivotal role in automating the build, test, and deployment processes. It seamlessly integrates with Selenium, facilitating parallel test execution and ensuring efficient software delivery pipelines.
• Maven. Serving as a robust build automation tool, Maven is invaluable for managing Selenium tests. It enables users to define dependencies, automate build and test processes, and streamline deployment procedures, thereby enhancing overall project management efficiency.

Finally, concerning the environment for execution, we can apply a Selenium grid or a local Selenium server. Selenium Grid facilitates the execution of Selenium scripts on remote machines or cloud-based platforms. It is particularly beneficial for parallel test execution across multiple machines and diverse browser and operating system configurations.

Cypress

The Cypress test runner provides a comprehensive graphical user interface for visualizing and debugging tests effortlessly. It features a dashboard showcasing test lists and a dedicated test runner window exhibiting commands and outcomes for each test. Additionally, a browser window offers a real-time view of the application under test.

In addition, the Cypress runner includes debugging tools like a console for logging messages and interacting with the app in real-time, and a network tab for monitoring HTTP requests and responses.

This runner also provides real-time insights into application behavior, helping QA professionals and developers quickly identify and resolve issues. Its interactive interface allows users to easily engage with the application and validate test results.

In addition, for Cypress test execution you can apply Mocha. Cypress uses Mocha as its default test framework. Mocha allows for the creation of test suites and test cases, supporting various testing styles (BDD, TDD) and offering robust reporting capabilities. 

Cypress also integrates Chai as its assertion library. Chai provides a natural language style for writing assertions, making the tests more readable and expressive.

Run orchestrators:

• GitHub Actions. GitHub Actions can be used to automate the execution of Cypress tests within a CI/CD pipeline. It integrates well with GitHub repositories, enabling the triggering of tests on each push or pull request.
• CircleCI. CircleCI provides robust CI/CD capabilities and integrates seamlessly with Cypress. It supports parallel test execution, which speeds up the testing process by distributing tests across multiple machines.

You also can apply Jenkins as a run orchestrator for Cypress test runs.

Environments for execution:

Cypress is designed to run tests directly on the local machine. It supports cross-browser testing, including Chrome, Firefox, and Edge, allowing for comprehensive local test execution.

The Cypress Dashboard provides a cloud-based platform for managing test results, debugging failures, and optimizing test performance. It offers features like test analytics, parallelization, and automatic load balancing.

In addition, Cypress tests can be executed in Docker containers, providing a consistent and isolated environment for running tests. Docker ensures that the test environment is reproducible across different machines and CI/CD pipelines. Additional environments for Cypress tests:

• Cypress cloud. The enterprise-ready companion to the open-source Cypress app. It allows users to view and debug past test results from the CI environment, analyze and diagnose test health, reduce test time and save money by orchestrating future runs across multiple machines.
• Zebrunner CyServer. This tool provides fast results for Cypress tests through parallel execution. It supports rapid scaling, enabling the execution of thousands of threads in parallel, and offers an easily connected reporting platform.

Playwright

For Playwright test execution we can apply:

• Playwright test runner. Playwright comes with its own built-in test runner designed specifically to run Playwright scripts. It supports features like test retries, parallel execution, and detailed reporting.
• Mocha also can be paired with Playwright for writing and running tests. Mocha offers a flexible and simple interface for creating test cases and suites.
• Jest. Another widely used JavaScript testing framework, Jest integrates well with Playwright, providing a comprehensive solution for writing, running, and organizing tests, with built-in support for assertions and mocking.

Run orchestrators:

• GitHub Actions. GitHub Actions can be used to automate the execution of Playwright tests as part of a CI/CD pipeline. It provides seamless integration with GitHub repositories, making it easy to trigger tests on every push or pull request.
• Azure Pipelines. Azure Pipelines supports Playwright test execution and allows for robust CI/CD workflows. It can orchestrate the testing process, ensuring tests are run on different environments and configurations.
• Jenkins. It integrates well with Playwright to automate the build, test, and deployment processes, facilitating efficient parallel test execution.

Environment for execution:

• Local execution. Playwright supports running tests on local machines, offering extensive cross-browser testing capabilities. It can launch and interact with multiple browser contexts, including Chromium, Firefox, and WebKit.
• Playwright Test Runner (Parallel Execution). The Playwright test runner itself supports parallel execution, enabling tests to run concurrently across multiple browser instances on a local machine or remote servers.
• Cloud-Based platforms. Playwright integrates with various cloud-based testing platforms like BrowserStack and Sauce Labs. These platforms allow tests to be executed on remote browsers hosted in the cloud, providing access to a wider range of browser and OS configurations.

Record and Playback feature

All three tools include Record & Playback features.

With Selenium IDE, you can effortlessly record and playback Selenium code. Installing the plugin in your browser is the only setup required before using Selenium IDE. Unlike Selenium WebDriver and RC, Selenium IDE doesn't necessitate programming logic for developing test scripts. Instead, you can simply record your browser interactions to generate test cases. These test scenarios can then be rerun using the playback option.

Cypress Studio facilitates test case creation within Cypress by capturing interactions with the application under test. It offers a visual approach, allowing users to record interactions directly against the application.

Using Cypress Studio, you can visually create tests by interacting with the Document Object Model. It supports Cypress commands such as .type(), .click(), .check(), .uncheck(), and .select(), which automatically generate corresponding test code.

In Playwright, you'll craft code to orchestrate user interactions. Utilize CodeGen to automatically generate actions executed within the browser. Playwright streamlines this process by automatically generating code for each user action. 

Language and browser support stands out as a key distinguishing factor when comparing Selenium, Cypress, and Playwright. 

Selenium, a suite of tools for web application testing, also provides extensive language support, featuring Java, Python, C#, Ruby, and JavaScript. With Selenium, you can write test scripts in any of these languages and execute them across various web browsers such as Google Chrome, Mozilla Firefox, Internet Explorer, Microsoft Edge, and Safari. Selenium further supports headless browsers, allowing test execution without launching a visible browser window. To utilize Selenium with a specific programming language and browser, installation of the corresponding language bindings and WebDriver is required, enabling interaction with the browser for actions like page navigation, element interaction, and text input.

On the other hand, Cypress is a JavaScript-centric testing tool tailored for web applications primarily built with HTML, CSS, and JavaScript. It seamlessly integrates with modern browsers including Google Chrome, Mozilla Firefox, Microsoft Edge, and Apple Safari.

Playwright boasts broad language support, accommodating Python, Java, JavaScript, and .Net among others. It offers comprehensive browser compatibility, including Google Chrome, Microsoft Edge (with Chromium), Safari (with WebKit), and Firefox.

Parallel Execution

Parallel execution is an essential capability in automating web browsers, enabling the simultaneous execution of multiple test cases and thereby minimizing overall execution time. Selenium, Cypress, and Playwright stand out as prominent options for parallel execution, widely embraced by developers. 

A flaky test produces unpredictable results, sometimes passing and other times failing, even when the tested code remains unchanged. This inconsistency presents challenges for software teams, unlike reliable tests, which consistently produce the same results.

Selenium

Flakiness in Selenium tests can arise from issues within the test code or the test infrastructure. 

To mitigate flakiness in Selenium tests, consider the following strategies:

• Utilize stable selectors. Opt for stable selectors like class names or IDs instead of dynamic ones like XPath to locate elements. Dynamic selectors are susceptible to changes in HTML structure, leading to test flakiness.
• Implement explicit waits. Replace implicit waits with explicit ones to allow the page to fully load and stabilize before interacting with elements. This prevents test failures caused by elements not being immediately accessible.
• Incorporate retry logic. Introduce retry mechanisms in tests to retry failed assertions a defined number of times before discontinuing. This helps alleviate flakiness induced by transient network or server issues.
• Ensure a stable testing environment. Verify that tests are executed in a stable environment with consistent network and server conditions to minimize external factors contributing to flakiness.
• Debug failing tests. Invest time in debugging and comprehending the underlying causes of test failures. Identifying and addressing root issues can effectively mitigate flakiness in tests.
• Leverage test frameworks. Employ robust test frameworks like JUnit or TestNG to organize tests and handle failures efficiently. These frameworks offer built-in support for managing test failures, simplifying debugging processes.
• Rely on a dependable CI server. Use a reliable continuous integration (CI) server such as Jenkins or Azure DevOps to automate test execution and monitor test results systematically.

Cypress

Cypress excels in handling flakiness, offering various methods to address this challenge effectively.

Here are some recommended approaches for managing flakiness in Cypress:

• Assertive commands. Employ assertions like `.contains('element')`, `.should('be.visible')`, and `.should('exist')` before interacting with elements. These assertions enhance test reliability by verifying the presence and visibility of elements before proceeding with actions.
• Test retries. Utilize test retries to mitigate flakiness. You can configure test retries at the global configuration, individual test, or test suite level to automatically rerun failed tests and increase their chances of success.
• Timeouts instead of waits. Opt for timeouts instead of traditional waits to handle scenarios where elements require more time to load. Setting timeouts at the field level allows Cypress to proceed with the next command if the element loads before the specified timeout, thereby improving test efficiency.
• Cypress command cy.intercept(). Leverage `cy.intercept()` to wait for XHR requests to complete execution. By intercepting specific network calls and waiting for their completion, you can prevent test failures caused by unfinished requests.

Playwright

There are various strategies to address flakiness in Playwright, and here are some effective approaches:

Auto-wait Feature. Utilize the auto-wait feature to ensure that actions such as clicking on elements or entering data in text fields are performed only after the element is fully loaded.

Use ‘waitFor’ methods. Employ methods like waitForLoadState, waitForNavigation, and waitForSelector to mitigate flakiness. These methods ensure that the page is fully loaded before interacting with elements.

Retries. Leverage the retries feature to handle scenarios where required elements may not be present during the initial page load, leading to test failures. By configuring retries in the Playwright.config.js file, the system automatically retries failed test cases, increasing the likelihood of test case success.

Screenshots

Screenshots are pivotal in diagnosing regularly failing test cases, offering insights into the steps where failures occur and aiding in the refinement of test cases to reduce flakiness.

In Selenium WebDriver, capturing screenshots is invaluable for bug investigation purposes. To capture a screenshot during execution in Selenium, you can utilize the TakesScreenshot method. This method instructs the WebDriver to capture the screenshot, providing crucial visual documentation for debugging and issue resolution.

Cypress comes equipped with a built-in feature for capturing screenshots. Whether running test cases via "cypress open" or "cypress run", you can manually take screenshots using cy.screenshot(). To disable the automatic capture of screenshots when a test fails, you can set screenshotOnRunFailure to false either within the Cypress configuration or by specifying screenshotOnRunFailure to false in the Cypress.Screenshot.defaults(). This provides flexibility in managing screenshot capture based on specific testing requirements.

Playwright provides three methods for capturing screenshots: full-page screenshots, capture into a buffer, and element screenshots.

These options enable comprehensive visual documentation of test executions, facilitating effective debugging and refinement of test scenarios.

Video

Video features also play an important role in facilitating bug identification and simplifying debugging processes, particularly in cases where test cases fail. By visually capturing the sequence of test steps, videos enable easy tracking of the precise step where a failure occurs.

Selenium WebDriver lacks a native feature for recording test case videos during execution. However, there are extensions available for video capture. These extensions enable the recording of test case execution steps, which is beneficial for pinpointing the exact cause of test step failures.

In contrast, Cypress automatically records videos of every .spec file. When executing the "cypress run" command, a video of each .spec file is recorded and saved in the cypress/videos folder. The video recording functionality can be disabled altogether by setting the video flag to false in the configuration file.

In Playwright, video recording of test cases is managed through the "video" option in the configuration file. By default, video recording is disabled.

When configuring the "video" option, the following parameters can be specified:

• "off": Disables video recording.
• "on": Records video for each test.
• "retain-on-failure": Records video for each test, but removes videos from successful test runs.
• "on-first-retry": Records video only when retrying a test for the first time.

These options provide flexibility in managing video recording based on specific testing requirements and preferences.

Iframes

Iframes are HTML documents embedded within other HTML documents, often used to add external content like YouTube videos and photos to web pages. If the iframe is from the same domain as the original page, you can automate it using browser automation actions.

Selenium offers a straightforward approach to working with iframes. Users need to navigate to the iframe window to perform desired actions and subsequently return to the main application page for further interaction. By default, Selenium interacts with the parent browser driver, necessitating a shift in driver focus from the main window to the targeted frame element.

There exist several methods to switch between frames in Selenium:

• switchTo().frame(id): Specifying the frame's id or name as input to the switchTo() method.
• switchTo().frame(i): Indicating the frame index as an argument, with the index starting from zero.
• switchTo().frame(webelement n): Passing the web element of the frame as an argument.
• switchTo().defaultContent(): Reverting focus from the frame to the main page.

In Cypress, direct handling of iframes is not supported. Instead, users must install a plugin and leverage it to execute actions on elements within the iframe. Installation of the plugin is followed by utilizing custom commands to interact with iframe elements effectively.

In Playwright, testing iframes involves utilizing the FrameLocator method, enabling the retrieval and element location within the iframe. This method encapsulates the logic necessary for accessing the iframe and its internal elements efficiently.

Conclusion

Choosing the right test automation framework – be it Selenium, Cypress, or Playwright – depends on your specific needs and the context of your project.

Selenium remains a versatile and robust tool with extensive browser and language support, making it ideal for projects requiring cross-browser testing and integration with multiple programming environments. Its long history and widespread adoption ensure a rich ecosystem of resources and community support.

Cypress, on the other hand, excels in ease of setup, speed, and user experience, particularly for front-end testing of modern web applications. Its real-time reloads, automatic waiting, and built-in debugging tools make it a preferred choice for developers and QA engineers looking for a seamless and efficient testing workflow.

Playwright stands out for its comprehensive cross-browser and cross-platform capabilities, with strong support for modern web technologies. Its robust auto-wait features and ability to handle complex scenarios like iframe testing and network interactions make it a powerful tool for end-to-end testing.

Ultimately, your choice should be guided by the specific requirements of your testing strategy, the technical stack of your project, and the unique strengths of each framework. Whether you prioritize broad compatibility with Selenium, streamlined efficiency with Cypress, or cutting-edge capabilities with Playwright, each of these frameworks offers valuable features to enhance your test automation efforts.