Basic testing vocabulary guide: test cases, runs, scenarios, and more testing words

What's the difference between a test script and a test scenario? Is a testing checklist the same as a testing matrix? To carry out testing efficiently, it's crucial to grasp the fundamental concepts and vocabulary. In this article, the Zebrunner QA team guides you through these basic testing terms, explaining their meanings, similarities, differences, and significance in the testing process. So read on and improve your understanding of the essential terminology!

A test case is one of the most essential testing elements. This is a set of inputs, execution conditions, and expected outcomes developed to verify a particular functionality of the software system under test. This term is generally applied in manual QA but it isn’t a mistake to use it in test automation. A test case is typically written in natural language and includes steps to be executed, input values, and expected results. 

If we want to test the functionality of a website, possible test cases might look like this:

1. Verify that the landing page is accessible and loads correctly in different web browsers, such as Google Chrome, Mozilla Firefox, and Safari.
2. Verify that all the links on the landing page are working and correctly redirecting to the appropriate pages.
3. Verify that the landing page displays the correct information, such as the features of the test case management tool, pricing plans, and customer testimonials.
4. Verify that the landing page is responsive and displays correctly on different screen sizes and resolutions, including desktop, tablet, and mobile devices.
5. Verify that the search functionality on the landing page is working and correctly filtering the test cases based on different search criteria, such as keyword, status, and priority.

A test script is a set of automated instructions or code that a testing tool or framework executes to validate a particular test case. The “test script” term is used in automated QA. Simply put, a test script is a synonym for a test case in test automation. It is typically written in programming language and includes instructions to interact with the software under test. 

The Python script below uses a Selenium package to automatically test Zebrunner.com website on the Google Chrome web browser. This script automates the process of opening a website in Chrome, waits for the page to load, and checks the website for accessibility issues.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument("--start-maximized")

# Set up the driver
driver = webdriver.Chrome(options=chrome_options)

# Open the website
driver.get("https://zebrunner.com")

# Wait for page to load
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.TAG_NAME, "body")))

# Check accessibility using Axe-Core
axe_script = """
   var callback = arguments[arguments.length - 1];
    axe.run(function(err, results) {
        callback(results);
   });
"""
 

A test scenario is one more term that sounds similar to a test script or a test case. However, it has a different meaning. A test scenario is a hypothetical story or situation that describes a particular use case or user interaction with a software system. While a test script is a set of instructions or steps that need to be followed to execute a test case, a test scenario is a high-level description of a test. Here is also the difference between test case and test scenario. A test scenario defines the objective of the test and the conditions under which it will be performed. The test scenario outlines the context or the environment in which a set of test cases will be executed. Test scenarios are typically used in manual testing, where testers manually execute test cases based on the defined scenarios.

Here is an example of the test scenario for verifying the functionality of the Zebrunner.com website:

Title: Verify the functionality of the Zebrunner.com website

Description:

This test scenario focuses on verifying the basic functionality of the Zebrunner.com website. The goal is to ensure that all website functions are working as expected and the user can perform all the necessary actions on the website without any issues.

Preconditions:

The user has access to the internet and a web browser
The user has access to the Zebrunner.com website 

Test Steps:

1. Open the web browser and navigate to the Zebrunner.com website
2. Verify that the website loads correctly without any errors
3. Verify that the website navigation menu is present and functional
4. Click on each menu item and verify that the corresponding page is loaded correctly
5. Verify that the search bar is present and functional

<…> [The test scenario includes as many test steps as you need to verify if the website features work as expected]

Expected Results:

• The website loads correctly without any errors
• All website functions are working as expected
• The website navigation menu is present and functional
• Each menu item loads the corresponding page correctly
• The search bar is present and functional 

<…> [Here are all desired results you want to get during testing]

In software testing, a test run refers to the process of actually executing a set of test cases or test scripts to verify the functionality of an application or system. This term is used when executing test cases to check a specific build or version of the software. A test run usually takes a short amount of time, usually from a few minutes to a few hours at most. This is because a test run is designed to cover only a specific build or version of the software, rather than the entire system.

You can streamline and accelerate your QA workflow using appropriate testing tools. Zebrunner test case management is a vital tool for manual QA engineers. With this software, you can effortlessly create and manage test cases, execute test runs, and ensure requirements traceability. In the screenshot, you can see an example of a test run in Zebrunner test case management.

A test suite is a collection of test cases that are grouped based on a specific feature. Test suites can be automated or manual, and may be categorized according to their purpose (e.g., functional, performance, security), scope, or level of testing (unit, integration, system). A test suite has a longer duration than a test run. It covers a specific feature that can be connected with multiple builds or versions of the software.

A test plan is a formal document that outlines the testing approach and objectives for a specific project or product. It includes the scope of testing, the test environment, the testing methodology, the test schedule, the test deliverables, and the roles and responsibilities of the testing team. A test plan is used to guide the testing process. It also ensures that all testing activities are executed systematically and that all the requirements are met. The test plan serves as a reference document for stakeholders to understand the testing process and the quality of the product.

See what the test plan outline looks like:

1. Introduction

• Purpose and scope of the test plan
• Overview of the system under test (SUT)
• Assumptions and dependencies

2. Test strategy

• Testing objectives
• Testing methods
• Testing levels
• Entry/exit criteria

3. Test environment

• Hardware and software requirements
• Test data requirements
• Test tools and automation frameworks

4. Test deliverables

• List of test cases
• Test execution reports
• Defect reports
• Test completion criteria

5. Test schedule

• Test timeline and milestones
• Resource allocation

6. Risks and contingencies

• Risk assessment
• Risk mitigation plan
• Contingency plan

7. Approval and sign-off

• Approval process
• Sign-off process

8. Appendix

• Glossary of terms
• Acronyms and abbreviations
• References and resources

A test plan is not the same with a test strategy. While a test plan focuses on specific testing activities for the project, a test strategy’s purpose is an overall testing approach for the project. In addition, a test plan provides detailed instructions for executing testing activities for the testing team and stakeholders directly involved in testing. A test strategy, for comparison, is created for a broader audience, including project managers and developers. 

Zebrunner testing platform is a comprehensive solution for both manual and automated QA teams. It enables you to successfully implement your test plan and achieve maximum efficiency. This tool offers a multitude of benefits, such as parallel test execution, testing on various browsers and platforms using simulators, emulators, and real devices, AI/ML-based analysis of test failures, and comprehensive automation reporting. With this testing platform, you can unlock a wide range of opportunities and streamline your testing process.

A testing checklist is a document that contains a list of tasks or steps that must be completed during the testing phase of a software development project. It serves as a reference guide for the testing team to ensure that all aspects of the software are thoroughly tested and any potential issues are identified and addressed before the software is released.

A QA team lead creates a customized checklist that suits specific project needs and follows a testing approach that the team had implemented. A checklist is useful when the QA team has a lack of testing time. The document provides all the critical steps that QA engineers should perform before the release. 

Note: a testing checklist is not a synonym for a test plan. A testing checklist is a part of the test plan and QA teams use it to ensure that all the necessary testing tasks are executed. 

A testing matrix, also known as a test matrix or testing traceability matrix, is a document that links test cases to specific requirements, features, or functional areas of a software application. It is a tool that helps ensure that all aspects of the software are tested and that testing is comprehensive and traceable. Below you can see a test matrix example.

The testing matrix typically consists of a table that lists the test cases and the requirements, features, or functional areas. Each cell of the table represents a specific test case and requirement, feature, or functional area combination. The table can also include additional information such as test case status, priority, and any associated defects.

By using a testing matrix, the QA team can ensure that each requirement or feature of the software has corresponding test cases and that all test cases are executed during the testing process. It also helps to ensure that any defects or issues identified during testing can be traced back to their specific requirements or functional areas, making it easier to track progress and ensure that all necessary testing has been completed.