Skip to content

Zebrunner MATLAB reporting agent#

The official Zebrunner MATLAB agent provides reporting capabilities and allows you to analyze tests results in Zebrunner.

Warning

Please note, that agent is currently in an incubating state, meaning that Zebrunner team can not guarantee its stable work.

Inclusion into your project#

Download and install version of the agent

Agent Version Compatible MATLAB versions
1.0.0 (latest) R2018a and higher Download

After downloading you can install the agent using either the MATLAB UI or programmatically.

Using MATLAB UI#

  1. Open MATLAB.
  2. In the HOME section at the top, click the Open button.
  3. Navigate to the location of the downloaded toolbox file (a .mltbx file) and open it.

Once you complete these steps, the Add-On manager will automatically take care of installing the agent toolbox for you.

Programmatically#

  1. Open MATLAB or use the MATLAB console
  2. Run the following script:
clc; clear variables;

import matlab.addons.toolbox.installToolbox;

toolboxFilePath = '<path-to-toolbox>';

toolbox = installToolbox(toolboxFilePath);

After these steps, the agent toolbox will be installed.

Configuring reporting of tests#

Once the agent is added to the test project, it is not automatically enabled. The following steps must be performed:

  1. Provide the valid configuration.
  2. Add plugin com.zebrunner.agent.matlab.ZebrunnerReportingPlugin to matlab.unittest.TestRunner.

Info

com.zebrunner.agent.matlab.ZebrunnerReportingPlugin supports running tests in parallel (If MATLAB Parallel Computing Toolbox Add-On is installed)

Configuration options#

It is currently possible to provide the configuration via:

  1. Environment variables
  2. Using com.zebrunner.agent.matlab.domain.config.ReportingConfig class

The configuration lookup will be performed in the order listed above, meaning that environment configuration will always take the highest precedence.

Common configuration#

The following subsections contain tables with configuration options. These tables contain three columns with configuration options. The first column contains the name of the option represented as an environment variable. The second column contains the name of the option represented as a reporting configuration property from com.zebrunner.agent.matlab.domain.config.ReportingConfig class. The third column contains a description of the configuration option.

Env var ReportingConfig Description
REPORTING_ENABLED enabled Enables or disables reporting. The default value is false.
REPORTING_PROJECT_KEY projectKey Optional value. It is the project that the launch belongs to. The default value is DEF. You can manage projects in Zebrunner in the appropriate section.
REPORTING_SERVER_HOSTNAME server.hostname Mandatory if reporting is enabled. It is Zebrunner server hostname.
REPORTING_SERVER_ACCESS_TOKEN server.accessToken Mandatory if reporting is enabled. Access token must be used to perform API calls.

Info

The server hostname and access token can be found under the Service URL field and API Access section of the Account & profile page in Zebrunner.

Automation launch configuration#

The following configuration options allow you to configure accompanying information that will be displayed in Zebrunner for the automation launch.

Env var ReportingConfig Description
REPORTING_LAUNCH_DISPLAY_NAME launch.displayName Optional value. It is the display name of the launch in Zebrunner. The default value has the following format: Matlab Launch (<launch-start-time> UTC).
REPORTING_LAUNCH_BUILD launch.build Optional value. It is the build number that is associated with the launch. It can depict either the test build number or the application build number.
REPORTING_LAUNCH_ENVIRONMENT launch.environment Optional value. It is the environment where the tests will run.
REPORTING_LAUNCH_TREAT_SKIPS_AS_FAILURES launch.treatSkipsAsFailures Optional value. If the value is set to true, skipped tests will be treated as failures when the result of the entire launch is calculated, otherwise skipped tests will be considered as passed. The default value is true.

Milestone#

Zebrunner Milestone for the automation launch can be configured using the following configuration options.

Env var ReportingConfig Description
REPORTING_MILESTONE_ID milestone.id Optional value. Id of the Zebrunner Milestone to link the automation launch to. The id is not displayed on Zebrunner UI, so the field is basically used for internal purposes. If the milestone does not exist, the launch will continue executing.
REPORTING_MILESTONE_NAME milestone.name Optional value. Name of the Zebrunner Milestone to link the automation launch to. If the milestone does not exist, the appropriate warning message will be displayed in logs, but the launch will continue executing.

Notifications#

Zebrunner provides notification capabilities for automation launch results. The following options configure notification rules and targets.

Env var ReportingConfig Description
REPORTING_NOTIFICATION_NOTIFY_ON_EACH_FAILURE notifications.notifyOnEachFailure Optional value. Specifies whether Zebrunner should send notifications to Slack/Teams on each test failure. The notifications will be sent even if the launch is still running. The default value is false.
REPORTING_NOTIFICATION_SLACK_CHANNELS notifications.slackChannels Optional value. A comma-separated list of Slack channels to send notifications to. Notifications will be sent only if the Slack integration is properly configured in Zebrunner with valid credentials for the project the launch is reported to. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the launch finish.
REPORTING_NOTIFICATION_MS_TEAMS_CHANNELS notifications.teamsChannels Optional value. A comma-separated list of Microsoft Teams channels to send notifications to. Notifications will be sent only if the Teams integration is configured in the Zebrunner project with valid webhooks for the channels. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the launch finish.
REPORTING_NOTIFICATION_EMAILS notifications.emails Optional value. A comma-separated list of emails to send notifications to. This type of notifications does not require further configuration on Zebrunner side. Unlike other notification mechanisms, Zebrunner can send emails only on the launch finish.

Integration with Test Case Management systems#

Zebrunner integrates with different Test Case Management (TCM) systems and provides the following capabilities:

  1. Linking test cases to test executions
  2. Previewing linked test cases in Zebrunner
  3. Pushing test execution results to the TCM system

This functionality is currently supported only for Zebrunner Test Case Management, TestRail, Xray, Zephyr Squad, and Zephyr Scale.

The link between execution of a test method and corresponding test cases can only be set from within the test method code. For more information about this, see the Linking test cases to tests section.

If you want to push the execution results to the TCM system, you need to provide additional configuration for the Agent. For all the supported TCMs, Zebrunner can push results to a pre-created test suite execution (this term has a different name in different systems). For TestRail, you can also create a new Test Run based on the Agent configuration and push the results into it. If enabled, the push can be performed either at the end of the whole launch, or in real time after each test.

The following subsection covers how to provide configuration for pushing results to each of the TCM systems.

Zebrunner Test Case Management (TCM)#
Env var ReportingConfig Description
REPORTING_TCM_ZEBRUNNER_PUSH_RESULTS tcm.zebrunner.pushResults Optional value. Logical value which specifies if the test results should be pushed to Zebrunner TCM. The default value is false.
REPORTING_TCM_ZEBRUNNER_PUSH_IN_REAL_TIME tcm.zebrunner.pushInRealTime Optional value. Logical value which specifies whether to push test results immediately after each test is finished (value true) or not (value false). The default value is false.
REPORTING_TCM_ZEBRUNNER_TEST_RUN_ID tcm.zebrunner.testRunId Optional value. Numeric id of the target Test Run in Zebrunner TCM. If a value is not provided, no new runs will be created.
TestRail#
Env var ReportingConfig Description
REPORTING_TCM_TESTRAIL_PUSH_RESULTS tcm.testRail.pushResults Optional value. Logical value which specifies if the test results should be pushed to TestRail. The default value is false.
REPORTING_TCM_TESTRAIL_PUSH_IN_REAL_TIME tcm.testRail.pushInRealTime Optional value. Logical value which specifies whether to push test results immediately after each test is finished (value true) or not (value false). The default value is false. Enabling of this option forces the includeAllTestCasesInNewRun to be true.
REPORTING_TCM_TESTRAIL_SUITE_ID tcm.testRail.suiteId Optional value. Specifies the numeric id of the TestRail Suite in which the tests reside. TestRail displays the ids prefixed with 'S' letter. You need to provide the id without this letter.
REPORTING_TCM_TESTRAIL_RUN_ID tcm.testRail.runId Optional value. The id of the TestRail Test Run where the results should be pushed. TestRail displays the ids prefixed with 'R' letter. You need to provide the id without this letter.
REPORTING_TCM_TESTRAIL_RUN_NAME tcm.testRail.runName Optional value. Specifies the name of a new Test Run in TestRail. If push is enabled and run id is not provided, Zebrunner will create a new run in TestRail. If the value is not provided, Zebrunner will use the launch display name.
REPORTING_TCM_TESTRAIL_INCLUDE_ALL_IN_NEW_RUN tcm.testRail.includeAllTestCasesInNewRun Optional value. If the value is set to true, all cases from the Suite will be added to the newly created Test Run. The value is forced to be true if real-time push is enabled. Default value is false.
REPORTING_TCM_TESTRAIL_MILESTONE_NAME tcm.testRail.milestoneName Optional value. The newly created Test Run will be associated with the milestone specified using this property.
REPORTING_TCM_TESTRAIL_ASSIGNEE tcm.testRail.assignee Optional value. Assignee of the newly created Test Run. The value should be the email of an existing TestRail user.
Xray#
Env var ReportingConfig Description
REPORTING_TCM_XRAY_PUSH_RESULTS tcm.xray.pushResults Optional value. Logical value which specifies if the test results should be pushed to Xray. The default value is false.
REPORTING_TCM_XRAY_PUSH_IN_REAL_TIME tcm.xray.pushInRealTime Optional value. Logical value which specifies whether to push test results immediately after each test is finished (value true) or not (value false). The default value is false.
REPORTING_TCM_XRAY_EXECUTION_KEY tcm.xray.executionKey Optional value. The key of the Xray Execution where the results should be pushed.
Zephyr#
Env var ReportingConfig Description
REPORTING_TCM_ZEPHYR_PUSH_RESULTS tcm.zephyr.pushResults Optional value. Logical value which specifies if the test results should be pushed to Zephyr. The default value is false.
REPORTING_TCM_ZEPHYR_PUSH_IN_REAL_TIME tcm.zephyr.pushInRealTime Optional value. Logical value which specifies whether to push test results immediately after each test is finished (value true) or not (value false). The default value is false.
REPORTING_TCM_ZEPHYR_JIRA_PROJECT_KEY tcm.zephyr.jiraProjectKey Optional value. Specifies the key of the Jira project where the tests reside.
REPORTING_TCM_ZEPHYR_TEST_CYCLE_KEY tcm.zephyr.testCycleKey Optional value. The key of the Zephyr Test Cycle where the results should be pushed.
Custom Result Statuses#

By default, when the test results are being pushed to a TCM system, Zebrunner maps each test result to an appropriate result status in the target TCM system. Most of the time this works perfectly, but in some cases Zebrunner is not able to derive the appropriate target result status.

One of the examples of such cases is when a test case result status does not correlate with the test status, or when you have conditional logic determining the actual result status for the test case. For such cases, the Agent comes with special classes which sets a specific Result Status to the test case. For more information about this, see the Linking test cases to tests section.

Another example is custom Result Statuses in the target TCM system. In this case, we cannot anticipate the correct status and simply skip the test execution. In order to tackle this, Zebrunner allows you to configure default status for passed, skipped and failed tests.

Env var ReportingConfig Description
REPORTING_TCM_TEST_CASE_STATUS_ON_PASS tcm.testCaseStatus.onPass Optional value. The default status that will be assigned to passed tests when they are pushed to a TCM system.
REPORTING_TCM_TEST_CASE_STATUS_ON_SKIP tcm.testCaseStatus.onSkip Optional value. The default status that will be assigned to skipped tests when they are pushed to a TCM system.
REPORTING_TCM_TEST_CASE_STATUS_ON_FAIL tcm.testCaseStatus.onFail Optional value. The default status that will be assigned to failed tests when they are pushed to a TCM system.

When pushing results to a TCM system, Zebrunner derives the Result Status in the following order:

  1. Checks the explicitly assigned value.
  2. Takes the default status provided via configuration for passed, skipped or failed tests.
  3. Uses internal mapping of Zebrunner statuses to the Result Statuses of the target TCM system.

Examples#

The following code snippet is a list of all configuration environment variables from .env file: 

REPORTING_ENABLED=true
REPORTING_PROJECT_KEY=DEF
REPORTING_SERVER_HOSTNAME=https://mycompany.zebrunner.com
REPORTING_SERVER_ACCESS_TOKEN=somesecretaccesstoken

REPORTING_LAUNCH_DISPLAY_NAME=Nightly Regression
REPORTING_LAUNCH_BUILD=2.41.2.2431-SNAPSHOT
REPORTING_LAUNCH_ENVIRONMENT=QA
REPORTING_LAUNCH_TREAT_SKIPS_AS_FAILURES=true

REPORTING_MILESTONE_ID=57
REPORTING_MILESTONE_NAME=Release 2.41.2

REPORTING_NOTIFICATION_NOTIFY_ON_EACH_FAILURE=false
REPORTING_NOTIFICATION_SLACK_CHANNELS=dev, qa
REPORTING_NOTIFICATION_MS_TEAMS_CHANNELS=dev-channel, management
REPORTING_NOTIFICATION_EMAILS=manager@mycompany.com

REPORTING_TCM_TEST_CASE_STATUS_ON_PASS=PASSED
REPORTING_TCM_TEST_CASE_STATUS_ON_SKIP=SKIPPED
REPORTING_TCM_TEST_CASE_STATUS_ON_FAIL=FAILED

REPORTING_TCM_ZEBRUNNER_PUSH_RESULTS=false
REPORTING_TCM_ZEBRUNNER_PUSH_IN_REAL_TIME=true
REPORTING_TCM_ZEBRUNNER_TEST_RUN_ID=42

REPORTING_TCM_TESTRAIL_PUSH_RESULTS=false
REPORTING_TCM_TESTRAIL_PUSH_IN_REAL_TIME=true
REPORTING_TCM_TESTRAIL_SUITE_ID=100
REPORTING_TCM_TESTRAIL_RUN_ID=500
REPORTING_TCM_TESTRAIL_INCLUDE_ALL_IN_NEW_RUN=true
REPORTING_TCM_TESTRAIL_RUN_NAME=New Demo Run
REPORTING_TCM_TESTRAIL_MILESTONE_NAME=Demo Milestone
REPORTING_TCM_TESTRAIL_ASSIGNEE=tester@mycompany.com

REPORTING_TCM_XRAY_PUSH_RESULTS=false
REPORTING_TCM_XRAY_PUSH_IN_REAL_TIME=true
REPORTING_TCM_XRAY_EXECUTION_KEY=QT-100

REPORTING_TCM_ZEPHYR_PUSH_RESULTS=false
REPORTING_TCM_ZEPHYR_PUSH_IN_REAL_TIME=true
REPORTING_TCM_ZEPHYR_JIRA_PROJECT_KEY=ZEB
REPORTING_TCM_ZEPHYR_TEST_CYCLE_KEY=ZEB-T1

Here you can see an example of the full configuration provided via com.zebrunner.agent.matlab.domain.config.ReportingConfig class:

clc; clear variables;

import com.zebrunner.agent.matlab.ZebrunnerReportingPlugin;
import com.zebrunner.agent.matlab.domain.config.LaunchConfig;
import com.zebrunner.agent.matlab.domain.config.MilestoneConfig;
import com.zebrunner.agent.matlab.domain.config.NotificationsConfig;
import com.zebrunner.agent.matlab.domain.config.ReportingConfig;
import com.zebrunner.agent.matlab.domain.config.ServerConfig;
import com.zebrunner.agent.matlab.domain.config.TestCaseStatusConfig;
import com.zebrunner.agent.matlab.domain.config.TcmConfig;
import com.zebrunner.agent.matlab.domain.config.TestRailTcmConfig;
import com.zebrunner.agent.matlab.domain.config.XrayTcmConfig;
import com.zebrunner.agent.matlab.domain.config.ZephyrTcmConfig;
import com.zebrunner.agent.matlab.domain.config.ZebrunnerTcmConfig;

reportingConfig = ReportingConfig( ...
    'enabled', true, ...
    'projectKey', "DEF", ...
    'server', ServerConfig( ...
        'hostname', "https://mycompany.zebrunner.com", ...
        'accessToken', "somesecretaccesstoken" ...
    ), ...
    'launch', LaunchConfig( ...
        'displayName', "Nightly Regression", ...
        'environment', "QA", ...
        'build', "2.41.2.2431-SNAPSHOT", ...
        'treatSkipsAsFailures', true ...
    ), ...
    'milestone', MilestoneConfig( ...
        'id', 57, ...
        'name', "Release 2.41.2" ...
    ), ...
    'notifications', NotificationsConfig( ...
        'notifyOnEachFailure', false, ...
        'slackChannels', "dev, qa", ...
        'teamsChannels', "dev-channel, management", ...
        'emails', "manager@mycompany.com" ...
    ), ...
    'tcm', TcmConfig( ...
        'testCaseStatus', TestCaseStatusConfig( ...
            'onPass', "PASSED", ...
            'onSkip', "SKIPPED", ...
            'onFail', "FAILED" ...
        ), ...
        'testRail', TestRailTcmConfig( ...
            'pushResults', false, ...
            'pushInRealTime', true, ...
            'suiteId', 100, ...
            'runId', 500, ...
            'includeAllTestCasesInNewRun', true, ...
            'runName', "New Demo Run", ...
            'milestoneName', "Demo Milestone", ...
            'assignee', "tester@mycompany.com" ...
        ), ...
        'xray', XrayTcmConfig( ...
            'pushResults', false, ...
            'pushInRealTime', true, ...
            'executionKey', "QT-100" ...
        ), ...
        'zephyr', ZephyrTcmConfig( ...
            'pushResults', false, ...
            'pushInRealTime', true, ...
            'jiraProjectKey', "ZEB", ...
            'testCycleKey', "ZEB-T1" ...
        ), ...
        'zebrunner', ZebrunnerTcmConfig( ...
            'pushResults', false, ...
            'pushInRealTime', true, ...
            'testRunId', 42 ...
        ) ...
    ) ...
);

zebrunnerReportingPlugin = ZebrunnerReportingPlugin.create(reportingConfig);

Here is the example of how to include plugin and run tests:

clc; clear variables;

import matlab.unittest.TestSuite;
import matlab.unittest.TestRunner;

import com.zebrunner.agent.matlab.ZebrunnerReportingPlugin;

runner = TestRunner.withNoPlugins;
zebrunnerReportingPlugin = ZebrunnerReportingPlugin.create();
runner.addPlugin(zebrunnerReportingPlugin);

% assuming 'tests' folder with tests inside exists
testSuites = TestSuite.fromFolder("tests");

zebrunnerReportingPlugin.registerLaunchStart();

% for serial execution
results = runner.run(testSuites);

% uncomment this line, if you need to run tests in parallel
% note: you need to pre-install 'Parallel Computing Toolbox' Add-On
% results = runner.runInParallel(testSuites);

zebrunnerReportingPlugin.registerLaunchFinish();

Collecting test logs#

The Zebrunner Reporting Plugin collects all the logs which were sent by matlab.unittest.TestCase#log method within a running test. All the logs will have default INFO level and timestamp provided by the test runner. For more customizations use com.zebrunner.agent.matlab.Log class. If a message was logged within the methods (TestClassSetup), methods (TestMethodSetup), methods (TestMethodTeardown) or methods (TestClassTeardown) sections of a test class, it will not be picked up by the Reporting Plugin because these methods are invoked outside the scope of a test.

Here are examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)
           testCase.log("executing empty test...");
        end

        function secondTest(testCase)

           import com.zebrunner.agent.matlab.Log;

           testCase.log(Log.error("This is 'ERROR' log"));
           testCase.log(Log.warn("This is 'WARN' log"));
           testCase.log(Log.info("This is 'INFO' log"));
           testCase.log(Log.debug("This is 'DEBUG' log"));
           testCase.log(Log.trace("This is 'TRACE' log"));

           logLevel = "CRITICAL";
           logMessage = "This is 'CRITICAL' log";

           testCase.log(Log(logLevel, logMessage));

           % some test code
        end

    end

end

Collecting captured screenshots and images#

The Agent allows you to manually take screenshots (and any other kind of image) and upload them to Zebrunner. These captured screenshots will be automatically attached as test log entries.

Manually taken screenshots can be sent to Zebrunner using the com.zebrunner.agent.matlab.Screenshot class. Its constructor accepts the following arguments:

  1. screenshotBytes - An array of uint8 values representing the raw image data.
  2. screenshotCapturedAt (optional) - A datetime object indicating when the screenshot was captured.

If a screenshot was captured within the methods (TestClassSetup), methods (TestMethodSetup), methods (TestMethodTeardown) or methods (TestClassTeardown) sections of a test class, it will be ignored.

Here is an example:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.Screenshot;

            % read screenshot file bytes,
            % assuming file under the path "../data/screenshots/myScreenshot.png" exists
            screenshotFileId = fopen(pwd + "/data/screenshots/myScreenshot.png", 'rb');
            screenshotBytes = fread(screenshotFileId, 'uint8');
            fclose(screenshotFileId);

            capturedAt = datetime("now", 'TimeZone', 'UTC');

            testCase.log(Screenshot(screenshotBytes, capturedAt));

            % some test code
        end

    end

end

Tracking test maintainer#

You may want to add transparency to the process of automation maintenance by having an engineer responsible for evolution of specific tests or test suites. To serve that purpose, Zebrunner comes with a concept of a maintainer.

In order to keep track of those, the Agent comes with the com.zebrunner.agent.matlab.Maintainer class. Its constructor accepts the username of an existing Zebrunner user. If there is no user with the given username, anonymous will be assigned.

Here are the examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.Maintainer;

            testCase.log(Maintainer("Deve Loper"));

            % some test code
        end    

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.Maintainer;

            testCase.log(Maintainer("Joel Miller"));

            % some test code
        end

        function thirdTest(testCase)
            % some test code
        end

    end

end

In this example, Deve Loper will be reported as a maintainer of firstTest(testCase), while Joel Miller will be reported as a maintainer of the secondTest(testCase). For thirdTest(testCase) the maintainer will be anonymous.

Attaching labels to test and launch#

In some cases, it may be useful to attach meta information related to a test or the entire launch.

The agent comes with a concept of labels. Label is a simple key-value pair. The label key and value are represented by a string.

Use com.zebrunner.agent.matlab.TestLabel class to attach a label to a test. You can use com.zebrunner.agent.matlab.LaunchLabel to attach label to the entire launch.

Here are the examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.LaunchLabel;

            testCase.log(LaunchLabel("Priority", "High"))

            % some test code
        end

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.TestLabel;

            testCase.log(TestLabel("Priority", "Medium"));

            % some test code
        end

    end

end

Attaching artifact references to test and launch#

Labels are not the only option for attaching meta information to a test and launch. If the information you want to attach is a link (to a file or webpage), it is more useful to attach it as an artifact reference (or to put it simply as a link).

For this purpose use com.zebrunner.agent.matlab.TestArtifactReference class to attach a reference to a test and com.zebrunner.agent.matlab.LaunchArtifactReference to attach a reference to the entire launch.

Here are the examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.LaunchArtifactReference;

            testCase.log(LaunchArtifactReference("zebrunner landing", "https://zebrunner.com"));

            % some test code
        end

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.TestArtifactReference;

            testCase.log(TestArtifactReference("zebrunner landing", "https://zebrunner.com"));

            % some test code
        end

    end

end

Attaching artifacts to test and launch#

In case your tests or the entire launch produce some artifacts, it may be useful to track them in Zebrunner. The agent comes with convenient classes for uploading artifacts in Zebrunner and linking them to the currently running test or the launch.

For this purpose, there are two classes: com.zebrunner.agent.matlab.TestArtifact for uploading test artifacts and com.zebrunner.agent.matlab.LaunchArtifact for uploading launch artifacts.

com.zebrunner.agent.matlab.TestArtifact can be created using the following ways:

  1. Using its constructor TestArtifact(artifactProvider). artifactProvider is an instance of com.zebrunner.agent.matlab.ArtifactProvider.
  2. Using a static method TestArtifact.asFile(artifactName, filePath, bufferSize). artifactName will be a name for the artifact in Zebrunner, filePath - an actual location of the file on file system (for example: ../data/screenshots/myScreenshot.png), bufferSize - optional argument (default value 1000000), which determines, how many file bytes are loaded into the memory during each read operation.
  3. Using a static method TestArtifact.asRawBytes(artifactName, artifactBytes). artifactName will be a name for the artifact in Zebrunner, artifactBytes - an array of uint8 values representing the raw artifact data.

com.zebrunner.agent.matlab.LaunchArtifact can be created using the following ways:

  1. Using its constructor LaunchArtifact(artifactProvider). artifactProvider is an instance of com.zebrunner.agent.matlab.ArtifactProvider.
  2. Using a static method LaunchArtifact.asFile(artifactName, filePath, bufferSize). artifactName will be a name for the artifact in Zebrunner, filePath - an actual location of the file on file system (for example: ../data/screenshots/myScreenshot.png), bufferSize - optional argument (default value 1000000), which determines, how many file bytes are loaded into the memory during each read operation.
  3. Using a static method LaunchArtifact.asRawBytes(artifactName, artifactBytes). artifactName will be a name for the artifact in Zebrunner, artifactBytes - an array of uint8 values representing the raw artifact data.

Here are the examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.LaunchArtifact;

            % assuming file under the path "../data/artifacts/myLargeImage.png" exists
            testCase.log(LaunchArtifact.asFile("My-Large-Image.png", pwd + "/data/artifacts/myLargeImage.png", 8192));

            % some test code
        end

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.LaunchArtifact;

            % assuming file under the path "../data/artifacts/mySmallImage.png" exists
            mySmallImageFileId = fopen(pwd + "/data/artifacts/mySmallImage.png", 'rb');
            mySmallImageBytes = fread(mySmallImageFileId, 'uint8');
            fclose(mySmallImageFileId);

            testCase.log(LaunchArtifact.asRawBytes("My-Small-Image.png", mySmallImageBytes));

            % some test code
        end

        function thirdTest(testCase)

            import com.zebrunner.agent.matlab.TestArtifact;

            % assuming file under the path "../data/artifacts/myLargeImage.png" exists
            testCase.log(TestArtifact.asFile("My-Large-Image.png", pwd + "/data/artifacts/myLargeImage.png", 8192));

            % some test code
        end

        function lastTest(testCase)

            import com.zebrunner.agent.matlab.TestArtifact;

            % assuming file under the path "../data/artifacts/mySmallImage.png" exists
            mySmallImageFileId = fopen(pwd + "/data/artifacts/mySmallImage.png", 'rb');
            mySmallImageBytes = fread(mySmallImageFileId, 'uint8');
            fclose(mySmallImageFileId);

            testCase.log(TestArtifact.asRawBytes("My-Small-Image.png", mySmallImageBytes));

            % some test code
        end

    end

end

In case if you need to provide artifact from different sources, abstract com.zebrunner.agent.matlab.ArtifactProvider class comes in handy. It provides an ability to upload file in chunks without fully loading it into the memory. See com.zebrunner.agent.matlab.FileArtifactProvider and com.zebrunner.agent.matlab.RawBytesArtifactProvider implementations as examples.

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.LaunchArtifact;
            import com.zebrunner.agent.matlab.FileArtifactProvider;

            % assuming file under the path "../data/artifacts/myLargeImage.png" exists
            fileArtifactProvider = FileArtifactProvider("My-Large-Image.png", pwd + "/data/artifacts/myLargeImage.png", 8192);

            testCase.log(LaunchArtifact(fileArtifactProvider));

            % some test code
        end

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.TestArtifact;
            import com.zebrunner.agent.matlab.RawBytesArtifactProvider;

            % assuming file under the path "../data/artifacts/mySmallImage.png" exists
            mySmallImageFileId = fopen(pwd + "/data/artifacts/mySmallImage.png", 'rb');
            mySmallImageBytes = fread(mySmallImageFileId, 'uint8');
            fclose(mySmallImageFileId);

            rawBytesArtifactProvider = RawBytesArtifactProvider("My-Small-Image.png", mySmallImageBytes);

            testCase.log(TestArtifact(rawBytesArtifactProvider));

            % some test code
        end

    end

end

Reverting test registration#

In some cases, it might be handy not to register tests in Zebrunner. This may be caused by very special circumstances of a test environment or execution conditions.

Zebrunner Agent comes with a com.zebrunner.agent.matlab.TestRegistrationReversion class for reverting test registration at runtime. The following code snippet shows a case where a test is not reported on the first day of a month.

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.TestRegistrationReversion;

            % some condition
            if (datetime().Day == 1)
                testCase.log(TestRegistrationReversion);
            end

            % some test code
        end

    end

end

It is worth mentioning that the registration reversion does not affect the test execution, but simply unregisters the test in Zebrunner. To interrupt the test execution, you need to perform additional actions, for example, throw an Error.

Linking test cases to tests#

Note: to learn more about pushing results to a TCM system, see the Integration with Test Case Management systems section.

Zebrunner TCM#

The Agent comes with the com.zebrunner.agent.matlab.TestCase class which can be used to link test case to a currently executing test. Its constructor accepts the following arguments:

  1. key - A string representing the unique identifier of the test case.
  2. resultStatus (optional) - A string indicating the result status of the test case ( e.g., PASSED, FAILED, SKIPPED etc.). This may be useful if the test case result status does not correlate with the test status, or if you have conditional logic determining the actual result status for the test case.

Only the last testCase#log invocation will take precedence. For example, if you logged the instance of com.zebrunner.agent.matlab.TestCase with key KEY-1 and result status SKIPPED, and then logged another instance with key KEY-1, then the result status you provided with the first instance will be ignored.

Here are the examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.TestCase;

            % links this test to test case with the 'KEY-1' key
            testCase.log(TestCase("KEY-1"));

            % some test code
        end

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.TestCase;

            % links this test to test cases with 'KEY-2' and 'KEY-3' keys
            testCase.log(TestCase("KEY-2"));
            testCase.log(TestCase("KEY-3"));

            % some test code
        end

        function thirdTest(testCase)

            import com.zebrunner.agent.matlab.TestCase;

            % links this test to test case with 'KEY-4'
            testCase.log(TestCase("KEY-4"));

            % some condition
            if (datetime().Day == 1)
                % overriddes the status of the test case when results are pushed to the Zebrunner TCM.
                testCase.log(TestCase("KEY-4", "SKIPPED"));
            end

            % some test code
        end        

    end

end

TestRail#

The Agent comes with the com.zebrunner.agent.matlab.TestRailTestCase class which can be used to link test case to a currently executing test. Its constructor accepts the following arguments:

  1. id - A string representing the unique identifier of the test case.
  2. resultStatus (optional) - A string indicating the result status of the test case ( e.g., PASSED, FAILED, SKIPPED etc.). This may be useful if the test case result status does not correlate with the test status, or if you have conditional logic determining the actual result status for the test case.

Only the last testCase#log invocation will take precedence. For example, if you logged the instance of com.zebrunner.agent.matlab.TestRailTestCase with id C1 and result status SKIPPED, and then logged another instance with id C1, then the result status you provided with the first instance will be ignored.

Here are the examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.TestRailTestCase;

            % links this test to test case with the 'C1' id
            testCase.log(TestRailTestCase("C1"));

            % some test code
        end

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.TestRailTestCase;

            % links this test to test cases with 'C2' and 'C3' ids
            testCase.log(TestRailTestCase("C2"));
            testCase.log(TestRailTestCase("C3"));

            % some test code
        end

        function thirdTest(testCase)

            import com.zebrunner.agent.matlab.TestRailTestCase;

            % links this test to test case with 'C4'
            testCase.log(TestRailTestCase("C4"));

            % some condition
            if (datetime().Day == 1)
                % overriddes the status of the test case when results are pushed to the TestRail.
                % by default Zebrunner maps the test execution result to a result status from TestRail.
                testCase.log(TestRailTestCase("C4", "SKIPPED"));
            end

            % some test code
        end        

    end

end

Xray#

The Agent comes with the com.zebrunner.agent.matlab.XrayTest class which can be used to link Xray test to a currently executing test. Its constructor accepts the following arguments:

  1. key - A string representing the unique identifier of the Xray test.
  2. resultStatus (optional) - A string indicating the result status of the Xray test ( e.g., PASSED, FAILED, SKIPPED etc.). This may be useful if the Xray test result status does not correlate with the test status, or if you have conditional logic determining the actual result status for the Xray test.

Only the last testCase#log invocation will take precedence. For example, if you logged the instance of com.zebrunner.agent.matlab.XrayTest with key KEY-1 and result status SKIPPED, and then logged another instance with key KEY-1, then the result status you provided with the first instance will be ignored.

Here are the examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.XrayTest;

            % links this test to Xray test with the 'KEY-1' key
            testCase.log(XrayTest("KEY-1"));

            % some test code
        end

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.XrayTest;

            % links this test to Xray tests with 'KEY-2' and 'KEY-3' keys
            testCase.log(XrayTest("KEY-2"));
            testCase.log(XrayTest("KEY-3"));

            % some test code
        end

        function thirdTest(testCase)

            import com.zebrunner.agent.matlab.XrayTest;

            % links this test to Xray tests with 'KEY-4' key
            testCase.log(XrayTest("KEY-4"));

            % some condition
            if (datetime().Day == 1)
                % overriddes the status of the test case when results are pushed to the Xray.
                % by default Zebrunner maps the test result to a result status from Xray.
                testCase.log(XrayTest("KEY-4", "SKIPPED"));
            end

            % some test code
        end        

    end

end

Zephyr#

The Agent comes with the com.zebrunner.agent.matlab.ZephyrTestCase class which can be used to link test case to a currently executing test. Its constructor accepts the following arguments:

  1. key - A string representing the unique identifier of the test case.
  2. resultStatus (optional) - A string indicating the result status of the test case ( e.g., PASSED, FAILED, SKIPPED etc.). This may be useful if the test case result status does not correlate with the test status, or if you have conditional logic determining the actual result status for the test case.

Only the last testCase#log invocation will take precedence. For example, if you logged the instance of com.zebrunner.agent.matlab.ZephyrTestCase with key KEY-1 and result status SKIPPED, and then logged another instance with key KEY-1, then the result status you provided with the first instance will be ignored.

Here are the examples:

classdef MyTestCase < matlab.unittest.TestCase

    methods (Test)

        function firstTest(testCase)

            import com.zebrunner.agent.matlab.ZephyrTestCase;

            % links this test to test case with the 'KEY-1' key
            testCase.log(ZephyrTestCase("KEY-1"));

            % some test code
        end

        function secondTest(testCase)

            import com.zebrunner.agent.matlab.ZephyrTestCase;

            % links this test to test cases with 'KEY-2' and 'KEY-3' keys
            testCase.log(ZephyrTestCase("KEY-2"));
            testCase.log(ZephyrTestCase("KEY-3"));

            % some test code
        end

        function thirdTest(testCase)

            import com.zebrunner.agent.matlab.ZephyrTestCase;

            % links this test to test case with 'KEY-4' key
            testCase.log(ZephyrTestCase("KEY-4"));

            % some condition
            if (datetime().Day == 1)
                % overriddes the status of the test case when results are pushed to the Zephyr.
                % by default Zebrunner maps the test result to a result status from Zephyr.
                testCase.log(ZephyrTestCase("KEY-4", "SKIPPED"));
            end

            % some test code
        end        

    end

end