Zebrunner Spock reporting agent

The official Zebrunner Spock agent provides reporting and smart reruns functionality. No special configuration is required to enable the Zebrunner Listener for Spock — service discovery mechanism will automatically register the listener as soon as it becomes available on the classpath of your test project.

Inclusion into your project

Including the agent into your project is easy — just add the dependency to the build descriptor.

GradleMaven

dependencies {
    testImplementation 'com.zebrunner:agent-spock:1.9.6'
}

<dependency>
    <groupId>com.zebrunner</groupId>
    <artifactId>agent-spock</artifactId>
    <version>1.9.6</version>
    <scope>test</scope>
</dependency>

Configuring reporting of test executions

Once the agent is available on the classpath of your test project, it is not automatically enabled. The valid configuration must be provided.

It is currently possible to provide the configuration via:

1. Environment variables
2. Program arguments
3. YAML file
4. Properties file

The configuration lookup will be performed in the order listed above, meaning that environment configuration will always take precedence over YAML and so on. It is also possible to override configuration parameters by passing them through a configuration provider with higher precedence.

via environment variables

The following configuration parameters are recognized by the agent:

Parameter	Description
REPORTING_ENABLED	Enables or disables reporting. The default value is false. If disabled, the agent will use no-op component implementations that will simply log output for tracing purposes with the trace level
REPORTING_SERVER_HOSTNAME	Mandatory if reporting is enabled. It is your Zebrunner hostname
REPORTING_SERVER_ACCESS_TOKEN	Mandatory if reporting is enabled. An access token must be used to perform API calls. It can be obtained in Zebrunner on the 'Account and profile' page in the 'API Access' section
REPORTING_PROJECT_KEY	Optional value. It is the project that a test run belongs to. The default value is DEF. You can manage projects in Zebrunner in the appropriate section
REPORTING_RUN_DISPLAY_NAME	Optional value. It is the display name of a test run. The default value has the following format Spock run [<run start time> (UTC)]
REPORTING_RUN_BUILD	Optional value. It is the build number that is associated with a test run. It can depict either the test build number or the application build number
REPORTING_RUN_ENVIRONMENT	Optional value. It is the environment where tests will run
REPORTING_RUN_LOCALE	Optional value. A locale that will be displayed for a run in Zebrunner if specified
REPORTING_RUN_TREAT_SKIPS_AS_FAILURES	Optional value. The default value is true. If the value is set to true, skipped tests will be treated as failures when processing test run results. As a result, if the value of the property is set to false and a test run contains only skipped and passed tests, the entire test run will be treated as passed.
REPORTING_NOTIFICATION_ENABLED	Enables or disables notifications. The default value is true. When set to true, notifications will be sent to appropriate targets (e.g. slack channels, microsoft teams channels or emails). When set to false, notifications will no be sent at all.
REPORTING_NOTIFICATION_NOTIFY_ON_EACH_FAILURE	Optional value. Specifies whether Zebrunner should send notifications to Slack/Teams on each test failure. The notifications will be sent even if the suite is still running. The default value is false
REPORTING_NOTIFICATION_SLACK_CHANNELS	Optional value. The list of comma-separated Slack channels to send notifications to. Notifications will only be sent if Slack integration is properly configured in Zebrunner with valid credentials for the project the tests are reported to. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the suite finish
REPORTING_NOTIFICATION_MS_TEAMS_CHANNELS	Optional value. The list of comma-separated Microsoft Teams channels to send notifications to. Notifications will only be sent if Teams integration is configured in the Zebrunner project with valid webhooks for the channels. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the suite finish
REPORTING_NOTIFICATION_EMAILS	Optional value. The list of comma-separated emails to send notifications to. This type of notifications does not require further configuration on Zebrunner side. Unlike other notification mechanisms, Zebrunner can send emails only on the suite finish
REPORTING_MILESTONE_ID	Optional value. The id of the Zebrunner milestone to link the suite execution to. The id is not displayed on Zebrunner UI, so the field is basically used for internal purposes. If the milestone does not exist, an appropriate warning message will be displayed in logs, but the test suite will continue executing
REPORTING_MILESTONE_NAME	Optional value. The name of the Zebrunner milestone to link the suite execution to. If the milestone does not exist, an appropriate warning message will be displayed in logs, but the test suite will continue executing
REPORTING_TCM_TEST_CASE_STATUS_ON_PASS	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a passed test (can be overridden at the class or method level)
REPORTING_TCM_TEST_CASE_STATUS_ON_FAIL	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a failed test (can be overridden at the class or method level)
REPORTING_TCM_TEST_CASE_STATUS_ON_SKIP	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a skipped test (can be overridden at the class or method level)
REPORTING_TCM_ZEBRUNNER_PUSH_RESULTS	The default value is true. If the value is set to true, the execution results will be pushed to Zebrunner TCM.
REPORTING_TCM_ZEBRUNNER_PUSH_IN_REAL_TIME	The default value is false. If the value is set to true, the execution results will be pushed immediately to Zebrunner TCM after each test is finished.
REPORTING_TCM_ZEBRUNNER_RUN_ID	Numeric id of the target Test Run in Zebrunner TCM. If the value is not provided, no new runs will be created.
REPORTING_TCM_TEST_RAIL_PUSH_RESULTS	The default value is true. If the value is set to true, the execution results will be pushed to TestRail.
REPORTING_TCM_TEST_RAIL_PUSH_IN_REAL_TIME	The default value is false. If the value is set to true, the execution results will be pushed immediately to TestRail after each test is finished.
REPORTING_TCM_TEST_RAIL_SUITE_ID	Numeric id of the target TestRail suite in which the tests reside. Note: TestRail displays the ids prefixed with 'S' letter. You need to provide the id without this letter.
REPORTING_TCM_TEST_RAIL_RUN_ID	Numeric id of the target TestRail test run where the execution results should be pushed. Note: TestRail displays the ids prefixed with 'R' letter. You need to provide the id without this letter.
REPORTING_TCM_TEST_RAIL_INCLUDE_ALL_TEST_CASES_IN_NEW_RUN	The default value is false. If the value is set to true, all cases from the Suite will be added to the newly created Test Run. The value is forced to be true if real-time push is enabled.
REPORTING_TCM_TEST_RAIL_RUN_NAME	If push is enabled and run id is not provided, Zebrunner will create a new run in TestRail. If the value is not provided, Zebrunner will use the launch display name.
REPORTING_TCM_TEST_RAIL_MILESTONE_NAME	The newly created Test Run will be associated with the milestone specified using this property.
REPORTING_TCM_TEST_RAIL_ASSIGNEE	Assignee of the newly created Test Run. The value should be the email of an existing TestRail user.
REPORTING_TCM_XRAY_PUSH_RESULTS	The default value is true. If the value is set to true, the execution results will be pushed to Xray.
REPORTING_TCM_XRAY_PUSH_IN_REAL_TIME	The default value is false. If the value is set to true, the execution results will be pushed immediately to Xray after each test is finished.
REPORTING_TCM_XRAY_EXECUTION_KEY	The key of the Xray Execution where the results should be pushed.
REPORTING_TCM_ZEPHYR_PUSH_RESULTS	The default value is true. If the value is set to true, the execution results will be pushed to Zephyr.
REPORTING_TCM_ZEPHYR_PUSH_IN_REAL_TIME	The default value is false. If the value is set to true, the execution results will be pushed immediately to Zephyr after each test is finished.
REPORTING_TCM_ZEPHYR_JIRA_PROJECT_KEY	Specifies the key of the Jira project where the tests reside.
REPORTING_TCM_ZEPHYR_TEST_CYCLE_KEY	The key of the Zephyr Test Cycle where the results should be pushed.

via program arguments

The following configuration parameters are recognized by the agent:

Parameter	Description
reporting.enabled	Enables or disables reporting. The default value is false. If disabled, the agent will use no-op component implementations that will simply log output for tracing purposes with the trace level
reporting.server.hostname	Mandatory if reporting is enabled. It is your Zebrunner hostname
reporting.server.access-token	Mandatory if reporting is enabled. An access token must be used to perform API calls. It can be obtained in Zebrunner on the 'Account and profile' page in the 'API Access' section
reporting.projectKey	Optional value. It is the project that a test run belongs to. The default value is DEF. You can manage projects in Zebrunner in the appropriate section
reporting.run.displayName	Optional value. It is the display name of a test run. The default value has the following format Spock run [<run start time> (UTC)]
reporting.run.build	Optional value. It is the build number that is associated with a test run. It can depict either the test build number or the application build number
reporting.run.environment	Optional value. It is the environment where tests will run
reporting.run.locale	Optional value. A locale that will be displayed for a run in Zebrunner if specified
reporting.run.treat-skips-as-failures	Optional value. The default value is true. If the value is set to true, skipped tests will be treated as failures when processing test run results. As a result, if the value of the property is set to false and a test run contains only skipped and passed tests, the entire test run will be treated as passed.
reporting.notification.enabled	Enables or disables notifications. The default value is true. When set to true, notifications will be sent to appropriate targets (e.g. slack channels, microsoft teams channels or emails). When set to false, notifications will no be sent at all.
reporting.notification.notify-on-each-failure	Optional value. Specifies whether Zebrunner should send notifications to Slack/Teams on each test failure. The notifications will be sent even if the suite is still running. The default value is false
reporting.notification.slack-channels	Optional value. The list of comma-separated Slack channels to send notifications to. Notifications will only be sent if Slack integration is properly configured in Zebrunner with valid credentials for the project the tests are reported to. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the suite finish
reporting.notification.ms-teams-channels	Optional value. The list of comma-separated Microsoft Teams channels to send notifications to. Notifications will only be sent if Teams integration is configured in the Zebrunner project with valid webhooks for the channels. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the suite finish
reporting.notification.emails	Optional value. The list of comma-separated emails to send notifications to. This type of notifications does not require further configuration on Zebrunner side. Unlike other notification mechanisms, Zebrunner can send emails only on the suite finish
reporting.milestone.id	Optional value. The id of the Zebrunner milestone to link the suite execution to. The id is not displayed on Zebrunner UI, so the field is basically used for internal purposes. If the milestone does not exist, an appropriate warning message will be displayed in logs, but the test suite will continue executing
reporting.milestone.name	Optional value. The name of the Zebrunner milestone to link the suite execution to. If the milestone does not exist, an appropriate warning message will be displayed in logs, but the test suite will continue executing
reporting.tcm.test-case-status.on-pass	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a passed test (can be overridden at the class or method level)
reporting.tcm.test-case-status.on-fail	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a failed test (can be overridden at the class or method level)
reporting.tcm.test-case-status.on-skip	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a skipped test (can be overridden at the class or method level)
reporting.tcm.zebrunner.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Zebrunner TCM.
reporting.tcm.zebrunner.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Zebrunner TCM after each test is finished.
reporting.tcm.zebrunner.run-id	Numeric id of the target Test Run in Zebrunner TCM. If the value is not provided, no new runs will be created.
reporting.tcm.test-rail.push-results	The default value is true. If the value is set to true, the execution results will be pushed to TestRail.
reporting.tcm.test-rail.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to TestRail after each test is finished.
reporting.tcm.test-rail.suite-id	Numeric id of the target TestRail suite in which the tests reside. Note: TestRail displays the ids prefixed with 'S' letter. You need to provide the id without this letter.
reporting.tcm.test-rail.run-id	Numeric id of the target TestRail test run where the execution results should be pushed. Note: TestRail displays the ids prefixed with 'R' letter. You need to provide the id without this letter.
reporting.tcm.test-rail.include-all-test-cases-in-new-run	The default value is false. If the value is set to true, all cases from the Suite will be added to the newly created Test Run. The value is forced to be true if real-time push is enabled.
reporting.tcm.test-rail.run-name	If push is enabled and run id is not provided, Zebrunner will create a new run in TestRail. If the value is not provided, Zebrunner will use the launch display name.
reporting.tcm.test-rail.milestone-name	The newly created Test Run will be associated with the milestone specified using this property.
reporting.tcm.test-rail.assignee	Assignee of the newly created Test Run. The value should be the email of an existing TestRail user.
reporting.tcm.xray.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Xray.
reporting.tcm.xray.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Xray after each test is finished.
reporting.tcm.xray.execution-key	The key of the Xray Execution where the results should be pushed.
reporting.tcm.zephyr.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Zephyr.
reporting.tcm.zephyr.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Zephyr after each test is finished.
reporting.tcm.zephyr.jira-project-key	Specifies the key of the Jira project where the tests reside.
reporting.tcm.zephyr.test-cycle-key	The key of the Zephyr Test Cycle where the results should be pushed.

via YAML file

Agent recognizes the agent.yaml or agent.yml file in the resources root folder. It is currently not possible to configure an alternative file location.

Below is a sample configuration file:

reporting:
  enabled: true
  project-key: WEB
  server:
    hostname: bestcompany.zebrunner.com
    access-token: <token>
  run:
    display-name: Nightly Regression
    build: 2.5.3.96-SNAPSHOT
    environment: TEST-1
    locale: en_US
    treat-skips-as-failures: false
  notification:
    enabled: true
    notify-on-each-failure: true
    slack-channels: automation, dev-team
    ms-teams-channels: automation, qa-team
    emails: boss@example.com
  milestone:
    name: Release 2.5.3
  tcm:
    test-case-status:
      on-pass: passed
      on-fail: failed
      on-skip: retest
    zebrunner:
      push-results: true
      push-in-real-time: false
      test-run-id: 1
    test-rail:
      push-results: true
      push-in-real-time: false
      suite-id: 1
      run-id: 1
      include-all-test-cases-in-new-run: false
      run-name: testRun
      milestone-name: testMilestone
      assignee: tester@mycompany.com
    xray:
      push-results: true
      push-in-real-time: false
      execution-key: QT-1
    zephyr:
      push-results: true
      push-in-real-time: false
      jira-project-key: ZEB
      test-cycle-key: ZEB-T1

The following configuration parameters are recognized by the agent:

Parameter	Description
reporting.enabled	Enables or disables reporting. The default value is false. If disabled, the agent will use no-op component implementations that will simply log output for tracing purposes with the trace level
reporting.server.hostname	Mandatory if reporting is enabled. It is your Zebrunner hostname
reporting.server.access-token	Mandatory if reporting is enabled. An access token must be used to perform API calls. It can be obtained in Zebrunner on the 'Account and profile' page in the 'API Access' section
reporting.projectKey	Optional value. It is the project that a test run belongs to. The default value is DEF. You can manage projects in Zebrunner in the appropriate section
reporting.run.displayName	Optional value. It is the display name of a test run. The default value has the following format Spock run [<run start time> (UTC)]
reporting.run.build	Optional value. It is the build number that is associated with a test run. It can depict either the test build number or the application build number
reporting.run.environment	Optional value. It is the environment where tests will run
reporting.run.locale	Optional value. A locale that will be displayed for a run in Zebrunner if specified
reporting.run.treat-skips-as-failures	Optional value. The default value is true. If the value is set to true, skipped tests will be treated as failures when processing test run results. As a result, if the value of the property is set to false and test run contains only skipped and passed tests, the entire test run will be treated as passed.
reporting.notification.enabled	Enables or disables notifications. The default value is true. When set to true, notifications will be sent to appropriate targets (e.g. slack channels, microsoft teams channels or emails). When set to false, notifications will no be sent at all.
reporting.notification.notify-on-each-failure	Optional value. Specifies whether Zebrunner should send notifications to Slack/Teams on each test failure. The notifications will be sent even if the suite is still running. The default value is false
reporting.notification.slack-channels	Optional value. The list of comma-separated Slack channels to send notifications to. Notifications will be sent only if Slack integration is properly configured in Zebrunner with valid credentials for the project the tests are reported to. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the suite finish
reporting.notification.ms-teams-channels	Optional value. The list of comma-separated Microsoft Teams channels to send notifications to. Notifications will be sent only if Teams integration is configured in the Zebrunner project with valid webhooks for the channels. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the suite finish
reporting.notification.emails	Optional value. The list of comma-separated emails to send notifications to. This type of notifications does not require further configuration on Zebrunner side. Unlike other notification mechanisms, Zebrunner can send emails only on the suite finish
reporting.milestone.id	Optional value. The id of the Zebrunner milestone to link the suite execution to. The id is not displayed on Zebrunner UI, so the field is basically used for internal purposes. If the milestone does not exist, an appropriate warning message will be displayed in logs, but the test suite will continue executing
reporting.milestone.name	Optional value. The name of the Zebrunner milestone to link the suite execution to. If the milestone does not exist, an appropriate warning message will be displayed in logs, but the test suite will continue executing
reporting.tcm.test-case-status.on-pass	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a passed test (can be overridden at the class or method level)
reporting.tcm.test-case-status.on-fail	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a failed test (can be overridden at the class or method level)
reporting.tcm.test-case-status.on-skip	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a skipped test (can be overridden at the class or method level)
reporting.tcm.zebrunner.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Zebrunner TCM.
reporting.tcm.zebrunner.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Zebrunner TCM after each test is finished.
reporting.tcm.zebrunner.run-id	Numeric id of the target Test Run in Zebrunner TCM. If the value is not provided, no new runs will be created.
reporting.tcm.test-rail.push-results	The default value is true. If the value is set to true, the execution results will be pushed to TestRail.
reporting.tcm.test-rail.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to TestRail after each test is finished.
reporting.tcm.test-rail.suite-id	Numeric id of the target TestRail suite in which the tests reside. Note: TestRail displays the ids prefixed with 'S' letter. You need to provide the id without this letter.
reporting.tcm.test-rail.run-id	Numeric id of the target TestRail test run where the execution results should be pushed. Note: TestRail displays the ids prefixed with 'R' letter. You need to provide the id without this letter.
reporting.tcm.test-rail.include-all-test-cases-in-new-run	The default value is false. If the value is set to true, all cases from the Suite will be added to the newly created Test Run. The value is forced to be true if real-time push is enabled.
reporting.tcm.test-rail.run-name	If push is enabled and run id is not provided, Zebrunner will create a new run in TestRail. If the value is not provided, Zebrunner will use the launch display name.
reporting.tcm.test-rail.milestone-name	The newly created Test Run will be associated with the milestone specified using this property.
reporting.tcm.test-rail.assignee	Assignee of the newly created Test Run. The value should be the email of an existing TestRail user.
reporting.tcm.xray.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Xray.
reporting.tcm.xray.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Xray after each test is finished.
reporting.tcm.xray.execution-key	The key of the Xray Execution where the results should be pushed.
reporting.tcm.zephyr.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Zephyr.
reporting.tcm.zephyr.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Zephyr after each test is finished.
reporting.tcm.zephyr.jira-project-key	Specifies the key of the Jira project where the tests reside.
reporting.tcm.zephyr.test-cycle-key	The key of the Zephyr Test Cycle where the results should be pushed.

via .properties file

The agent recognizes only the agent.properties file in the resources root folder. It is currently not possible to configure an alternative file location.

Below is a sample configuration file:

reporting.enabled=true
reporting.project-key=WEB

reporting.server.hostname=bestcompany.zebrunner.com
reporting.server.access-token=<token>

reporting.run.display-name=Nightly Regression
reporting.run.build=2.5.3.96-SNAPSHOT
reporting.run.environment=TEST-1
reporting.run.locale=en_US
reporting.run.treat-skips-as-failures=false

reporting.notification.enabled=true
reporting.notification.notify-on-each-failure=true
reporting.notification.slack-channels=automation, dev-team
reporting.notification.ms-teams-channels=automation, qa-team
reporting.notification.emails=boss@example.com

reporting.milestone.name=Release 2.5.3

reporting.tcm.test-case-status.on-pass=passed
reporting.tcm.test-case-status.on-fail=failed
reporting.tcm.test-case-status.on-skip=retest

reporting.tcm.zebrunner.push-results=true
reporting.tcm.zebrunner.push-in-real-time=false
reporting.tcm.zebrunner.test-run-id=1

reporting.tcm.test-rail.push-results=true
reporting.tcm.test-rail.push-in-real-time=false
reporting.tcm.test-rail.suite-id=1
reporting.tcm.test-rail.run-id=1
reporting.tcm.test-rail.include-all-test-cases-in-new-run=false
reporting.tcm.test-rail.run-name=testRun
reporting.tcm.test-rail.milestone-name=testMilestone
reporting.tcm.test-rail.assignee=tester@mycompany.com

reporting.tcm.xray.push-results=true
reporting.tcm.xray.push-in-real-time=false
reporting.tcm.xray.execution-key=QT-1

reporting.tcm.zephyr.push-results=true
reporting.tcm.zephyr.push-in-real-time=false
reporting.tcm.zephyr.jira-project-key=ZEB
reporting.tcm.zephyr.test-cycle-key=ZEB-T1

The following configuration parameters are recognized by the agent:

Parameter	Description
reporting.enabled	Enables or disables reporting. The default value is false. If disabled, the agent will use no-op component implementations that will simply log output for tracing purposes with the trace level
reporting.server.hostname	Mandatory if reporting is enabled. It is your Zebrunner hostname
reporting.server.access-token	Mandatory if reporting is enabled. An access token must be used to perform API calls. It can be obtained in Zebrunner on the 'Account and profile' page in the 'API Access' section
reporting.projectKey	Optional value. It is the project that a test run belongs to. The default value is DEF. You can manage projects in Zebrunner in the appropriate section
reporting.run.displayName	Optional value. It is the display name of a test run. The default value has the following format Spock run [<run start time> (UTC)]
reporting.run.build	Optional value. It is the build number that is associated with the test run. It can depict either the test build number or the application build number
reporting.run.environment	Optional value. It is the environment where tests will run
reporting.run.locale	Optional value. A locale that will be displayed for a run in Zebrunner if specified
reporting.run.treat-skips-as-failures	Optional value. The default value is true. If the value is set to true, skipped tests will be treated as failures when processing test run results. As a result, if the value of the property is set to false and test run contains only skipped and passed tests, the entire test run will be treated as passed.
reporting.notification.enabled	Enables or disables notifications. The default value is true. When set to true, notifications will be sent to appropriate targets (e.g. slack channels, microsoft teams channels or emails). When set to false, notifications will no be sent at all.
reporting.notification.notify-on-each-failure	Optional value. Specifies whether Zebrunner should send notifications to Slack/Teams on each test failure. The notifications will be sent even if the suite is still running. The default value is false
reporting.notification.slack-channels	Optional value. The list of comma-separated Slack channels to send notifications to. Notifications will be sent only if Slack integration is properly configured in Zebrunner with valid credentials for the project the tests are reported to. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the suite finish
reporting.notification.ms-teams-channels	Optional value. The list of comma-separated Microsoft Teams channels to send notifications to. Notifications will be sent only if Teams integration is configured in the Zebrunner project with valid webhooks for the channels. Zebrunner can send two types of notifications: on each test failure (if the appropriate property is enabled) and on the suite finish
reporting.notification.emails	Optional value. The list of comma-separated emails to send notifications to. This type of notifications does not require further configuration on Zebrunner side. Unlike other notification mechanisms, Zebrunner can send emails only on suite finish
reporting.milestone.id	Optional value. The id of the Zebrunner milestone to link the suite execution to. The id is not displayed on Zebrunner UI, so the field is basically used for internal purposes. If the milestone does not exist, an appropriate warning message will be displayed in logs, but the test suite will continue executing
reporting.milestone.name	Optional value. The name of the Zebrunner milestone to link the suite execution to. If the milestone does not exist, an appropriate warning message will be displayed in logs, but the test suite will continue executing
reporting.tcm.test-case-status.on-pass	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a passed test (can be overridden at the class or method level)
reporting.tcm.test-case-status.on-fail	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a failed test (can be overridden at the class or method level)
reporting.tcm.test-case-status.on-skip	If automated tests are associated with test cases from a TCM system (TestRail, Xray, Zephyr) and there is a setting to push the execution results to the TCM, then this config determines what status will be set for a skipped test (can be overridden at the class or method level)
reporting.tcm.zebrunner.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Zebrunner TCM.
reporting.tcm.zebrunner.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Zebrunner TCM after each test is finished.
reporting.tcm.zebrunner.run-id	Numeric id of the target Test Run in Zebrunner TCM. If the value is not provided, no new runs will be created.
reporting.tcm.test-rail.push-results	The default value is true. If the value is set to true, the execution results will be pushed to TestRail.
reporting.tcm.test-rail.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to TestRail after each test is finished.
reporting.tcm.test-rail.suite-id	Numeric id of the target TestRail suite in which the tests reside. Note: TestRail displays the ids prefixed with 'S' letter. You need to provide the id without this letter.
reporting.tcm.test-rail.run-id	Numeric id of the target TestRail test run where the execution results should be pushed. Note: TestRail displays the ids prefixed with 'R' letter. You need to provide the id without this letter.
reporting.tcm.test-rail.include-all-test-cases-in-new-run	The default value is false. If the value is set to true, all cases from the Suite will be added to the newly created Test Run. The value is forced to be true if real-time push is enabled.
reporting.tcm.test-rail.run-name	If push is enabled and run id is not provided, Zebrunner will create a new run in TestRail. If the value is not provided, Zebrunner will use the launch display name.
reporting.tcm.test-rail.milestone-name	The newly created Test Run will be associated with the milestone specified using this property.
reporting.tcm.test-rail.assignee	Assignee of the newly created Test Run. The value should be the email of an existing TestRail user.
reporting.tcm.xray.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Xray.
reporting.tcm.xray.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Xray after each test is finished.
reporting.tcm.xray.execution-key	The key of the Xray Execution where the results should be pushed.
reporting.tcm.zephyr.push-results	The default value is true. If the value is set to true, the execution results will be pushed to Zephyr.
reporting.tcm.zephyr.push-in-real-time	The default value is false. If the value is set to true, the execution results will be pushed immediately to Zephyr after each test is finished.
reporting.tcm.zephyr.jira-project-key	Specifies the key of the Jira project where the tests reside.
reporting.tcm.zephyr.test-cycle-key	The key of the Zephyr Test Cycle where the results should be pushed.

Once the configuration is set up, the agent is ready to track your test run events, with no additional configuration required. However, to get the most out of reporting capabilities, it is also possible to configure the agent to collect additional test data such as logs, screenshots (if applicable) and more generic artifacts. Additionally, custom metadata can be attached to both test run itself and tests, improving your reporting experience with Zebrunner.

Collecting test logs

It is also possible to enable the log collection for your tests. Currently, three logging frameworks are supported out of the box: logback, log4j, log4j2. We recommend using slf4j (Simple Logging Facade for Java) which provides abstraction over logging libraries. All you have to do to enable logging is to register the reporting appender in your test framework configuration file.

with Logback

Add logback (and, optionally, slf4j) dependencies to your build descriptor.

GradleMaven

dependencies {
    implementation 'org.slf4j:slf4j-api:1.7.30'
    implementation 'ch.qos.logback:logback-core:1.2.3'
    implementation 'ch.qos.logback:logback-classic:1.2.3'
}

<dependencies>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.30</version>
    </dependency>
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-core</artifactId>
        <version>1.2.3</version>
    </dependency>
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-classic</artifactId>
        <version>1.2.3</version>
    </dependency>
</dependencies>

Add logging appender to the logback.xml file. Feel free to customize the logging pattern according to your needs:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="ZebrunnerAppender" class="com.zebrunner.agent.core.logging.logback.ReportingAppender">
       <encoder>
          <pattern>%d{HH:mm:ss.SSS} [%t] %-5level - %msg%n</pattern>
       </encoder>
    </appender>
    <root level="info">
        <appender-ref ref="ZebrunnerAppender" />
    </root>
</configuration>

with Log4j

Add log4j (and, optionally, slf4j) dependency to your build descriptor.

GradleMaven

dependencies {
    implementation 'org.slf4j:slf4j-api:1.7.30'
    implementation 'log4j:log4j:1.2.17'
}

<dependencies>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.30</version>
    </dependency>
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
    </dependency>
</dependencies>

Add logging appender in the log4j.properties file. Feel free to customize the logging pattern according to your needs:

log4j.rootLogger = INFO, zebrunner
log4j.appender.zebrunner=com.zebrunner.agent.core.logging.log4j.ReportingAppender
log4j.appender.zebrunner.layout=org.apache.log4j.PatternLayout
log4j.appender.zebrunner.layout.conversionPattern=pattern">[%d{HH:mm:ss}] %-5p (%F:%L) - %m%n

with Log4j 2

Add log4j2 (and, optionally, slf4j) dependency to your build descriptor:

GradleMaven

dependencies {
    implementation 'org.slf4j:slf4j-api:1.7.36'
    implementation 'org.apache.logging.log4j:log4j-api:2.17.2'
    implementation 'org.apache.logging.log4j:log4j-core:2.17.2'
    implementation 'org.apache.logging.log4j:log4j-slf4j-impl:2.17.2'
}

<dependencies>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.36</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>2.17.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.17.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-slf4j-impl</artifactId>
        <version>2.17.2</version>
    </dependency>
</dependencies>

Add logging appender to the log4j2.xml file. Feel free to customize the logging pattern according to your needs:

<?xml version="1.0" encoding="UTF-8"?>
<configuration packages="com.zebrunner.agent.core.logging.log4j2">
   <properties>
      <property name="pattern">[%d{HH:mm:ss}] %-5p (%F:%L) - %m%n</property>
   </properties>
   <appenders>
      <ReportingAppender name="ReportingAppender">
         <PatternLayout pattern="${pattern}" />
      </ReportingAppender>
   </appenders>
   <loggers>
      <root level="info">
         <appender-ref ref="ReportingAppender"/>
      </root>
   </loggers>
</configuration>

Logger usage

No additional steps are required to collect test logs and track them in Zebrunner.

import org.slf4j.Logger
import org.slf4j.LoggerFactory
import spock.lang.Specification

import java.lang.invoke.MethodHandles

class AwesomeSpecification extends Specification {

    private static final Logger LOGGER = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass())

    def "my awesome feature"() {
        when:
        // some code here
        LOGGER.info("Feature info")

        then:
        // meaningful assertions
    }

}

Collecting captured screenshots

In case you are using Spock as a UI testing framework, it may be useful to have an ability to track captured screenshots in scope of Zebrunner reporting. The agent comes with a Java API allowing you to send your screenshots to Zebrunner, so they will be attached to the test.

Below is a sample code of test sending a screenshot to Zebrunner:

import com.zebrunner.agent.core.registrar.Screenshot
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    def "my awesome feature"() {
        when:
        // some code here
        byte[] screenshotBytes = // capture screenshot
        Long capturedAtMillis = // track screenshot capture time
        Screenshot.upload(screenshotBytes, capturedAtMillis)

        then:
        // meaningful assertions
    }

}

A screenshot should be passed as a byte array along with a unix timestamp in milliseconds corresponding to the moment when the screenshot was captured. If null is supplied instead of a timestamp, it will be generated automatically. However, it is recommended to use an accurate timestamp in order to get accurate tracking.

The uploaded screenshot will appear among test logs. The actual position depends on the provided (or generated) timestamp.

Additional reporting capabilities

This section uncovers extra reporting capabilities allowing to track additional test data and programmatically manipulate the tracking process.

Tracking test maintainer

You may want to add transparency to the process of automation maintenance by having an engineer responsible for evolution of specific tests or test classes. To serve that purpose, Zebrunner comes with a concept of a maintainer. In order to keep track of those, the agent comes with the @Maintainer annotation.

This annotation can be placed on both test class and method. It is also possible to override a class-level maintainer on a method-level. If a base test class is marked with this annotation, all child classes will inherit the annotation unless they have an explicitly specified one.

See a sample test class below:

import com.zebrunner.agent.core.annotation.Maintainer
import spock.lang.Specification

@Maintainer("kenobi")
class AwesomeSpecification extends Specification {

    @Maintainer("skywalker")
    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

    def "another awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

}

In the example above, kenobi will be reported as a maintainer of another awesome feature (class-level value taken into account), while skywalker will be reported as a maintainer of my awesome feature.

The maintainer username should be a valid Zebrunner username, otherwise it will be set to anonymous.

Attaching labels

In some cases, it may be useful to attach some meta information related to a test.

The agent comes with a concept of labels. Label is a key-value pair associated with a test or test run. The key is represented by a String, the label value accepts a vararg of Strings.

There is a repeatable @TestLabel annotation that can be used to attach labels to a test. The annotation can be used on both class and method levels and will attach labels to a test. It is also possible to override a class-level label on a method-level.

There is also a Java API to attach labels during a test or execution. The Label class has static methods that can be used to attach labels.

Here is a sample:

import com.zebrunner.agent.core.annotation.TestLabel
import com.zebrunner.agent.core.registrar.Label
import spock.lang.Specification

@TestLabel(name = "specification version", value = "v1.0")
class AwesomeSpecification extends Specification {

    @TestLabel(name = "feature", value = "labels")
    @TestLabel(name = "app", value = ["reporting-service:v1.0", "reporting-service:v1.1"])
    def "my awesome feature"() {
        when:
        Label.attachToTest("Chrome", "85.0")
        Label.attachToTestRun("Author", "Deve Loper")
        // some code here

        then:
        // meaningful assertions
    }

}

In the sample above there are 5 attached test-level labels (1 specification version label, 2 app labels, 1 feature label, 1 Chrome label) and 1 run-level label (Author).

The values of attached labels will be displayed in Zebrunner under the name of a corresponding test or run.

Reverting test registration

In some cases, it may be handy not to register test execution in Zebrunner. This may be caused by very special circumstances of the test environment or execution conditions.

Zebrunner agent comes with a convenient method #revertRegistration() from CurrentTest class for reverting test registration at runtime. The following code snippet shows a case where a test is not reported on Wednesday.

import com.zebrunner.agent.core.registrar.CurrentTest
import spock.lang.Specification

import java.time.DayOfWeek
import java.time.LocalDate

class AwesomeSpecification extends Specification {

    def "my awesome feature"() {
        when:
        if (LocalDate.now().getDayOfWeek() == DayOfWeek.WEDNESDAY) {
            CurrentTest.revertRegistration()
        }
        // some code here

        then:
        // meaningful assertions
    }

}

It is worth mentioning that the method invocation does not affect the test execution, but simply unregisters the test in Zebrunner. To interrupt the test execution, you need to do additional actions, for example, throw a TestAbortedException.

Overriding run attributes

This section contains information on agent APIs allowing to manipulate test run attributes during runtime.

Setting build at runtime

All the configuration mechanisms listed above provide the possibility to declaratively set a test run build. But there might be cases when an actual build becomes available only at runtime.

For such cases, Zebrunner agent has a special method that can be used at any moment of the specification execution:

import com.zebrunner.agent.core.registrar.CurrentTestRun
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    def setupSpec() {
        String build = // resolve build
        CurrentTestRun.setBuild(build)
    }

    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

}

In the above example, the #setupSpec() method resolves and sets the test run build.

Setting locale

If you want to get full reporting experience and collect as much information in Zebrunner as possible, you may want to report the test run locale.

For this, Zebrunner agent has a special method that can be used at any moment of the specification execution:

import com.zebrunner.agent.core.registrar.CurrentTestRun
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    def setupSpec() {
        String locale = // resolve locale
        CurrentTestRun.setLocale(locale)
    }

    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

}

In the above example, the #setupSpec() method resolves and sets the test run locale.

Overriding platform

A test run in Zebrunner may have a platform associated with it. If there is at least one initiated RemoteDriverSession within a test run, then its platform will be displayed as a platform of the whole test run. Even if subsequent RemoteDriverSessions are initiated on another platform, the very first one will be displayed as the run platform.

In some cases, you may want to override the platform of the first RemoteDriverSession. Another problem is that it is not possible to specify API as a platform.

Zebrunner provides two special methods to solve both of these problems: CurrentTestRun.setPlatform(name) and CurrentTestRun.setPlatform(name, version).

In the example below, the #setupSpec() method sets the API as a platform associated with the current test run.

import com.zebrunner.agent.core.registrar.CurrentTestRun
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    def setupSpec() {
        CurrentTestRun.setPlatform("API")
    }

    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

}

Collecting additional artifacts

In case your tests or an entire test run produce some artifacts, it may be useful to track them in Zebrunner. The agent comes with a few convenient methods for uploading artifacts in Zebrunner and linking them to the currently running test or the test run.

Artifacts can be uploaded using the Artifact class. This class has a bunch of static methods to either attach an arbitrary artifact reference or upload artifacts represented by any Java type associated with the files.

The #attachToTestRun(name, file) and #attachToTest(name, file) methods can be used to upload and attach an artifact file to a test run and a test respectively.

The #attachReferenceToTestRun(name, reference) and #attachReferenceToTest(name, reference) methods can be used to attach an arbitrary artifact reference to a test run and a test respectively.

Together with an artifact or an artifact reference, you must provide the display name. For the file, this name must contain the file extension that reflects the actual content of the file. If the file extension does not match the file content, this file will not be saved in Zebrunner. Artifact reference can have an arbitrary name.

Here is a sample test that uploads 3 artifacts for a test, 1 artifact for a test run, and attaches 1 artifact reference to a test and 1 to a test run:

import com.zebrunner.agent.core.registrar.Artifact
import spock.lang.Specification

import java.nio.file.Path

class AwesomeSpecification extends Specification {

    def "my awesome feature"() {
        when:
        InputStream inputStream
        byte[] byteArray
        File file
        Path path

        Artifact.attachToTestRun("file.docx", inputStream)

        Artifact.attachToTest("image.png", byteArray)
        Artifact.attachToTest("application.apk", file)
        Artifact.attachToTest("test-log.txt", path)

        Artifact.attachReferenceToTestRun("Zebrunner in Github", "https://github.com/zebrunner")

        Artifact.attachReferenceToTest("zebrunner.com", "https://zebrunner.com/")
        // some code here

        then:
        // meaningful assertions
    }

}

Artifact upload process is performed in the background, so it will not affect test execution. The uploaded artifacts will appear under the test or test run name in the run results in Zebrunner.

Syncing test executions with external TCM systems

Zebrunner provides an ability to upload test results to external test case management systems (TCMs) on the test run finish. For some TCMs, it is possible to upload the results in real time during the test run execution.

This functionality is currently supported for Zebrunner Test Case Management (TCM), TestRail, Xray, Zephyr Squad and Zephyr Scale.

Zebrunner Test Case Management (TCM)

Zebrunner agent has a special TestCase class with a bunch of methods to control the results upload:

#setTestRunId(String)

Mandatory. The method sets the Zebrunner test run id for the current automation test run. This method must be invoked before all tests. Thus, it should be invoked from #setupSpec() method. If your specification is composed of multiple specifications, you should invoke this method only for the first specification

#setTestCaseKey(String) or @TestCaseKey(array of Strings)

Mandatory. Using these mechanisms, you can set Zebrunner's case associated with a specific automated test. It is highly recommended to use the @TestCaseKey annotation instead of static method invocation. Use the static method only for special cases

#disableSync()

Optional. Disables the result upload. Same as #setTestRunId(String), this method must be invoked before all tests

#enableRealTimeSync()

Optional. Enables real-time results upload. In this mode, the result of the test execution will be uploaded immediately after the feature finish. Same as #setTestRunId(String), this method must be invoked before all feature methods

Example

In the example below, a new run with the name "Best run ever" will be created in Zebrunner Test Case Management on the test run finish. The run id is 1. The results of the awesomeTest1 will be uploaded as the results of the cases with key DEF-1, DEF-2, DEF-3. The results of awesomeTest2 will be uploaded as the results of the case with key DEF-4.

import com.zebrunner.agent.core.annotation.TestCaseKey
import com.zebrunner.agent.core.registrar.TestCase
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    def setupSpec() {
        TestCase.setTestRunId("1")
    }

    @TestCaseKey("DEF-1")
    @TestCaseKey(["DEF-2", "DEF-3"])
    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

    def "another awesome feature"() {
        when:
        TestCase.setTestCaseKey("DEF-4")
        // some code here

        then:
        // meaningful assertions
    }

}

TestRail

For successful upload of test run results in TestRail, two steps should be performed:

1. Integration with TestRail is configured and enabled for the Zebrunner project
2. Configuration is performed on the tests side

Configuration

Zebrunner agent has a special TestRail class with a bunch of methods to control the results upload:

#setSuiteId(String)

Mandatory. The method sets TestRail suite id for the current test run. This method must be invoked before all tests. Thus, it should be invoked from the #setupSpec() method. If your specification is composed of multiple specifications, you should invoke this method only for the first specification

#setCaseId(String) or @TestRailCaseId(array of Strings)

Mandatory. Using these mechanisms, you can set TestRail's case associated with a specific automated test. It is highly recommended to use the @TestRailCaseId annotation instead of static method invocation. Use the static method only for special cases

#disableSync()

Optional. Disables the result upload. Same as #setSuiteId(String), this method must be invoked before all tests

#includeAllTestCasesInNewRun()

Optional. Includes all cases from the suite into a newly created run in TestRail. Same as #setSuiteId(String), this method must be invoked before all tests

#enableRealTimeSync()

Optional. Enables real-time results upload. In this mode, the result of the test execution will be uploaded immediately after the test finish. This method also automatically invokes #includeAllTestCasesInNewRun(). Same as #setSuiteId(String), this method must be invoked before all tests

#setRunId(String)

Optional. Adds result into an existing TestRail run. If not provided, the test run is treated as new. Same as #setSuiteId(String), this method must be invoked before all tests

#setRunName(String)

Optional. Sets custom name for a new TestRail run. By default, Zebrunner test run name is used. Same as #setSuiteId(String), this method must be invoked before all tests

#setMilestone(String)

Optional. Adds result in a TestRail milestone with the given name. Same as #setSuiteId(String), this method must be invoked before all tests

#setAssignee(String)

Optional. Sets TestRail run assignee. Same as #setSuiteId(String), this method must be invoked before all tests

By default, a new run containing only cases assigned to the tests will be created in TestRail on the test run finish.

Example

In the example below, a new run with the name "Best run ever" will be created in TestRail on the test run finish. The suite id is 321 and the assignee is "Deve Loper". The results of the awesomeTest1 will be uploaded as the results of the cases with id 10000, 10001, 10002. The results of the awesomeTest2 will be uploaded as the results of the case with id 20000.

import com.zebrunner.agent.core.annotation.TestRailCaseId
import com.zebrunner.agent.core.registrar.TestRail
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    def setupSpec() {
        TestRail.setSuiteId("321")
        TestRail.setRunName("Best run ever")
        TestRail.setAssignee("Deve Loper")
    }

    @TestRailCaseId("10000")
    @TestRailCaseId(["10001", "10002"])
    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

    def "another awesome feature"() {
        when:
        TestRail.setTestCaseId("20000")
        // some code here

        then:
        // meaningful assertions
    }

}

Xray

For successful upload of test run results in Xray, two steps should be performed:

1. Xray integration is configured and enabled in the Zebrunner project
2. Xray configuration is performed on the tests side

Configuration

Zebrunner agent has a special Xray class with a bunch of methods to control the results upload:

#setExecutionKey(String)

Mandatory. The method sets Xray execution key. This method must be invoked before all tests. Thus, it should be invoked from the #setupSpec() method. If your specification is composed of multiple specifications, you should invoke this method only for the first specification

#setTestKey(String) or @XrayTestKey(array of Strings)

Mandatory. Using these mechanisms, you can set test keys associated with a specific automated test. It is highly recommended to use the @XrayTestKey annotation instead of static method invocation. Use the static method only for special cases

#disableSync()

Optional. Disables the result upload. Same as #setExecutionKey(String), this method must be invoked before all tests

#enableRealTimeSync()

Optional. Enables real-time results upload. In this mode, the result of the test execution will be uploaded immediately after the test finish. Same as #setExecutionKey(String), this method must be invoked before all tests

By default, the results will be uploaded to Xray on the test run finish.

Example

In the example below, the results will be uploaded to the execution with key ZBR-42. The results of the awesomeTest1 will be uploaded as the results of the tests with key ZBR-10000, ZBR-10001, ZBR-10002. The results of the awesomeTest2 will be uploaded as the results of the test with key ZBR-20000.

import com.zebrunner.agent.core.annotation.XrayTestKey
import com.zebrunner.agent.core.registrar.Xray
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    def setupSpec() {
        Xray.setExecutionKey("ZBR-42")
    }

    @XrayTestKey("ZBR-10000")
    @XrayTestKey(["ZBR-10001", "ZBR-10002"])
    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

    def "another awesome feature"() {
        when:
        Xray.setTestKey("ZBR-20000")
        // some code here

        then:
        // meaningful assertions
    }

}

Zephyr

For successful upload of test run results in Zephyr, two steps should be performed:

1. Zephyr integration is configured and enabled in the Zebrunner project
2. Zephyr configuration is performed on the tests side

The described steps work for both Zephyr Squad and Zephyr Scale.

Configuration

Zebrunner agent has a special Zephyr class with a bunch of methods to control the results upload:

#setTestCycleKey(String)

Mandatory. The method sets the Zephyr test cycle key. This method must be invoked before all tests. Thus, it should be invoked from #setupSpec() method. If your specification is composed of multiple specifications, you should invoke this method only for the first specification

#setJiraProjectKey(String)

Mandatory. Sets the Zephyr Jira project key. Same as #setTestCycleKey(String), this method must be invoked before all tests

#setTestCaseKey(String) or @ZephyrTestCaseKey(array of Strings)

Mandatory. Using these mechanisms you can set test case keys associated with a specific automated test. It is highly recommended to use the @ZephyrTestCaseKey annotation instead of static method invocation. Use the static method only for special cases

#disableSync()

Optional. Disables the result upload. Same as #setTestCycleKey(String), this method must be invoked before all tests

#enableRealTimeSync()

Optional. Enables real-time results upload. In this mode, the result of the test execution will be uploaded immediately after the test finish. Same as #setTestCycleKey(String), this method must be invoked before all tests

By default, the results will be uploaded to Zephyr on the test run finish.

Example

In the example below, the results will be uploaded to the test cycle with key ZBR-R42 from the project with key ZBR. The results of the awesomeTest1 will be uploaded as the results of the tests with key ZBR-T10000, ZBR-T10001, ZBR-T10002. The results of the awesomeTest2 will be uploaded as the result of the test with key ZBR-T20000.

import com.zebrunner.agent.core.annotation.ZephyrTestCaseKey
import com.zebrunner.agent.core.registrar.Zephyr
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    def setupSpec() {
        Zephyr.setTestCycleKey("ZBR-R42")
        Zephyr.setJiraProjectKey("ZBR")
    }

    @ZephyrTestCaseKey("ZBR-T10000")
    @ZephyrTestCaseKey(["ZBR-T10001", "ZBR-T10002"])
    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

    def "another awesome feature"() {
        when:
        Zephyr.setTestCaseKey("ZBR-T20000")
        // some code here

        then:
        // meaningful assertions
    }

}

Setting execution result of a test case

Zebrunner allows pushing test execution results to external TCM systems. Test cases in these TCMs must have a result status.

By default, Zebrunner tries to derive the most suitable status in the target TCM system. Most of the time, this works correctly. Sometimes, it becomes necessary to have control over the status of a test case in the TCM. This can be extremely important when you are using custom result statuses.

For such cases, Zebrunner Agent provides two capabilities:

• automatically set a specific result status for passed, failed or skipped tests
• set specific result status for a test case directly from the code of the automated test

Automatic setting of a result status can be performed using:

• Configuration properties (env vars, program arguments, yaml file, properties file)
• TestCaseStatusOnPass, TestCaseStatusOnFail or TestCaseStatusOnSkip annotation at the class level
• TestCaseStatusOnPass, TestCaseStatusOnFail or TestCaseStatusOnSkip annotation at the method level

If you specify several of the options at the same time, the narrowest one will take precedence. Put differently, a method level annotation takes precedence over a class level annotation and configuration properties a class level annotation takes precedence only over configuration properties.

In order to set a specific result status for a test case, you can use one of the following static methods:

TestCase#setTestCaseStatus(String testCaseKey, String resultStatus)
TestRail#setTestCaseStatus(String testCaseId, String resultStatus)
Xray#setTestStatus(String testKey, String resultStatus)
Zephyr#setTestCaseStatus(String testCaseKey, String resultStatus)

The TestCase, TestRail, Xray and Zephyr classes come with nested classes with constants for system result statuses. For more details, take a look at the TestCase.SystemTestCaseStatus, TestRail.SystemTestCaseStatus, Xray.SystemTestStatus, Zephyr.Scale.SystemTestCaseStatus and Zephyr.Squad.SystemTestCaseStatus classes.

Note. In case you need to set a custom status for a TestRail case, contact your TestRail administrator to get the correct system name for the desired status.

Example 1 — Automatically set specific result status

Let's say you have the following configuration in your agent.yaml file

reporting:
  # some configuration
  run:
    test-case-status:
      on-pass: passed
      on-fail: blocked
      on-skip: not-tested

Class with an automated test looks like the following

import com.zebrunner.agent.core.annotation.TestCaseStatusOnFail
import com.zebrunner.agent.core.annotation.TestCaseStatusOnSkip
import com.zebrunner.agent.core.annotation.TestRailCaseId
import com.zebrunner.agent.core.registrar.TestRail
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    @TestRailCaseId("10000")
    @TestCaseStatusOnFail(TestRail.SystemTestCaseStatus.FAILED)
    def "my awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

    @TestRailCaseId("20000")
    @TestCaseStatusOnSkip(TestRail.SystemTestCaseStatus.RETEST)
    def "another awesome feature"() {
        when:
        // some code here

        then:
        // meaningful assertions
    }

}

If awesomeTest1 is passed, it will have the result status green (from a class level annotation).

If awesomeTest1 is failed, it will have the result status failed (from a method level annotation).

If awesomeTest1 is skipped, it will have the result status not-tested (from a yaml file).

If awesomeTest2 is passed, it will have the result status green (from a class level annotation).

If awesomeTest2 is failed, it will have the result status blocked (from a yaml file).

If awesomeTest2 is skipped, it will have the result status retest (from a method level annotation).

Example 2 — Set specific result status for a test case

Let's say, a class with an automated test looks like the following

import com.zebrunner.agent.core.annotation.TestCaseStatusOnFail
import com.zebrunner.agent.core.annotation.TestCaseStatusOnSkip
import com.zebrunner.agent.core.annotation.TestRailCaseId
import com.zebrunner.agent.core.registrar.TestRail
import spock.lang.Specification

class AwesomeSpecification extends Specification {

    @TestRailCaseId(["10000", "10001"])
    @TestCaseStatusOnFail(TestRail.SystemTestCaseStatus.FAILED)
    def "my awesome feature"() {
        when:
        TestRail.setTestCaseStatus("10000", TestRail.SystemTestCaseStatus.RETEST)   // line A
        // some code here

        then:
        // meaningful assertions
    }

    @TestCaseStatusOnSkip(TestRail.SystemTestCaseStatus.RETEST)
    def "another awesome feature"() {
        when:
        TestRail.setTestCaseStatus("20000", TestRail.SystemTestCaseStatus.BLOCKED)   // line B
        // some code here

        then:
        // meaningful assertions
    }

}

If awesomeTest1 is executed with any result (passed, failed or skipped) and line A is also executed, then the result status retest will be set for test case 10000 regardless of TestCaseStatusOn* annotations and configuration.

If awesomeTest1 is executed and failed, and the result status of test case 10001 was not set within the method using the TestRail#setTestCaseStatus method, then the result status failed will be set for test case 10001.

If awesomeTest2 is executed any result (passed, failed or skipped) and line B is also executed, then the result status blocked will be set for test case 20000. Notice that there are no TestCaseStatusOn* annotations at class and method levels.

Selenium WebDriver support

The Zebrunner test agent is capable of tracking tests along with remote Selenium WebDriver sessions. After providing additional configuration, the agent captures all events of RemoteDriverSession instances (or instances of its subclasses) and reports them to Zebrunner.

Capturing sessions

Firstly, add a byte-buddy dependency with version 1.12.10 or higher to your project. It is required to proxy method invocations of RemoteDriverSession in order to track session start and end events and their metadata. Your project classpath may already contain byte-buddy dependency (either declared explicitly or fetched transitively), but it is crucial to make sure that its version is 1.12.10 or higher in order to guarantee proper work with JDK11+.

GradleMaven

dependencies {
    // other project dependencies

    testImplementation("net.bytebuddy:byte-buddy:1.12.10")

    // other project dependencies
}

<dependencies>
    <!-- other project dependencies -->

    <dependency>
        <groupId>net.bytebuddy</groupId>
        <artifactId>byte-buddy</artifactId>
        <version>1.12.10</version>
        <scope>test</scope>
    </dependency>

    <!-- other project dependencies -->
</dependencies>

Secondly, you need to add a VM argument referencing the core Zebrunner agent jar file. This can be done in several ways: using a build tool (Maven or Gradle) or directly from the IDE.

Option 1: Run with Gradle

Gradle provides support for adding a VM argument out of the box. The only thing you need to do is to add the jvmArgs property to the test task. The value of this property must point to the local path to the Zebrunner agent.

The following code snippet shows the content of the build.gradle file.

def coreAgentArtifact = configurations.testRuntimeClasspath.resolvedConfiguration.resolvedArtifacts.find { it.name == 'agent-core' }
test.doFirst {
    jvmArgs "-javaagent:${coreAgentArtifact.file}"
}

Option 2: Run with Maven

There are two ways here: Dependency or AntRun plugin configuration.

Dependency plugin

The maven-surefire-plugin provides the ability to add VM arguments in a convenient way. You only need to provide the absolute path to the jar file with the Zebrunner agent.

The maven-dependency-plugin can be used to obtain the absolute path to a project's dependency. The properties goal of this plugin supplies a set of properties with paths to all project dependencies. If your project is already using the maven-dependency-plugin, this is the best way to go.

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>3.1.2</version>
        <executions>
            <execution>
                <goals>
                    <goal>properties</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M4</version>
        <configuration>
            <argLine>-javaagent:${com.zebrunner:agent-core:jar}</argLine>
        </configuration>
    </plugin>
</plugins>

The ${com.zebrunner:agent-core:jar} property is generated by maven-dependency-plugin during the initialization phase. Maven automatically sets the generated value when maven-surefire-plugin launches tests.

AntRun plugin

The maven-surefire-plugin provides the ability to add VM arguments in a convenient way. You only need to provide the absolute path to the jar file with the Zebrunner agent.

The maven-antrun-plugin can be used to obtain the absolute path to a project dependency. If your project is already using the maven-antrun-plugin, this is the best way to go.

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-antrun-plugin</artifactId>
        <version>1.8</version>
        <executions>
            <execution>
                <phase>initialize</phase>
                <configuration>
                    <exportAntProperties>true</exportAntProperties>
                    <tasks>
                        <basename file="${maven.dependency.com.zebrunner.agent-core.jar.path}"
                                  property="com.zebrunner:agent-core:jar"/>
                    </tasks>
                </configuration>
                <goals>
                    <goal>run</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M4</version>
        <configuration>
            <argLine>-javaagent:${com.zebrunner:agent-core:jar}</argLine>
        </configuration>
    </plugin>
</plugins>

The ${com.zebrunner:agent-core:jar} property is generated by the maven-antrun-plugin during the initialization phase. Maven automatically sets the generated value when maven-surefire-plugin launches tests.

Option 3: Run with IDE

The most modern IDEs provide the ability to run tests locally along with specifying environment and/or VM arguments. This allows to run tests locally on dev machines and report the results to Zebrunner.

We strongly encourage you to delegate test execution to a build tool rather than to run tests locally using IDE support along with Zebrunner agent.

To add a VM argument to run via the IDE, open the run configuration for tests. Then find the VM arguments setting and append the following line to the property value.

-javaagent:<path-to-core-agent-jar>

The value you append to VM arguments setting must contain a valid path to your local agent-core jar file. In most cases, this jar has been downloaded by your IDE and saved in the local repository of your build tool.

With Maven, this path should have the following pattern: <path-to-user-folder>/.m2/repository/com/zebrunner/agent-core/<agent-core-version>/agent-core-<agent-core-version>.jar.

With Gradle, this path should have the following pattern: <path-to-user-folder>/.gradle/caches/modules-2/files-2.1/com.zebrunner/agent-core/<agent-core-version>/agent-core-<agent-core-version>.jar.

IDE with Gradle

If a test project is imported into IDE as Gradle project, it should be enough to apply normal Gradle configuration for the Zebrunner agent.

Just add the following config to the build.gradle file.

def coreAgentArtifact = configurations.testRuntimeClasspath.resolvedConfiguration.resolvedArtifacts.find { it.name == 'agent-core' }
test.doFirst {
    jvmArgs "-javaagent:${coreAgentArtifact.file}"
}

IDE with Maven

If a test project is imported into IDE as Maven project, some routine configuration is required to run tests using IDE support.

The proposed solution is based on an approach with dependency plugin, but with additional configuration.

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>3.1.2</version>
        <executions>
            <execution>
                <id>copy</id>
                <phase>initialize</phase>
                <goals>
                    <goal>copy</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <artifactItems>
                <artifactItem>
                    <groupId>com.zebrunner</groupId>
                    <artifactId>agent-core</artifactId>
                    <version>RELEASE</version>
                    <outputDirectory>${project.build.directory}/agent</outputDirectory>
                    <destFileName>zebrunner-core-agent.jar</destFileName>
                </artifactItem>
            </artifactItems>
        </configuration>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M4</version>
        <configuration>
            <argLine>-javaagent:${project.build.directory}/agent/zebrunner-core-agent.jar</argLine>
        </configuration>
    </plugin>
</plugins>

The maven-dependency-plugin from preceding code snippet copies the jar file with the core Zebrunner agent to the project build directory during initialization phase. The maven-surefire-plugin then uses the copied version of the Zebrunner agent as java instrumentation agent.

In some cases, it may not be enough to simply apply such a configuration, but you also need to manually start the initialization phase at first.

Once the configuration is in place the agent will automatically report to Zebrunner WebDriver session events. However, it is also possible to collect additional test session artifacts in order to improve overall reporting experience.

Capturing session artifacts

Zebrunner supports 3 types of test session artifacts:

• Video recording
• Session log
• VNC streaming

Test agent itself does not capture those artifacts since it has no control over underlying Selenium Grid implementation, however, it is possible to attach appropriate artifact references by providing specially designed set of driver session capabilities (enabling capabilities) — see the table below for more details. Only the true value is considered as trigger to save the link.

Artifact	Display name	Enabling capability	Default reference	Reference overriding capability
Video recording	Video	enableVideo	artifacts/test-sessions/<session-id>/video.mp4	videoLink
Session log	Log	enableLog	artifacts/test-sessions/<session-id>/session.log	logLink
VNC streaming		enableVNC	<provider-integration-host>/ws/vnc/<session-id>	vncLink

The display name is the name of the artifact that will be displayed on Zebrunner UI. This value is predefined and unfortunately can not be changed at the moment.

The default reference is a reference to a location, where artifact is expected to reside in S3-compatible storage once created by test environment — it is important that it stays in sync with test environment configuration. It is possible to override references if needed by providing reference overriding capabilities. Note, that special <session-id> placeholder is supported and can be used as part of the value of those capabilities allowing runtime session id (generated by WebDriver) to be included into actual reference value.

VNC streaming

VNC is an artifact of a special type. They don't have a name and are not displayed among other artifacts. They are displayed in the video section on Zebrunner UI during session execution and are dropped off on session close.

Default reference to the VNC streaming is based on provider capability. Value of this capability will be converted to preconfigured integration from Test Environment Provider group. The resolved integration must have a filled in URL property and be enabled in order to save the link to VNC streaming. The <provider-integration-host> placeholder of the default link will be replaced by the host of the obtained integration URL. Also, the http protocol in the VNC streaming url will be automatically replaced by ws, and https protocol will be replaced by wss. Currently, we only support Zebrunner Selenium Grid and Zebrunner Device Farm integrations.

Examples

The following code snippet shows the creation of RemoteWebDriver with enabled VNC streaming, video recording and session log artifacts, but overrides the link to VNC streaming. The <session-id> placeholder of vncLink capability will be replaced by the actual session id.

import org.openqa.selenium.chrome.ChromeOptions
import org.openqa.selenium.remote.RemoteWebDriver

class WebDriverManager {

    RemoteWebDriver initWebDriver() {
        ChromeOptions capabilities = new ChromeOptions()
        capabilities.setCapability("enableVNC", "true")
        capabilities.setCapability("vncLink", "wss://example.com/vnc/<session-id>")
        capabilities.setCapability("enableVideo", "true")
        capabilities.setCapability("enableLog", "true")

        return new RemoteWebDriver(new URL("https://user:pass@example.com/wd/hub"), capabilities)
    }

}

The following code snippet shows the creation of RemoteWebDriver with enabled video and session log artifacts, overridden link to VNC streaming and session log, but disabled VNC streaming. As a result, VNC streaming will not be available, video recording will have default link with populated placeholders, and session log will have custom link with populated <session-id> placeholder.

import org.openqa.selenium.chrome.ChromeOptions
import org.openqa.selenium.remote.RemoteWebDriver

class WebDriverManager {

    RemoteWebDriver initWebDriver() {
        ChromeOptions capabilities = new ChromeOptions()
        capabilities.setCapability("enableVNC", "false")
        capabilities.setCapability("vncLink", "wss://example.com/vnc/<session-id>")
        capabilities.setCapability("enableVideo", "true")
        capabilities.setCapability("enableLog", "true")
        capabilities.setCapability("logLink", "https://example.com/driver-sessions/logs/<session-id>.log")

        return new RemoteWebDriver(new URL("https://user:pass@example.com/wd/hub"), capabilities)
    }

}

Contribution

To check out the project and build from the source, do the following:

git clone git://github.com/zebrunner/java-agent-spock.git
cd java-agent-spock
./gradlew build

License

Zebrunner reporting agent for Spock is released under version 2.0 of the Apache License.