User Tools

Manage Health Status

Health is defined as the current condition or state of a software build associated with a particular Space. A healthy Space is without build issues, test failures (unless exempt) and failing indicators such as thresholds and criteria for code coverage, static analysis or other custom metrics.

Space health combines the results from your build, test, coverage and analysis into single indicator:

Healthy Healthy 0 nonexempt test failures and with all metric criteria met
Unhealthy Unhealthy 1 or more nonexempt test failures, or unmet metric criteria
Invalid Invalid A result set missing required data. See explanation below.

A result set will be declared invalid based on the following conditions:

  • Test case count = 0
  • Metric used in the Health status is missing
  • Five percent or greater drop in the test suite count from the previous result set (the suite count threshold). Note, this is only checked if the case count has dropped by at least 1. In this case all subsequent results will be marked invalid until:
    • Suite count is restored
    • Five results have been published without Suite count restoration. At this point, a new suite count threshold will be established from the highest suite count from the 5 results.

Exempt Failing Test Cases

Test exemptions allow you to exclude failing cases from the calculation of health. Exemptions are persisted and are applied to each subsequent result set until removed. Removal can be done manually or will be done automatically after the 5th consecutive passing result. Exempting a test case does not hide or alter its passed|failed status.

Exemptions are managed from the Space Schema tab in the test suite dialog as shown below.

Descriptions of numbered areas:

  1. Non-exempt failing test case
  2. Exempt failing test case with a link to a repo issue for tracking.

To exempt a failing test case, click on the exemption icon (circled exclamation) next to the passed|failed status to open the exemption dialog. Enter a reason to enable an exemption. Use markdown syntax to add links to your defect and issue tracking systems. Clear the dialog text to remove it.

Exemptions are removed automatically after the 5th consecutive passing result.

Tracking Exemptions

Project Notes are used to track exemptions for all Spaces under the project. A new note is automatically created when an exemption is enabled. Each exemption enabled for that result set is tracked by a comment added to the note. An additional comment is added for each exemption when removed. The total number of exempt test cases is summarized in the results notification email.

Current exemption totals for each suite are shown in the Schema view.

Exemption history is maintained with each result set on the Results tab.

Adjust Code Coverage Thresholds

Code coverage thresholds allow you to set minimum levels of code execution for a Space to remain healthy. A default threshold of 50% is assigned to all coverage Metrics at the time of Metric creation, which occurs the first time you upload a coverage file to a Space.

To change the current Threshold, select the Space Metrics tab and click on the options icon (hamburger) to the left of the Metric graph name and select edit.1)

Update the Min threshold value in the edit Metric dialog as show below and SUBMIT your change.

Setup Static Analysis ​Criteria ​

Static analysis, in addition to more traditional dynamic testing, can help to identify defects or bugs (issues) early and throughout the development process. Although informational by default, setting ​Pass/​Fail​ criteria for Analysis Metrics can prevent new issues from going unnoticed. The level of issue severity (high, medium, low) for reporting or failing Health can be configured as well.

To setup analysis criteria, select the Metrics tab in the Space of interest and click on the options icon (hamburger) to the left of the Metric graph name, then select edit.2)

Update the Health and reporting criteria in the edit Metric dialog as show below and ​SUBMIT​ your changes.

Invalid Result Sets

Results sets are marked invalid (1) for the following reasons:

  • The data for a Metric that has a threshold criteria set is missing.
  • A drop in the number of Test Suites has occurred.

Results due to drop in the number of Test Suites will be marked invalid until either the number of Test Suites recovers or after five results a new baseline will be set.

How To

1) , 2) For additional editing information see Metrics Tab.

Page Tools