User Tools

Manage Health Status

Health is defined as the current condition or state of a software build associated with a particular Space. A healthy Space is without build issues and failing indicators such as criteria for passing percentage, code coverage, static analysis or other custom metrics.

Space health combines all of the above quality metrics into single indicator:

Healthy Healthy When the Test Case Min. pass threshold % and all metric criteria have been met
Unhealthy Unhealthy When any metric criteria – standard or custom – have not be met.
Invalid Invalid A result set missing required data. See explanation below.

A result set will be declared invalid based on the following conditions:

  • Test case count = 0
  • Metric used in the Health status is missing
  • Five percent or greater drop in the test suite count from the previous result set (the suite count threshold). Note, this is only checked if the case count has dropped by at least 1. In this case all subsequent results will be marked invalid until:
    • Suite count is restored
    • Five results have been published without Suite count restoration. At this point, a new suite count threshold will be established from the highest suite count from the 5 results.

Testspace provides two methods of managing the effect of Test Failures on results health:

  1. Changing the Minimum Pass Percentage Threshold - irrespective of which cases are failing.
  2. Exempting Failing Test Cases - to exempt specific test cases from the calculation of health.

Changing the Minimum Pass Percentage Threshold

The Minimum Pass Percentage Threshold is managed on the Test Cases Metric Graph

There may be times during development when dependencies on other components, contributors or hardware can prevent a number of test cases from passing.

Lowering the minimum pass percentage – default is 100% – allows a Branch/Space (depending on Project Type) to remain healthy even with the existence of test failures.

Exempt Failing Test Cases

Test exemptions allow you to exclude failing cases from the calculation of health. Exemptions are persisted and are applied to each subsequent result set until removed. Removal can be done manually or will be done automatically after the 5th consecutive passing result. Exempting a test case does not hide or alter its passed|failed status.

Exemptions are managed from the Space Schema tab in the test suite dialog as shown below.

Descriptions of numbered areas:

  1. Non-exempt failing test case
  2. Exempt failing test case with a link to a repo issue for tracking.

To exempt a failing test case, click on the exemption icon (circled exclamation) next to the passed|failed status to open the exemption dialog. Enter a reason to enable an exemption. Use markdown syntax to add links to your defect and issue tracking systems. Clear the dialog text to remove it.

Exemptions are removed automatically after the 5th consecutive passing result.

Tracking Exemptions

Project Notes are used to track exemptions for all Spaces under the project. A new note is automatically created when an exemption is enabled. Each exemption enabled for that result set is tracked by a comment added to the note. An additional comment is added for each exemption when removed. The total number of exempt test cases is summarized in the results notification email.

Current exemption totals for each suite are shown in the Schema view.

Exemption history is maintained with each result set on the Results tab.

Adjust Code Coverage Thresholds

Code coverage thresholds allow you to set minimum levels of code execution for a Space to remain healthy. A default threshold of 50% is assigned to all coverage Metrics at the time of Metric creation, which occurs the first time you upload a coverage file to a Space.

To change the current Threshold, select the Space Metrics tab and click on the options icon (hamburger) to the left of the Metric graph name and select edit.1)

Update the Min threshold value in the edit Metric dialog as shown below and SUBMIT your change.

Setup Static Analysis ​Criteria ​

Static analysis, in addition to more traditional dynamic testing, can help to identify defects or bugs (issues) early and throughout the development process. Although informational by default, setting ​Pass/​Fail​ criteria for Analysis Metrics can prevent new issues from going unnoticed. The level of issue severity (high, medium, low) for reporting or failing Health can be configured as well.

To setup analysis criteria, select the Metrics tab in the Space of interest and click on the options icon (hamburger) to the left of the Metric graph name, then select edit.2)

Update the Health and reporting criteria in the edit Metric dialog as shown below and ​SUBMIT​ your changes.

Invalid Result Sets

Results sets are marked invalid (1) for the following reasons:

  • The data for a Metric that has a threshold criteria set is missing.
  • A drop in the number of Test Suites has occurred.

Results due to drop in the number of Test Suites will be marked invalid until either the number of Test Suites recovers or after five results a new baseline will be set.

How To

1) , 2) For additional editing information see Standard Metrics.

Page Tools