Health is defined as the current condition or state of a software build associated with a particular Space. A healthy Space is without build issues and failing indicators such as criteria for
static analysis or other
Space health combines all of the above quality metrics into single indicator:
|Healthy||When the Test Case Min. pass threshold % and all metric criteria have been met|
|Unhealthy||When any metric criteria – standard or custom – have not be met.|
|Invalid||A result set missing required data. See explanation below.|
A result set will be declared invalid based on the following conditions:
Testspace provides two methods of managing the effect of Test Failures on results health:
Changing the Minimum Pass Percentage Threshold- irrespective of which cases are failing.
Exempting Failing Test Cases- to exempt specific test cases from the calculation of health.
Minimum Pass Percentage Threshold is managed on the Test Cases Metric Graph
There may be times during development when dependencies on other components, contributors or hardware can prevent a number of test cases from passing.
Lowering the minimum pass percentage – default is 100% – allows a Branch/Space (depending on Project Type) to remain healthy even with the existence of test failures.
Test exemptions allow you to exclude failing cases from the calculation of health. Exemptions are persisted and are applied to each subsequent result set until removed. Removal can be done manually or will be done automatically after the 5th consecutive passing result. Exempting a test case does not hide or alter its
Exemptions are managed from the Space Schema tab in the test suite dialog as shown below.
Descriptions of numbered areas:
To exempt a failing test case, click on the exemption icon (circled exclamation) next to the
passed|failed status to open the exemption dialog. Enter a reason to enable an exemption. Use markdown syntax to add links to your defect and issue tracking systems. Clear the dialog text to remove it.
Exemptions are removed automatically after the 5th consecutive passing result.
Project Notes are used to track exemptions for all Spaces under the project. A new note is automatically created when an exemption is enabled. Each exemption enabled for that result set is tracked by a comment added to the note. An additional comment is added for each exemption when removed. The total number of exempt test cases is summarized in the results notification email.
Current exemption totals for each suite are shown in the Schema view.
Exemption history is maintained with each result set on the Results tab.
Code coverage thresholds allow you to set minimum levels of code execution for a Space to remain healthy. A default threshold of 50% is assigned to all coverage Metrics at the time of Metric creation, which occurs the first time you upload a coverage file to a Space.
Min threshold value in the edit Metric dialog as shown below and
SUBMIT your change.
Static analysis, in addition to more traditional dynamic testing, can help to identify defects or bugs (issues) early and throughout the development process. Although informational by default, setting
Pass/Fail criteria for Analysis Metrics can prevent new issues from going unnoticed. The level of issue severity (high, medium, low) for reporting or failing Health can be configured as well.
To setup analysis criteria, select the Metrics tab in the Space of interest and click on the options icon (hamburger) to the left of the Metric graph name, then select edit.2)
Update the Health and reporting criteria in the edit Metric dialog as shown below and
SUBMIT your changes.
Results sets are marked invalid (1) for the following reasons:
Results due to drop in the number of Test Suites will be marked invalid until either the number of Test Suites recovers or after five results a new baseline will be set.