When test failures occur, they can be deeply nested and widely dispersed throughout a Result hierarchy, as shown in the example below.
When triaging test failures, the type of failure pattern can help in addressing the problem.
- Is the failure new, or did it occur in the previous or other recent builds?
- Is the case intermittently (aka flaky) or consistently failing?
- How can one tell when failures have been resolved?
Testspace automatically tracks test cases that continue to fail over subsequent Results and are reported in one of the following states:
New– the first time a test fails and is not being tracked.
Consistent- a test case that is consistently failing.
Passing- a test case passing that is being tracked.
Flaky- a test case that is oscillating between passing or failing.
Resolved– a test case passing 5 times.
Test Failure Filtering enables you to find and triage failures quickly.
Filtering organizes content by the following groups: Suite name, Failure state, Failure cause
N Failures filter button is shown (only when failures occur) in both the
Results tabs, and at every level when traversing the results hierarchy. The
Failures tracking (column 4) displays the number of
Clicking on the
N Failures button reduces the hierarchy of test folders and suites into a single flat view showing only the suites that contain test failures.
Types- suites that contain Failed/Errored (default), Failed ony, Errored only, or all Tracked cases.
- New - failing cases that passed in the previous five consecutive results.
- Flaky - failing cases that are not either consistently passing or failing.
- Consistent - failing cases that are in a consistent failure state.
- Exempt - failing cases that have been Exempt from failing the Health of the results.
View all… link is used to disable the view.
Each of the columns is sortable by clicking on the corresponding name.
Failure state group organizes tracked test cases based on their state.
The following is an example of all of the
Flaky test cases associated with the Result record.
Failure cause group organizes failing test cases based on the
error message associated with the failure.
The following is an example of all current test failures with a
Asserted: Node does not appear based on timeout error message.
When opening a Suite with failures, by selecting the Suite Name, only failing cases are listed by default as shown below.
The Failures types (column 5) are tracking using the following icons and names with the most recent result as the larger icon on left :
View all… link at the bottom of the dialog is used to disable the view.
Test exemptions allow you to exclude failing cases from the calculation of the pass-fail rate. Exemptions are persisted and are applied to each subsequent result until removed. Removal can be done manually or will be done automatically after the 5th consecutive passing result. Exempt test cases are not counted in
Exemptions can only be managed from the Current Result view.
To exempt a failing test case, only while on the Current tab, click on the exclamation
(!) button in the column. From this dialog, you can add a Note for tracking (discussed in the section below) and also select the Exempt
Exemptions are removed automatically after the 5th consecutive passing result.
Exempt test cases are identified with a yellow circled exclamation symbol (refer to 4 below).
passed|failed state is still shown, but grayed-out and strike-through. The existence of exempt test cases is identified – next to the
passed|failed totals in all views, however, its status is not added to totals. Notes that have been added by the triage dialog are displayed under the case name.
Project Notes are used to track Exemptions and Triage Notes for all Spaces under the Project. A new note is automatically created when an exemption is enabled. Each Exemption and/or Triage Note enabled for that Result is tracked by an additional comment added to the note. An additional comment is added for each Exemption when removed. The total number of exempt test cases is summarized in the results notification email.