Skip to main content

Failures

When test failures occur, they can be deeply nested and widely dispersed throughout a Result hierarchy, as shown in the example below.

Results Failures

When triaging test failures, the type of failure pattern can help in addressing the problem.

  • Is the failure new, or did it occur in the previous or other recent builds?
  • Is the case intermittently (aka flaky) or consistently failing?
  • How can one tell when failures have been resolved?

Testspace automatically tracks test cases that continue to fail over subsequent Results and are reported in one of the following states:

  1. New – the first time a test fails and is not being tracked.
  2. Consistent - a test case that is consistently failing.
  3. Passing - a test case passing that is being tracked.
  4. Flaky - a test case that is oscillating between passing or failing.
  5. Resolved – a test case passing 5 times.

Test Failure Filtering enables you to find and triage failures quickly.

info

Filtering organizes content by the following groups: Suite name, Failure state, Failure cause

Filtering​

The N Failures filter button is shown (only when failures occur) in both the Current and Results tabs, and at every level when traversing the results hierarchy. The Failures tracking (column 4) displays the number of New, Tracked, and Resolved failures.

Suite name​

Clicking on the N Failures button reduces the hierarchy of test folders and suites into a single flat view showing only the suites that contain test failures.

Failure Filtering

  • Types - suites that contain Failed/Blocked (default), Failed only, Blocked only, or all Tracked cases.
  • Filter:
    • New - failing cases that passed in the previous five consecutive results.
    • Flaky - failing cases that are not either consistently passing or failing.
    • Consistent - failing cases that are in a consistent failure state.
    • Exempt - failing cases that have been Exempt from failing the Health of the results.

The View all… link is used to disable the view.

tip

Each of the columns is sortable by clicking on the corresponding name.

Failure state​

Selecting the Failure state group organizes tracked test cases based on their state.

Failure Filtering State

The following is an example of all of the Flaky test cases associated with the Result record.

Failure Filtering Flaky State

Failure cause​

Selecting the Failure cause group organizes failing test cases based on the error message associated with the failure.

Failure Filtering Cause

The following is an example of all current test failures with a Asserted: Node does not appear based on timeout error message.

Failure Filtering Cause Suite

Test Suite​

When opening a Suite with failures, by selecting the Suite Name, only failing cases are listed by default as shown below.

Test Suite Failures

The Failures types (column 5) are tracking using the following icons and names with the most recent result as the larger icon on left :

Test Suite Failures

The View all… link at the bottom of the dialog is used to disable the view.

Exemptions​

Test exemptions allow you to exclude failing cases from the calculation of the pass-fail rate. Exemptions are persisted and are applied to each subsequent result until removed. Removal can be done manually or will be done automatically after the 5th consecutive passing result. Exempt test cases are not counted in passed|failed totals.

Exemptions can only be managed from the Current Result view.

To exempt a failing test case, only while on the Current tab, click on the exclamation (!) button in the column. From this dialog, you can add a Note for tracking (discussed in the section below) and also select the Exempt check-box.

Test Suite Exemption

Exemptions are removed automatically after the 5th consecutive passing result.

Exempt test cases are identified with a yellow circled exclamation symbol (refer to 4 below).

Test Suite Exemption View

The passed|failed state is still shown, but grayed-out and strike-through. The existence of exempt test cases is identified – next to the passed|failed totals in all views, however, its status is not added to totals. Notes that have been added by the triage dialog are displayed under the case name.

Notes​

Project Notes are used to track Exemptions and Triage Notes for all Spaces under the Project. A new note is automatically created when an exemption is enabled. Each Exemption and/or Triage Note enabled for that Result is tracked by an additional comment added to the note. An additional comment is added for each Exemption when removed. The total number of exempt test cases is summarized in the results notification email.