Skip to main content

Execution

The Manual tab lists the test specs discovered in the repository for a specific branch. A spec can be executed simply by clicking on it and hitting the START button.

The following listing is from the hello manual repo's main branch.

Specs Listing

tip

To quickly run a single spec just click on it and hit START.

Executing a spec outside a formally defined session is called standalone execution.

Run Spec

A spec is selected to execute as a standalone or within a predefined session. The test cases are defined within the spec and are provided status individually while running.

info

A spec is selected to execute as a standalone or within a predefined session.

Start

To run the spec, open the file and press the START button to begin execution. The left panel represents the set of cases associated with the spec.

Run Spec

A case defaults to Untested. The following table lists the available case status that can be assigned:

STATUSdescription
UntestedNo testing as of yet
Not applicableMarked as NA in the case
BlockedNot able to execute the test case.
PassedWorking as expected
FailedNot working as expected

Issues can be directly listed or referenced from within the comments form. For more details see further down.

The comments form supports markdown, including drag and drop images, and in-place issue references. There is also a toggle providing access to previous execution history:

The timer starts after clicking START.

Run Comments

Stop

On selecting STOP the execution, if there are any untested test cases, a dialog will be presented:

Carry Previous Status?

If left checked (default), all test cases not executed will carry their previous status.

important

By default, test cases that are not executed will carry over their previous status.

Session

A test session is used to define a set of specs and describe the purpose for the testing activity such as validating the "xyz" build. There are no limits to the number of test sessions that can coexist.

tip

Use a session to define a collection of specs for execution.

New

The listing is used to select specs for a new session:

New Session Selection

The Select spec options:

  • All used for choosing all specs
  • None used for de-selecting all specs
  • Only failing selects specs that have 1 or more failing cases
  • Never ran selects specs that have not been executed

Once the specs have been selected, click on the New Test Session button.

New Session Button

A dialog will be presented.

New Session

  • Add a Name for the session, or use the default (i.e. Sequence_n)
  • Enable or disable Exploratory
  • Add an optional Description (e.g. "testing feature ABC")
  • Select SUBMIT

When a session is created it is tagged as open, until explicitly completed. There is no limit on the number of concurrent open sessions.

Open Sessions

tip

There is no limit to the number of concurrent sessions opened.

To see the status of a specific session, highlight it using a single click.

Session highlight

Complete

A session can be completed at any time, independent of what specs have been executed. Click on the solid blue circle w/ check or use the hamburger menu.

Complete Session

  • Completing a Test Session removes the active session from the Manual tab.
  • Once a session has been completed no additional updates can be made.
note

If no Specs have been executed and the session is completed, the results record will be ignored (same behavior as deleting the session).

Groups

Test Specs can be grouped using repo source folders. When a large number of test specs are required, it is often useful to group similar tests in folders. The following repo example - https://github.com/testspace-com/demo - uses two groups: account and power:

root
└─ specs
└─ account
└─ logout.md
└─ sign-in.md
└─ sign-up.md
└─ power
└─ turn-off-power.md
└─ turn-on-power.md
└─ ..
└─ .testspace.yml
..

Testspace automatically recognizes the folders and presents them in the listing.

Listing Groups

tip

Each group will maintain its status, along with an Overall status.

The results tab will also maintain status corresponding to the grouped defined in the repo. Using the example above:

Results Groups

Assignee

Using the specs listing, a user can be pre-assigned for the execution of a spec, which will then be used as the default on all subsequent created sessions.

By clicking on the Assignee cell of the spec a dialog of available users will be presented.

Spec User Assignee

A user can also self-assign, and overwrite an existing assignment when running a spec.

Spec Changes

When a spec with existing status has its corresponding spec source file changed (i.e. via repo commit), a special orange warning icon will be displayed, indicating newer source code changes.

Listing Status with Change

note

For existing open sessions, there will be no updates or indications of changes, however, any newly created sessions will automatically pick up the changes.

Testspace tries to gracefully handle source changes and preserve execution history and failure tracking. However, renaming a test case (i.e. changing the markdown <H2> heading that denotes it) can not be handled and will result in loss of history.

Issues

GitHub issues can be associated with test cases. There are two ways - push new issues or reference existing issues.

Push New Issue

GitHub issues can be auto-generated within the test case execution. By default, on a failing test case, the Push new issue checkbox is selected.

Issues

The auto-generated Issue will insert the status in the title [Failed], along with the spec name and test case name. Comments from the test case will be added to the issue automatically as well.

The Issue is generated on the spec run STOP event.

GitHub Issue

The generated Issue will be automatically added to the ISSUES form for future reference.

Reference Existing Issue

If you want to reference an existing GitHub issue in a test case, you could either list it in the ISSUES form or simply mention it in the comments form inside any message describing your testing. In both cases, it needs to be formatted a #-prefixed Github issue number, e.g. #123.

Automated Fixtures

Specs that contains the automated fixture before, and optional after, have additional execution constraints. There will be an exclamation icon next to ! START indicating a Required action to be performed:

Automation Execution Required

Before Fixture

The before fixture is required to be successfully processed before test cases can be executed. It is automatically triggered by clicking on the blue START button. While running a gray spinning icon would appear.

In case of failure, a red cross icon would indicate unsuccessful automation. The tooltip will provide additional failure information.

Automation Failed

The before fixture has several constraints that govern the execution of the spec:

  • Closing the dialog while executing will cancel the execution
  • Closing and opening the Spec will require re-execution of the automation
  • Executing longer than 5 minutes, the user will be prompted to Continue with execution
important

For Fixtures executing longer than 5 minutes, the user will be prompted to Continue with execution.

After Fixture

The after fixture is optional and could only exist if there ia also a before fixture. If the Spec has successfully executed, once the Spec dialog is closed, the associated after fixture automation is executed, in the background, without any status available to the tester.

important

The after fixture will automatically execute when:

  • The user selects STOP or X with in the spec run dialog
  • The browser tab or computer is shutdown
  • No user activity for 10 minutes (user prompted with 60-second countdown)

Exploratory

Exploratory testing is a test flow that provides additional test coverage and usability assessment, using a non-scripted testing approach.

info

Exploratory testing is non-scripted, no-constrained observational-based testing.

There are two variants of exploratory supported by Testspace:

  • Spec based observations
  • Session based observations

Both types support pushing issues to GitHub.

Spec Observations

When running a spec additional observational content can be added. To add observational comments click on Observations, located at the bottom left of the spec dialog:

Spec Exploratory Observations

  • The ISSUES associated with observations is defaulted to Do not report

Session Observations

Session observations are created by selecting the Exploratory checkbox in the creation dialog:

New Session

Once the Session is created a built-in Exploratory session spec will be provided.

The Exploratory is a generated spec used to capture observations by one or more testers for the session.

The Exploratory session can be executed by multiple users. All of the observations will be automatically added to a GitHub Issue, named after the session.

Results

A test session suite's status, when completed, are aggregated together with all other previous sessions and reflected in the Space Results. A Space always represents the most up to date status for each suite.

Results

Aggregation

A Space presentation of results is aggregated by default. Aggregation is the inclusion of status for all Suites based on their most recent completed session.

When a session has completed, the status for the suites executed in that session are aggregated with the nonexecuted suites from previous results. This results in suites that were not executed to be carried forward.

important

Carrying forward is including suite status from previous sessions

For Suites that have been carried the Summary section of the Suite report provides context information in the Session field:

Suite Report

When non-current sessions (older sessions) are completed the status of any carried forward suites will be updated in newer completed sessions as appropriate.

Session

Results can be viewed both from an Aggregated view or a Session view. In the right corner of the page, a selection box exists: Aggregate | Session.

When Session is selected the page presents only suites executed for the session in view.

Results Session View

Issues Metric

When Issues are pushed and/or referenced during a test session, they are automatically captured in an Issues metric once the session is completed.

Issues Metric

The Issues metric suite presents a report listing of all issues' references and status at the time when the results were published:

Issues Report

The Issues metric graph is automatically generated tracking the number of issues as seen in the report.

Issues Graph

There is a badge that contains the number of open issues at the time of publishing.