Push Results

Files – output from the Continuous Integration process – are pushed to the Testspace Server using a simple command line client; Windows, Linux, and macOS versions available. Adding the Testspace client to a CI workflow is covered in CI/CD Support.

To push content to Testspace simply provide the client a list of files and destination URL:

$ testspace file1 file2 .. my-access-token:@my-org.testspace.com/my-project/my-space

Note: The following examples are show different ways of publishing content. It is assumed you have created a Project and a Space and have appropriately configured the Testspace client's destination URL.

File Content

The following types of file content are supported, with additional information available here. How each type appears in a Result is described in Content Types.

Type	Description
Test Output	Widely adopted test framework output formats such as JUnit, NUnit, Visual Studio TRX, xUnit.net, etc.
Code Coverage	Common coverage tools output formats such as Cobertura, Clover, Bullseye, Visual Studio Coverage, etc.
Static Analysis	Large set of widely used formats such as Checkstyle, Klocwork, PMD, Visual Studio FxCop & PREfast, etc.
Custom Metrics	A CSV (.csv) file associated with a Text File or a Test Output Suite.
Text File	Plain text (.txt or .log extension), HTML (.html), and Markdown (.md) files.

Test Output

To publish test result simply push a xml output file generated by the test tool. Refer here for the test output formats supported.

$ testspace tests-results.xml

Some test frameworks produce multiple results files that you may want to aggregate into a single Testspace Result. There are several ways to accomplish this and is demonstrated with examples below.

In each of the separate following examples the result files listed will be aggregated and published as a single Result:

$ testspace myResults1.xml myResults2.xml

$ testspace myResults*.xml

$ testspace /path/to/*.xml ReleaseNotes.txt

Sub-directories

To aggregate result files from multiple sub-directories use the double star recursive search notation:

$ testspace /path/to/results/root/**/*.xml

To aggregate result files from sub-directories use the single star search notation:

$ testspace /path/*/results/root/*.xml

Process Output

For a step in your automation process (install, build, etc.) that do not have test cases associated with it, but if it fails you would like it be noted. The output log (for example) and the process return code could be used to report status. The Suite status is set by adding the process return code as an integer 'N' as metadata to a file being pushed to Testspace. Any non-zero integer will make the Suite failed.

Example of pushing a log file and its return code to Testspace.

# In this example $MYPROC is set to the return code of the process that generated myproc.log
# Publish output and exit code
$ testspace "myproc.log{$MYPROC:my description}"

Code Coverage

To publish code coverage simply push a xml output file generated by the coverage tool. Refer here for the code coverage formats supported.

$ testspace code-coverage.xml

Third Party

Testspace supports two third-party coverage services, Coveralls.io and Codecov.io, with the client's --link option. When the Result is completed, Testspace fetches the lines covered that the service reported for the commit and adds them to the Result as a Coveralls.io or Codecov.io Suite, with a link to the service's report. This creates a code coverage metric and graph; for a public Project, the metric's badge is the service's own badge. Testspace uses the minimum coverage configured in the service as the threshold (70% for Codecov and 80% for Coveralls if none is set); to change it, see Codecov configuration or Coveralls configuration.

For example:

$ testspace tests-results.xml --link=codecov

$ testspace tests-results.xml --link=coveralls

Example using a private repo:

$ testspace tests-results.xml --link="TOKEN@codecov"

Static Analysis

To publish static analysis simply push a xml output file generated by the analysis tool. Refer here for the static analysis formats supported.

$ testspace static-analysis.xml

Some compilers and SA tools report results in plain text. The lint {meta-tag} identifies the output as analysis results.

$ testspace "build-output.log{lint}"

Custom Content

Preformatted human-readable output files can be published as custom Suites. There are 4 types of output files supported .txt, .log, .md, and .html.

A simple example of publishing a log file.

$ testspace myOutput.log

External Content

Any externally hosted content can be published as a linked Suite. The URL of the external entity needs to be stored in a .url file, e.g. myLink.url:

InternetShortcut.URL=https://domain.com/whatever

Having that, it could be published as simple as:

$ testspace myLink.url

Custom Metrics

To publish your own measurements as metrics, write them to a CSV file (one row per metric: an optional name, then its values) and attach it in braces to a file you publish:

$ testspace results.xml "perf.log{perf.csv}"

Attach it to one of your test Suites with "[Suite name].{perf.csv}", or to a link to external content. Publishing the data doesn't create a graph: add it once with New Metric on the Space's Metrics tab. Custom Metrics walks through all four steps, with examples.

Annotations

The addition of supplemental data files (log files, screen shots, etc.) to your Result is easily done using the plus symbol (+) as a prefix.

Example annotating the root folder:

$ testspace myResults*.xml "+log.txt{this is my log}"

Test Session Annotations

A simple example of annotating a well known test Suite reported from your test session.

This requires knowing the name of the Suite

If your test session generates supplemental data files, e.g. screen.png, those files can be associated with a well known test Suite reported in that test session, e.g. MySuiteName:

$ testspace test-results*.xml "[MySuiteName]+/path/to/screen.png{screen shot}"

Folders

Results can be organized into folders via the client and/or the repo's source structure.

Custom

The organization of results into a folder can be done by bracket-surrounded-path prefixing the file being pushed:

$ testspace "[myFolder]myResults*.xml"

When specifying a sub-folder use the forward-slash (/) as the folder separator.

$ testspace "[myTests/mySubfolder]myResults*.xml"

Similarly, annotations could be associated with any folder:

$ testspace [myFolder]myResults*.xml "[myFolder]+/path/to/screen.png{screen shot}"

One example when using a matrix, a folder is used to store the results specific to each matrix entry:

name: CI
on: push
jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    steps:
      ..
      - name: Publish Results to Testspace
        run: testspace "[ ${{ matrix.os }} ]results.xml"

Repo Source

Using your test source directory to organize your results can significantly improve your results organization. With path/to {metadata} the Testspace client could be directed to follow the source tree organization:

$ testspace "[Folder1]myResults*.xml{path/to/test-source}"

note

Using the {path/to/test-source} report descriptor allows automatic organization of the results based on the tests source directory structure

Notice, if your source location is not in a subdirectory of the current directory you may need to specify either absolute (e.g. {/path/to/test-source}) or '..' relative (e.g. {../path/to/test-source}) path.

For example, using the following hypothetical directory structure and assuming you are running from the mydir directory:

mydir
  src
  test
    controller
      test1.php
    model
      test2.php
      test3.php
    view
      testN.php
    reports
      results1.xml
      results2.xml
      coverage.xml

The test source directory can be provided as {metadata} and the Testspace Client will organize all results based on this hierarchy.

$ testspace "[Tests]test/reports/results*.xml{test}"

Content List

Instead of listing the files to aggregate explicitly on the command line you can provide them via a "content-list" file.

For example having a content-list.txt file:

Tests.xml
Coverage.xml
Analysis.xml
Text.txt
$PATH_TO/thisfile.txt

When specifying a file containing a list of files use the at symbol (@):

$ testspace @content-list.txt

Note if a file is missing it will be captured as an error annotation in the results root folder

Incremental Pushing

You can invoke the Testspace Client multiple times and aggregate all of the results into one Result using the how parameter. This can be useful when the tests run on multiple independent environments. ?start creates an incomplete Result, ?add adds content to it, and ?finish completes it.

Give every push the same Result name (#my-results below). A push joins a Result of that name only while it is still incomplete; once that Result is finished, a push with the same name creates a new Result.

$ testspace myResults1.xml "space-name?start#my-results"
$ testspace myResults2.xml "space-name?add#my-results"
$ testspace "space-name?finish#my-results"

$ testspace "[Folder1]/path/to/*.xml" "space-name?start#my-results"
$ testspace "[Folder2]/path/to/*.xml" "space-name?add#my-results"
$ testspace "space-name?finish#my-results"