Testspace accepts by default many widely adopted
code coverage and
static analysis formats. In case your test framework does not generate any of these standard formats, this article provides a reference for a generic format for implementing a converter to transform results so that they can be published in Testspace.
Unit/integration test results, depending on the format, are processed and organized in a number of Suites. Optionally Testspace can organize the Suites in a Folder hierarchy matching their source location.
Many popular test frameworks can produce JUnit output directly, for example: GoogleTest, Jasmine, JUnit, Nose, PHPUnit, RSpec, etc.
This is the XML report format of the Microsoft's unit test framework. A XSD schema of it can be found at your Visual Studio's installation directory -
Testspace defines a generic XML report format (see below). Independent of your test framework and/or language, allows for converting of test output into the generic format.
Each code coverage result file is placed in a Suite named "Code Coverage". If merging of results is required, it has to be done before publishing using the same toolset that generated the results.
Many popular code coverage tools can produce Clover output directly, for example: PHPUnit, Devel:Cover, etc.
Many popular code coverage tools can produce Cobertura output directly, for example: gcov, coverage.py, SimpleCov, JSCover, etc.
This is the XML report format of dotCover (.NET unit test runner and code coverage tool).
This is the XML report format of OpenCover (code coverage tool for .NET).
This is the BINARY or XML report format of the Microsoft's unit test framework.
Static analysis (aka lint) results are placed in a Suite named "Static Analysis". If multiple static analysis results are provided they will all be automatically merged into the same Suite.
This is the XML report format of Android lint (code scanning tool that can help to identify and correct problems with the structural quality of Java code).
This is the XML report format of Checkstyle (development tool to help programmers write Java code that adheres to a coding standard).
Many popular static analysis tools can produce Checkstyle output directly or via add-on reporters, for example: Checkstyle, PHP_CodeSniffer, Rubocop, Brakeman, JSHint, ESLint, Vera++, etc.
This is the XML report format of Cppcheck (static analysis tool for C/C++ code).
This is the XML report format of Klockwork (static code analysis tool for C/C++, Java, and C#).
This is the XML report format of PMD (source code analyzer for Java).
Many popular static analysis tools can produce PMD output directly or via add-on reporters, for example: PHPMD, OCLint, etc.
This is the XML report format of the Microsoft's .NET Managed code analysis tool.
This is the XML report format of Microsoft's native C/C++ code analysis tool.
GCC (aka "GNU Emacs") and/or Visual Studio compatible TEXT report format is an optional output for many static analysis tools.
To be recognized, a reported "lint" in a text log has to be formatted on a single line in one of the following ways:
GCC (aka "GNU Emacs")
file[:line[:column]]: type: message
file[(line[,column])]: type: message
The Testspace generic XML format is defined to allow detailed reporting of any test results. Use the following reference when developing a converter to transform a foreign result format so that it can be uploaded to Testspace.
The entities in the Testspace result model are:
- Test Suite
- Test Case
- Custom Data
The Test Suite and Test Case entities are essential to any report. Other entities are not required and can be used to supplement the report with additional data.
A Folder is used to group and organize Test Suites and other Folders. It has a name and an optional description.
A Test Suite serves to group similar Test Cases and optionally holds supplemental information via Annotations or Custom Data. It has a name and an optional description.
Each Test Suite also provides a roll-up subtotal of its children's pass/fail/error counts and durations, which allows you to better pinpoint where failures or other problems are occurring in your tests.
The Test Case is the individual unit of pass/fail as optionally could contain supplemental information via Annotations or Custom Data. It has a name, a result status, and an optional description.
An Annotation can be attached to either a Test Suite or a Test Case to provide supplemental information. It has a name, a level (either "info", "warn" or "error") and an optional description.
An Annotation can be used to hold text (typically using one or more Comments), and it can also be used to hold a URL link or to attach data from a file to the test result.
A Comment is used to attach additional text to an annotation.
Custom Data is used to include
metrics data in your test report. The values supplied with Custom Data can be displayed graphically on the Testspace Space Metrics tab.
Here is a format definition followed by an XSD that you can use to validate candidate documents.
The required elements in a Testspace report are
- reporter (root element)
Optional elements are:
The diagram shows how these items are related.
This is the root element of the document.
Allowed child elements are:
|anything||yes||Typically, the name and version of the script used to convert from native format is put here. This value is for reference only, it's not used by Testspace.|
|"1.0"||yes||This value must be as shown.|
This element functions as either:
- a Test Suite (container for Test Cases) with allowed child elements:or* test_case* annotation* custom_data
- a Folder (container of other Test Suites) with allowed child elements:
Note: If a test_suite element has a mix of test_suite and test_case elements as children, it will function as a Test Suite and any child test_suite elements will be ignored.
|Name of the test suite||yes||Must be unique compared to all sibling test_suites. Also when functions as a Folder the characters |
|Description of the test suite||no|
|Date and time this suite began executing||no||Required to be in ISO 8601 format.|
|Total execution time of this test suite (milliseconds)||no||If missing, upon upload, this value is (re)calculated by Testspace out of the test_cases times under that suite.|
These attributes are roll-up values calculated by Testspace and will be seen in exported report data. They are not required in XML that is uploaded. If given, these attributes will be ignored by Testspace.
|Total of all child test cases with passed status|
|Total of all child test cases with failed status|
|Total of all child test cases with not_applicable status|
|Total of all child test cases with errored status|
|Total of all child test cases with untested (aka unknown) status|
This element represents a Test Case.
Allowed child elements are:
|Name of the test case||yes||Must be unique compared to all sibling test_cases|
|Description of the test case||no|
|Result of the test||yes||Must be one of five values. See Test Case Status Values (below)|
|Time under test for this test case (milliseconds)||yes||Value can contain decimal point: |
|Date and time test case execution began||no||Required to be in IS0 860 format|
|Test has no result (e.g. test case was skipped or its result was suppressed)|
|(deprecated) Test crashed or experienced hang|
|Test had an unanticipated problem. (Typically due to a crash or timeout/hang, an unchecked throwable; or a problem with the implementation of the test)|
|Test was not run. (Typically due to prior test crashing or hanging)|
|Test has no result (e.g. test case used to report metrics)|
This element represents an Annotation and is used for applying supplemental information to a test_suite, a test_case, or the document root.
Annotations allow you to add text along with other data. The other data can be either a URL link or file data that is attached to the report.
Specifically, there are three types of annotation:
These attributes are common for all annotation types.
|Name of the annotation||yes||The name is displayed in the left-most column of the annotation display.|
|Error level of the annotation||yes||Value is either |
|Description of the annotation||no||Description displayed in Testspace|
This attribute is additionally required for link-attached annotations
|URL of the linked file||yes||This can be any valid URL|
These attributes are additionally used for data-attached annotations
|Name of the linked file||yes||Filename to be used when attached data is downloaded as a file from Testspace|
|Mime type of data||no||Defaults to |
This element represents a Comment and is used for providing additional label-text pairs with an annotation.
text must be put in a CDATA section as child text element.
This element represents a Custom Data and is used for including custom metrics into your report (i.e. test_suite or test_case element) so that you can track them in Testspace.1
Custom data comprises a name and value. The values must be a comma-separated list of numbers.
|Custom data name||yes|
value must be put in a CDATA section as child text element. You can specify one, two, or three numbers separated by commas.
The numbers can be whole integers, floating-point numbers, or a mix of the two.
- Within Testspace, you can configure historical tracking and notification of custom metrics.↩