Data Formats

Testspace accepts by default many widely adopted test output, code coverage and static analysis formats. In case your test framework does not generate any of these standard formats, this article provides a reference for a generic format for implementing a converter to transform results so that they can be published in Testspace.

Test Output Formats

Unit/integration test results, depending on the format, are processed and organized in a number of Suites. Optionally Testspace can organize the Suites in a Folder hierarchy matching their source location.

JUnit

The origin of the JUnit XML report format is the JUnit ANT task. No official specification for the format has ever been defined, however a very detailed XSD schema can be found here.

Many popular test frameworks can produce JUnit output directly, for example: GoogleTest, Jasmine, JUnit, Nose, PHPUnit, RSpec, etc.

NUnit

This is the XML report format of NUnit (unit-testing framework for all .NET languages). The standard definition of it can be found in here.

Visual Studio TRX

This is the XML report format of the Microsoft's unit test framework. A XSD schema of it can be found at your Visual Studio's installation directory - %VSINSTALLDIR%\xml\Schemas\vstst.xsd.

xUnit.net

This is the XML report format of xUnit.net (free, open-source, community-focused unit testing tool for the .NET Framework). The standard definition of it can be found in here.

Testspace

Testspace defines a generic XML report format (see below). Independent of your test framework and/or language, allows for converting of test output into the generic format.

Code Coverage Formats

Each code coverage result file is placed in a Suite named "Code Coverage". If merging of results is required, it has to be done before publishing using the same toolset that generated the results.

Bullseye

This is the XML report format of Bullseye (C/C++ code coverage analyzer). The standard definition of it can be found in here.

Clover

This is the XML report format of Clover (Java and Groovy code coverage). A sample XSD schema of it can be found here.

Many popular code coverage tools can produce Clover output directly, for example: PHPUnit, Devel:Cover, etc.

Cobertura

This is the XML report format of Cobertura (Java code coverage utility). A definition of it can be found here.

Many popular code coverage tools can produce Cobertura output directly, for example: gcov, coverage.py, SimpleCov, JSCover, etc.

dotCover

This is the XML report format of dotCover (.NET unit test runner and code coverage tool).

JaCoCo

This is the XML report format of JaCoCo (code coverage library for Java). The standard definition of it can be found here.

OpenCover

This is the XML report format of OpenCover (code coverage tool for .NET).

Visual Studio Coverage

This is the BINARY or XML report format of the Microsoft's unit test framework.

Static Analysis Formats

Static analysis (aka lint) results are placed in a Suite named "Static Analysis". If multiple static analysis results are provided they will all be automatically merged into the same Suite.

Android lint

This is the XML report format of Android lint (code scanning tool that can help to identify and correct problems with the structural quality of Java code).

Checkstyle

This is the XML report format of Checkstyle (development tool to help programmers write Java code that adheres to a coding standard).

Many popular static analysis tools can produce Checkstyle output directly or via add-on reporters, for example: Checkstyle, PHP_CodeSniffer, Rubocop, Brakeman, JSHint, ESLint, Vera++, etc.

Cppcheck

This is the XML report format of Cppcheck (static analysis tool for C/C++ code).

Klocwork

This is the XML report format of Klockwork (static code analysis tool for C/C++, Java, and C#).

PMD

This is the XML report format of PMD (source code analyzer for Java).

Many popular static analysis tools can produce PMD output directly or via add-on reporters, for example: PHPMD, OCLint, etc.

Visual Studio FxCop

This is the XML report format of the Microsoft's .NET Managed code analysis tool.

Visual Studio PREfast

This is the XML report format of Microsoft's native C/C++ code analysis tool.

Text Log

GCC (aka "GNU Emacs") and/or Visual Studio compatible TEXT report format is an optional output for many static analysis tools.

To be recognized, a reported "lint" in a text log has to be formatted on a single line in one of the following ways:

GCC (aka "GNU Emacs")
file[:line[:column]]: type: message
Visual Studio
file[(line[,column])]: type: message

Generic Format

The Testspace generic XML format is defined to allow detailed reporting of any test results. Use the following reference when developing a converter to transform a foreign result format so that it can be uploaded to Testspace.

Reporting Entities

The entities in the Testspace result model are:

Folder
Test Suite
Test Case
Annotation
Comment
Custom Data

The Test Suite and Test Case entities are essential to any report. Other entities are not required and can be used to supplement the report with additional data.

Folder

A Folder is used to group and organize Test Suites and other Folders. It has a name and an optional description.

Test Suite

A Test Suite serves to group similar Test Cases and optionally holds supplemental information via Annotations or Custom Data. It has a name and an optional description.

Each Test Suite also provides a roll-up subtotal of its children's pass/fail/error counts and durations, which allows you to better pinpoint where failures or other problems are occurring in your tests.

Test Case

The Test Case is the individual unit of pass/fail as optionally could contain supplemental information via Annotations or Custom Data. It has a name, a result status, and an optional description.

Annotation

An Annotation can be attached to either a Test Suite or a Test Case to provide supplemental information. It has a name, a level (either "info", "warn" or "error") and an optional description.

An Annotation can be used to hold text (typically using one or more Comments), and it can also be used to hold a URL link or to attach data from a file to the test result.

Comment

A Comment is used to attach additional text to an annotation.

Custom Data

Custom Data is used to include metrics data in your test report. The values supplied with Custom Data can be displayed graphically on the Testspace Space Metrics tab.

XML Elements

Here is a format definition followed by an XSD that you can use to validate candidate documents.

Reporting Elements

The required elements in a Testspace report are

reporter (root element)
test_suite
test_case

Optional elements are:

annotation
comment
custom_data

The diagram shows how these items are related.

`<reporter>`

This is the root element of the document.

Allowed child elements are:

test_suite
annotation

Attributes

Name	Value	Required?	Comments
`product_version`	anything	yes	Typically, the name and version of the script used to convert from native format is put here. This value is for reference only, it's not used by Testspace.
`schema_version`	"1.0"	yes	This value must be as shown.

Example

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<reporter product_version="my product" schema_version="1.0">
    <!-- report body goes here -->
</reporter>

`<test_suite>`

This element functions as either:

a Test Suite (container for Test Cases) with allowed child elements:
```
* test_case
* annotation
* custom_data
```
or
a Folder (container of other Test Suites) with allowed child elements:
- test_suite
- annotation
- custom_data

Note: If a test_suite element has a mix of test_suite and test_case elements as children, it will function as a Test Suite and any child test_suite elements will be ignored.

Attributes

Name	Value	Required?	Comments
`name`	Name of the test suite	yes	Must be unique compared to all sibling test_suites. Also when functions as a Folder the characters `/&#?` are not allowed.
`description`	Description of the test suite	no
`start_time`	Date and time this suite began executing	no	Required to be in ISO 8601 format.
`duration`	Total execution time of this test suite (milliseconds)	no	If missing, upon upload, this value is (re)calculated by Testspace out of the test_cases times under that suite.

Examples

<!-- outer test suite has one or more test_case as children, so functions as a Suite -->
<test_suite name="Test-Suite-name" description="Test suite description">
   <!-- test cases go here -->
</test_suite>

<!-- outer test suite has no test_case as children, so functions as a Folder -->
<test_suite name="Test-Suite Used as Folder">
  <test_suite name="Test-Suite-name" description="Test suite description" start_time="2015-04-23T15:26:48" duration="1523">
    <!-- test cases go here -->
  </test_suite>
</test_suite>

test_suite Attributes Calculated by Testspace

These attributes are roll-up values calculated by Testspace and will be seen in exported report data. They are not required in XML that is uploaded. If given, these attributes will be ignored by Testspace.

Name	Value
`passed`	Total of all child test cases with passed status
`failed`	Total of all child test cases with failed status
`not_applicable`	Total of all child test cases with not_applicable status
`errored`	Total of all child test cases with blocked (aka errored) status
`unknown`	Total of all child test cases with untested (aka unknown) status

`<test_case>`

This element represents a Test Case.

Allowed child elements are:

annotation
custom_data

Attributes

Name	Value	Required?	Comments
`name`	Name of the test case	yes	Must be unique compared to all sibling test_cases
`description`	Description of the test case	no
`status`	Result of the test	yes	Must be one of five values. See Test Case Status Values (below)
`duration`	Time under test for this test case (milliseconds)	yes	Value can contain decimal point: `"9.123456"`. If unknown set value to ''"0"''
`start_time`	Date and time test case execution began	no	Required to be in IS0 860 format

Test Case Status Values

Status	Description
`"passed"`	Test passed
`"failed"`	Test failed
`"not_applicable"`	Test has no result (e.g. test case was skipped or its result was suppressed)
`"in_progress"`	(deprecated) Test crashed or experienced hang
`"errored"`	Test had an unanticipated problem. (Typically due to a crash or timeout/hang, an unchecked throwable; or a problem with the implementation of the test)
`"blocked"`	Same as `errored`
`"unknown"`	Test was not run. (Typically due to prior test crashing or hanging)
`"untested"`	Same as `unknown`
`"ignored"`	Test has no result (e.g. test case used to report metrics)

Example

<test_suite name="Test-Suite-name" start_time="" description="Test suite description">
  <!-- test_case must be a child of test_suite element -->
  <test_case name="Test1" status="passed" duration="13.000000" start_time="2015-01-22T23:41:08" />
  <test_case name="Test2" description="Test case description" status="passed" duration="4.000000" start_time="2015-01-22T23:41:08" />
</test_suite>

`<annotation>`

This element represents an Annotation and is used for applying supplemental information to a test_suite, a test_case, or the document root.

Annotations allow you to add text along with other data. The other data can be either a URL link or file data that is attached to the report.

Specifically, there are three types of annotation:

no-attachment
link-attached
data-attached

Attributes

These attributes are common for all annotation types.

Name	Value	Required	Comments
`name`	Name of the annotation	yes	The name is displayed in the left-most column of the annotation display.
`level`	Error level of the annotation	yes	Value is either `"info"`, `"warn"`, or `"error"`
`description`	Description of the annotation	no	Description displayed in Testspace

additional link-attached attributes

This attribute is additionally required for link-attached annotations

Name	Value	Required?	Comments
`link_file`	"true"	yes
`file`	URL of the linked file	yes	This can be any valid URL

additional data-attached attributes

These attributes are additionally used for data-attached annotations

Name	Value	Required?	Comments
`file_name`	Name of the linked file	yes	Filename to be used when attached data is downloaded as a file from Testspace
`mime_type`	Mime type of data	no	Defaults to `"octet/stream"`

The data attached to the annotation must be compressed using gzip and converted to text using Base64, then placed in a CDATA section as child text element.

Examples

<!-- no-attachment annotations -->
<annotation  name="My Info Annotation" level="info" description="demo of no-attachment info-level annotation"/>
<annotation name="My Error Annotation" level="error" description="demo of no-attachment error-level annotation"/>
<!-- link-attached annotation -->
<annotation name="My File Link" level="info" description="demo of link-attached annotation" file="http://eudoxos.de/gtk/databox/screenshots/grid.png" link_file="true"/>
<!-- data-attached annotation -->
<annotation name="My File Attach" description="demo of data-attached annotation" level="error" file_name="wbemcore.log" link_file="false">
  <![CDATA[H4sICLhdg0MAAHdtaXByb3YubG9nAJ2TP08DMQzFdyS+g0cYqGLHviTeKsHAUAaE1Lm6BnRSuUPX
  P5+ftE1PtB3IVcqQIe/n52fnYR6XMN1+ASKgUyS1DGQMT6x4g/IICvPnGdSL1Qr6uNn2bRLEvu96
  BU4P7+8uEGSU8IhA591tBM4m2Dgxe8J7rGOzS89edrHdXEtQRbJv9sGML2pZmXLRgGzdeARjOkdE
  MCFUfo94/f6J/bprF5uma+Fz0ayS5gmmdR3Xa1jGtonLRPrYRnjrdoACqRebEjB7kkwQq0BYYuYC
  gWp8RrCw2OsU/9EISqEmDBoXqtEa74rqWE2Xg4YcerYlkcy69oCg44hJMbfHlQtyC8KryS48cup2
  xIjPSWzT0uZ+KrESxpsRq+JPCIu+aGVPqRKBCYdU6YQIhf/9HCGpF5eH6WjkrmZEUBpcuMLZ/kGg
  UcPpxwzr4XyJi1+WXbTO/AQAAA==]]>
</annotation>

`<comment>`

This element represents a Comment and is used for providing additional label-text pairs with an annotation.

Attributes

Name	Value	Required?	Comments
`label`	Comment label	yes

The text must be put in a CDATA section as child text element.

Example

<annotation name="My Error Annotation" level="error" description="demo of no-attachment error-level annotation"/>
  <!-- comment must be a child of an annotation element -->
  <comment label="Comment Label">
    <![CDATA[Comment contents.]]>
  </comment>
</annotation>

`<custom_data>`

This element represents a Custom Data and is used for including custom metrics into your report (i.e. test_suite or test_case element) so that you can track them in Testspace.¹

Custom data comprises a name and value. The values must be a comma-separated list of numbers.

Attributes

Name	Value	Required?	Comments
`name`	Custom data name	yes

The value must be put in a CDATA section as child text element. You can specify one, two, or three numbers separated by commas. The numbers can be whole integers, floating-point numbers, or a mix of the two.

Example

<!-- custom_data with one number value -->
<custom_data name="CustomValue1"><![CDATA[12]]></custom_data>
<!-- custom_data with two floating point numbers value -->
<custom_data name="CustomValue2"><![CDATA[12.1,12.456]]></custom_data>
<!-- custom_data with three numbers value -->
<custom_data name="CustomValue3"><![CDATA[12.25,1,13]]></custom_data>

XML Schema Definition

<?xml version="1.0"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">

<xs:complexType name="commentType">
    <xs:simpleContent>
        <xs:extension base="xs:string">
            <xs:attribute name="label" type="xs:string" use="required" />
         </xs:extension>
    </xs:simpleContent>
</xs:complexType>

<xs:complexType name="customdataType">
    <xs:simpleContent>
        <xs:extension base="xs:string">
            <xs:attribute name="name" type="xs:string" use="required" />
        </xs:extension>
    </xs:simpleContent>
</xs:complexType>

<xs:complexType name="annotationType" mixed="true">
    <xs:sequence minOccurs="0" maxOccurs="unbounded">
        <xs:element name="comment" type="commentType"/>
    </xs:sequence>
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="description" type="xs:string"/>
    <xs:attribute name="file" type="xs:string"/>
    <xs:attribute name="level" use="required">
        <xs:simpleType>
            <xs:restriction base="xs:string">
                <xs:enumeration value="info"/>
                <xs:enumeration value="warn"/>
                <xs:enumeration value="error"/>
            </xs:restriction>
        </xs:simpleType>
    </xs:attribute>
    <xs:attribute name="link_file" type="xs:boolean"/>
    <xs:attribute name="file_name" type="xs:string"/>
    <xs:attribute name="mime_type" type="xs:string"/>
</xs:complexType>

<xs:complexType name="testcaseType">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element name="custom_data" type="customdataType"/>
        <xs:element name="annotation" type="annotationType"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="description" type="xs:string"/>
    <xs:attribute name="start_time" type="xs:string" />
    <xs:attribute name="status" use="required">
        <xs:simpleType>
            <xs:restriction base="xs:string">
                <xs:enumeration value="passed"/>
                <xs:enumeration value="failed"/>
                <xs:enumeration value="errored"/>
                <xs:enumeration value="blocked"/>
                <xs:enumeration value="not_applicable"/>
                <xs:enumeration value="in_progress"/>
                <xs:enumeration value="unknown"/>
                <xs:enumeration value="untested"/>
                <xs:enumeration value="ignored"/>
            </xs:restriction>
        </xs:simpleType>
    </xs:attribute>
    <xs:attribute name="duration" type="xs:float"/>
</xs:complexType>

<xs:complexType name="testsuiteType">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element name="test_case" type="testcaseType"/>
        <xs:element name="test_suite" type="testsuiteType"/>
        <xs:element name="custom_data" type="customdataType"/>
        <xs:element name="annotation" type="annotationType"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="description" type="xs:string" />
    <xs:attribute name="start_time" type="xs:string" />
    <xs:attribute name="passed" type="xs:unsignedInt" />
    <xs:attribute name="failed" type="xs:unsignedInt" />
    <xs:attribute name="not_applicable" type="xs:unsignedInt" />
    <xs:attribute name="in_progress" type="xs:unsignedInt" />
    <xs:attribute name="errored" type="xs:unsignedInt" />
    <xs:attribute name="unknown" type="xs:unsignedInt" />
    <xs:attribute name="duration" type="xs:float"/>
</xs:complexType>

<xs:element name="reporter">
    <xs:complexType>
        <xs:choice minOccurs="1" maxOccurs="unbounded">
            <xs:element name="test_suite" type="testsuiteType"/>
            <xs:element name="annotation" type="annotationType"/>
        </xs:choice>
        <xs:attribute name="product_version" type="xs:string"/>
        <xs:attribute name="schema_version" type="xs:string"/>
    </xs:complexType>
</xs:element>

</xs:schema>

Within Testspace, you can configure historical tracking and notification of custom metrics.↩

Test Output Formats​

JUnit​

NUnit​

Visual Studio TRX​

xUnit.net​

Testspace​

Code Coverage Formats​

Bullseye​

Clover​

Cobertura​

dotCover​

JaCoCo​

OpenCover​

Visual Studio Coverage​

Static Analysis Formats​

Android lint​

Checkstyle​

Cppcheck​

Klocwork​

PMD​

Visual Studio FxCop​

Visual Studio PREfast​

Text Log​

Generic Format​

Reporting Entities​

Folder​

Test Suite​

Test Case​

Annotation​

Comment​

Custom Data​

XML Elements​

<reporter>​

Attributes​

<test_suite>​

Attributes​

test_suite Attributes Calculated by Testspace​

<test_case>​

Attributes​

Test Case Status Values​

<annotation>​

Attributes​

additional link-attached attributes​

additional data-attached attributes​

<comment>​

Attributes​

<custom_data>​

Attributes​

XML Schema Definition​

Test Output Formats

JUnit

NUnit

Visual Studio TRX

xUnit.net

Testspace

Code Coverage Formats

Bullseye

Clover

Cobertura

dotCover

JaCoCo

OpenCover

Visual Studio Coverage

Static Analysis Formats

Android lint

Checkstyle

Cppcheck

Klocwork

PMD

Visual Studio FxCop

Visual Studio PREfast

Text Log

Generic Format

Reporting Entities

Folder

Test Suite

Test Case

Annotation

Comment

Custom Data

XML Elements

`<reporter>`

Attributes

`<test_suite>`

Attributes

test_suite Attributes Calculated by Testspace

`<test_case>`

Attributes

Test Case Status Values

`<annotation>`

Attributes

additional link-attached attributes

additional data-attached attributes

`<comment>`

Attributes

`<custom_data>`

Attributes

XML Schema Definition