Skip to main content

Data Formats

Testspace accepts by default many widely adopted test output, code coverage and static analysis formats. In case your test framework does not generate any of these standard formats, this article provides a reference for a generic format for implementing a converter to transform results so that they can be published in Testspace.

Test Output Formats#

Unit/integration test results, depending on the format, are processed and organized in a number of Suites. Optionally Testspace can organize the Suites in a Folder hierarchy matching their source location.

JUnit#

The origin of the JUnit XML report format is the JUnit ANT task. No official specification for the format has ever been defined, however a very detailed XSD schema can be found here.

Many popular test frameworks can produce JUnit output directly, for example: GoogleTest, Jasmine, JUnit, Nose, PHPUnit, RSpec, etc.

NUnit#

This is the XML report format of NUnit (unit-testing framework for all .NET languages). The standard definition of it can be found in here.

Visual Studio TRX#

This is the XML report format of the Microsoft's unit test framework. A XSD schema of it can be found at your Visual Studio's installation directory - %VSINSTALLDIR%\xml\Schemas\vstst.xsd.

xUnit.net#

This is the XML report format of xUnit.net (free, open-source, community-focused unit testing tool for the .NET Framework). The standard definition of it can be found in here.

Testspace#

Testspace defines a generic XML report format (see below). Independent of your test framework and/or language, allows for converting of test output into the generic format.

Code Coverage Formats#

Each code coverage result file is placed in a Suite named "Code Coverage". If merging of results is required, it has to be done before publishing using the same toolset that generated the results.

Bullseye#

This is the XML report format of Bullseye (C/C++ code coverage analyzer). The standard definition of it can be found in here.

Clover#

This is the XML report format of Clover (Java and Groovy code coverage). A sample XSD schema of it can be found here.

Many popular code coverage tools can produce Clover output directly, for example: PHPUnit, Devel:Cover, etc.

Cobertura#

This is the XML report format of Cobertura (Java code coverage utility). A definition of it can be found here.

Many popular code coverage tools can produce Cobertura output directly, for example: gcov, coverage.py, SimpleCov, JSCover, etc.

dotCover#

This is the XML report format of dotCover (.NET unit test runner and code coverage tool).

JaCoCo#

This is the XML report format of JaCoCo (code coverage library for Java). The standard definition of it can be found here.

OpenCover#

This is the XML report format of OpenCover (code coverage tool for .NET).

Visual Studio Coverage#

This is the BINARY or XML report format of the Microsoft's unit test framework.

Static Analysis Formats#

Static analysis (aka lint) results are placed in a Suite named "Static Analysis". If multiple static analysis results are provided they will all be automatically merged into the same Suite.

Android lint#

This is the XML report format of Android lint (code scanning tool that can help to identify and correct problems with the structural quality of Java code).

Checkstyle#

This is the XML report format of Checkstyle (development tool to help programmers write Java code that adheres to a coding standard).

Many popular static analysis tools can produce Checkstyle output directly or via add-on reporters, for example: Checkstyle, PHP_CodeSniffer, Rubocop, Brakeman, JSHint, ESLint, Vera++, etc.

Cppcheck#

This is the XML report format of Cppcheck (static analysis tool for C/C++ code).

Klocwork#

This is the XML report format of Klockwork (static code analysis tool for C/C++, Java, and C#).

PMD#

This is the XML report format of PMD (source code analyzer for Java).

Many popular static analysis tools can produce PMD output directly or via add-on reporters, for example: PHPMD, OCLint, etc.

Visual Studio FxCop#

This is the XML report format of the Microsoft's .NET Managed code analysis tool.

Visual Studio PREfast#

This is the XML report format of Microsoft's native C/C++ code analysis tool.

Text Log#

GCC (aka "GNU Emacs") and/or Visual Studio compatible TEXT report format is an optional output for many static analysis tools.

To be recognized, a reported "lint" in a text log has to be formatted on a single line in one of the following ways:

  • GCC (aka "GNU Emacs")

    file[:line[:column]]: type: message

  • Visual Studio

    file[(line[,column])]: type: message

Generic Format#

The Testspace generic XML format is defined to allow detailed reporting of any test results. Use the following reference when developing a converter to transform a foreign result format so that it can be uploaded to Testspace.

Reporting Entities#

The entities in the Testspace result model are:

  • Folder
  • Test Suite
  • Test Case
  • Annotation
  • Comment
  • Custom Data

The Test Suite and Test Case entities are essential to any report. Other entities are not required and can be used to supplement the report with additional data.

Folder#

A Folder is used to group and organize Test Suites and other Folders. It has a name and an optional description.

Test Suite#

A Test Suite serves to group similar Test Cases and optionally holds supplemental information via Annotations or Custom Data. It has a name and an optional description.

Each Test Suite also provides a roll-up subtotal of its children's pass/fail/error counts and durations, which allows you to better pinpoint where failures or other problems are occurring in your tests.

Test Case#

The Test Case is the individual unit of pass/fail as optionally could contain supplemental information via Annotations or Custom Data. It has a name, a result status, and an optional description.

Annotation#

An Annotation can be attached to either a Test Suite or a Test Case to provide supplemental information. It has a name, a level (either "info", "warn" or "error") and an optional description.

An Annotation can be used to hold text (typically using one or more Comments), and it can also be used to hold a URL link or to attach data from a file to the test result.

Comment#

A Comment is used to attach additional text to an annotation.

Custom Data#

Custom Data is used to include metrics data in your test report. The values supplied with Custom Data can be displayed graphically on the Testspace Space Metrics tab.

XML Elements#

Here is a format definition followed by an XSD that you can use to validate candidate documents.

Reporting Elements

The required elements in a Testspace report are

  • reporter (root element)
  • test_suite
  • test_case

Optional elements are:

  • annotation
  • comment
  • custom_data

The diagram shows how these items are related.

<reporter>#

This is the root element of the document.

Allowed child elements are:

  • test_suite
  • annotation
Attributes#
NameValueRequired?Comments
product_versionanythingyesTypically, the name and version of the script used to convert from native format is put here. This value is for reference only, it's not used by Testspace.
schema_version"1.0"yesThis value must be as shown.

Example

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<reporter product_version="my product" schema_version="1.0">
<!-- report body goes here -->
</reporter>

<test_suite>#

This element functions as either:

  • a Test Suite (container for Test Cases) with allowed child elements:
    * test_case
    * annotation
    * custom_data
    or
  • a Folder (container of other Test Suites) with allowed child elements:
    • test_suite
    • annotation
    • custom_data

Note: If a test_suite element has a mix of test_suite and test_case elements as children, it will function as a Test Suite and any child test_suite elements will be ignored.

Attributes#
NameValueRequired?Comments
nameName of the test suiteyesMust be unique compared to all sibling test_suites. Also when functions as a Folder the characters /&#? are not allowed.
descriptionDescription of the test suiteno
start_timeDate and time this suite began executingnoRequired to be in ISO 8601 format.
durationTotal execution time of this test suite (milliseconds)noIf missing, upon upload, this value is (re)calculated by Testspace out of the test_cases times under that suite.

Examples

<!-- outer test suite has one or more test_case as children, so functions as a Suite -->
<test_suite name="Test-Suite-name" description="Test suite description">
<!-- test cases go here -->
</test_suite>
<!-- outer test suite has no test_case as children, so functions as a Folder -->
<test_suite name="Test-Suite Used as Folder">
<test_suite name="Test-Suite-name" description="Test suite description" start_time="2015-04-23T15:26:48" duration="1523">
<!-- test cases go here -->
</test_suite>
</test_suite>

test_suite Attributes Calculated by Testspace#

These attributes are roll-up values calculated by Testspace and will be seen in exported report data. They are not required in XML that is uploaded. If given, these attributes will be ignored by Testspace.

NameValue
passedTotal of all child test cases with passed status
failedTotal of all child test cases with failed status
not_applicableTotal of all child test cases with not_applicable status
erroredTotal of all child test cases with errored status
unknownTotal of all child test cases with untested (aka unknown) status

<test_case>#

This element represents a Test Case.

Allowed child elements are:

  • annotation
  • custom_data
Attributes#
NameValueRequired?Comments
nameName of the test caseyesMust be unique compared to all sibling test_cases
descriptionDescription of the test caseno
statusResult of the testyesMust be one of five values. See Test Case Status Values (below)
durationTime under test for this test case (milliseconds)yesValue can contain decimal point: "9.123456". If unknown set value to ''"0"''
start_timeDate and time test case execution begannoRequired to be in IS0 860 format

Test Case Status Values#

StatusDescription
"passed"Test passed
"failed"Test failed
"not_applicable"Test has no result (e.g. test case was skipped or its result was suppressed)
"in_progress"(deprecated) Test crashed or experienced hang
"errored"Test had an unanticipated problem. (Typically due to a crash or timeout/hang, an unchecked throwable; or a problem with the implementation of the test)
"unknown"Test was not run. (Typically due to prior test crashing or hanging)
"ignored"Test has no result (e.g. test case used to report metrics)

Example

<test_suite name="Test-Suite-name" start_time="" description="Test suite description">
<!-- test_case must be a child of test_suite element -->
<test_case name="Test1" status="passed" duration="13.000000" start_time="2015-01-22T23:41:08" />
<test_case name="Test2" description="Test case description" status="passed" duration="4.000000" start_time="2015-01-22T23:41:08" />
</test_suite>

<annotation>#

This element represents an Annotation and is used for applying supplemental information to a test_suite, a test_case, or the document root.

Annotations allow you to add text along with other data. The other data can be either a URL link or file data that is attached to the report.

Specifically, there are three types of annotation:

  • no-attachment
  • link-attached
  • data-attached
Attributes#

These attributes are common for all annotation types.

NameValueRequiredComments
nameName of the annotationyesThe name is displayed in the left-most column of the annotation display.
levelError level of the annotationyesValue is either "info", "warn", or "error"
descriptionDescription of the annotationnoDescription displayed in Testspace
additional link-attached attributes#

This attribute is additionally required for link-attached annotations

NameValueRequired?Comments
link_file"true"yes
fileURL of the linked fileyesThis can be any valid URL
additional data-attached attributes#

These attributes are additionally used for data-attached annotations

NameValueRequired?Comments
file_nameName of the linked fileyesFilename to be used when attached data is downloaded as a file from Testspace
mime_typeMime type of datanoDefaults to "octet/stream"

The data attached to the annotation must be compressed using gzip and converted to text using Base64, then placed in a CDATA section as child text element.

Examples

<!-- no-attachment annotations -->
<annotation name="My Info Annotation" level="info" description="demo of no-attachment info-level annotation"/>
<annotation name="My Error Annotation" level="error" description="demo of no-attachment error-level annotation"/>
<!-- link-attached annotation -->
<annotation name="My File Link" level="info" description="demo of link-attached annotation" file="http://eudoxos.de/gtk/databox/screenshots/grid.png" link_file="true"/>
<!-- data-attached annotation -->
<annotation name="My File Attach" description="demo of data-attached annotation" level="error" file_name="wbemcore.log" link_file="false">
<![CDATA[H4sICLhdg0MAAHdtaXByb3YubG9nAJ2TP08DMQzFdyS+g0cYqGLHviTeKsHAUAaE1Lm6BnRSuUPX
P5+ftE1PtB3IVcqQIe/n52fnYR6XMN1+ASKgUyS1DGQMT6x4g/IICvPnGdSL1Qr6uNn2bRLEvu96
BU4P7+8uEGSU8IhA591tBM4m2Dgxe8J7rGOzS89edrHdXEtQRbJv9sGML2pZmXLRgGzdeARjOkdE
MCFUfo94/f6J/bprF5uma+Fz0ayS5gmmdR3Xa1jGtonLRPrYRnjrdoACqRebEjB7kkwQq0BYYuYC
gWp8RrCw2OsU/9EISqEmDBoXqtEa74rqWE2Xg4YcerYlkcy69oCg44hJMbfHlQtyC8KryS48cup2
xIjPSWzT0uZ+KrESxpsRq+JPCIu+aGVPqRKBCYdU6YQIhf/9HCGpF5eH6WjkrmZEUBpcuMLZ/kGg
UcPpxwzr4XyJi1+WXbTO/AQAAA==]]>
</annotation>

<comment>#

This element represents a Comment and is used for providing additional label-text pairs with an annotation.

Attributes#
NameValueRequired?Comments
labelComment labelyes

The text must be put in a CDATA section as child text element.

Example

<annotation name="My Error Annotation" level="error" description="demo of no-attachment error-level annotation"/>
<!-- comment must be a child of an annotation element -->
<comment label="Comment Label">
<![CDATA[Comment contents.]]>
</comment>
</annotation>

<custom_data>#

This element represents a Custom Data and is used for including custom metrics into your report (i.e. test_suite or test_case element) so that you can track them in Testspace.1

Custom data comprises a name and value. The values must be a comma-separated list of numbers.

Attributes#
NameValueRequired?Comments
nameCustom data nameyes

The value must be put in a CDATA section as child text element. You can specify one, two, or three numbers separated by commas. The numbers can be whole integers, floating-point numbers, or a mix of the two.

Example

<!-- custom_data with one number value -->
<custom_data name="CustomValue1"><![CDATA[12]]></custom_data>
<!-- custom_data with two floating point numbers value -->
<custom_data name="CustomValue2"><![CDATA[12.1,12.456]]></custom_data>
<!-- custom_data with three numbers value -->
<custom_data name="CustomValue3"><![CDATA[12.25,1,13]]></custom_data>

XML Schema Definition#

<?xml version="1.0"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:complexType name="commentType">
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="label" type="xs:string" use="required" />
</xs:extension>
</xs:simpleContent>
</xs:complexType>
<xs:complexType name="customdataType">
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="name" type="xs:string" use="required" />
</xs:extension>
</xs:simpleContent>
</xs:complexType>
<xs:complexType name="annotationType" mixed="true">
<xs:sequence minOccurs="0" maxOccurs="unbounded">
<xs:element name="comment" type="commentType"/>
</xs:sequence>
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="description" type="xs:string"/>
<xs:attribute name="file" type="xs:string"/>
<xs:attribute name="level" use="required">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="info"/>
<xs:enumeration value="warn"/>
<xs:enumeration value="error"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="link_file" type="xs:boolean"/>
<xs:attribute name="file_name" type="xs:string"/>
<xs:attribute name="mime_type" type="xs:string"/>
</xs:complexType>
<xs:complexType name="testcaseType">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="custom_data" type="customdataType"/>
<xs:element name="annotation" type="annotationType"/>
</xs:choice>
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="description" type="xs:string"/>
<xs:attribute name="start_time" type="xs:string" />
<xs:attribute name="status" use="required">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="passed"/>
<xs:enumeration value="failed"/>
<xs:enumeration value="not_applicable"/>
<xs:enumeration value="in_progress"/>
<xs:enumeration value="errored"/>
<xs:enumeration value="unknown"/>
<xs:enumeration value="ignored"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="duration" type="xs:float"/>
</xs:complexType>
<xs:complexType name="testsuiteType">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="test_case" type="testcaseType"/>
<xs:element name="test_suite" type="testsuiteType"/>
<xs:element name="custom_data" type="customdataType"/>
<xs:element name="annotation" type="annotationType"/>
</xs:choice>
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="description" type="xs:string" />
<xs:attribute name="start_time" type="xs:string" />
<xs:attribute name="passed" type="xs:unsignedInt" />
<xs:attribute name="failed" type="xs:unsignedInt" />
<xs:attribute name="not_applicable" type="xs:unsignedInt" />
<xs:attribute name="in_progress" type="xs:unsignedInt" />
<xs:attribute name="errored" type="xs:unsignedInt" />
<xs:attribute name="unknown" type="xs:unsignedInt" />
<xs:attribute name="duration" type="xs:float"/>
</xs:complexType>
<xs:element name="reporter">
<xs:complexType>
<xs:choice minOccurs="1" maxOccurs="unbounded">
<xs:element name="test_suite" type="testsuiteType"/>
<xs:element name="annotation" type="annotationType"/>
</xs:choice>
<xs:attribute name="product_version" type="xs:string"/>
<xs:attribute name="schema_version" type="xs:string"/>
</xs:complexType>
</xs:element>
</xs:schema>

  1. Within Testspace, you can configure historical tracking and notification of custom metrics.