User Tools



Result File Format

Testspace accepts by default several widely adopted unit test, code coverage and static analysis result formats. In case your test framework does not generate any of those formats, this article provides a reference for a generic format that could guide you develop a converter to transform results so that they can be uploaded to Testspace.


Unit Test Formats

Unit test results, depending on the input format, are processed and organized in number of Suites. Optionally Testspace could organized the Suites in a Folder hierarchy matching their source location.

JUnit

The origin of the JUnit XML report format is the JUnit ANT task. No official specification for the format has ever been defined, however a very detailed XSD schema can be found here.

Many popular test frameworks can produce JUnit output directly, for example: GoogleTest, Jasmine, JUnit, Nose, PHPUnit, RSpec, etc.

NUnit

This is the XML report format of NUnit (unit-testing framework for all .NET languages). The standard definition of it can be found in here.

Visual Studio TRX

This is the XML report format of the Microsoft's unit test framework. A XSD schema of it can be found at your Visual Studio's installation directory - %VSINSTALLDIR%\xml\Schemas\vstst.xsd.

xUnit.net

This is the XML report format of xUnit.net (free, open source, community-focused unit testing tool for the .NET Framework). The standard definition of it can be found in here.

Testspace

Testspace defines a Generic XML report format. Independent of your test framework and/or language, converting your test output into the generic format, it will allow you to upload, organize and manage your results.


Code Coverage Formats

Each code coverage result file is placed in a Suite named "Code Coverage". If merging of results is required, it has to be done before publishing using the same tool set that generated the results.

Bullseye

This is the XML report format of Bullseye (C/C++ code coverage analyzer). The standard definition of it can be found in here.

Clover

This is the XML report format of Clover (Java and Groovy code coverage). A sample XSD schema of it can be found in here.

Many popular code coverage tools can produce Clover output directly, for example: PHPUnit, Devel:Cover, etc.

Cobertura

This is the XML report format of Cobertura (Java code coverage utility). A definition of it can be found in here.

Many popular code coverage tools can produce Cobertura output directly, for example: gcov, coverage.py, SimpleCov, JSCover, etc.

dotCover

This is the XML report format of dotCover (.NET unit test runner and code coverage tool).

JaCoCo

This is the XML report format of JaCoCo (code coverage library for Java). The standard definition of it can be found in here.

OpenCover

This is the XML report format of OpenCover (code coverage tool for .NET).

Visual Studio Coverage

This is the BINARY or XML report format of the Microsoft's unit test framework.


Static Analysis Formats

Static analysis (aka lint) results are placed in a Suite named "Static Analysis". If multiple static analysis results are provided they will all be automatically merged into the same Suite.

Android lint

This is the XML report format of Android lint (code scanning tool that can help identifying and correcting problems with the structural quality of Java code).

Checkstyle

This is the XML report format of Checkstyle (development tool to help programmers write Java code that adheres to a coding standard).

Many popular static analysis tools can produce Checkstyle output directly or via add-on reporters, for example: Checkstyle, PHP_CodeSniffer, Rubocop, Brakeman, JSHint, ESLint, Vera++, etc.

Cppcheck

This is the XML report format of Cppcheck (static analysis tool for C/C++ code).

Klocwork

This is the XML report format of Klocwork (static code analysis tool for C/C++, Java, and C#).

PMD

This is the XML report format of PMD (source code analyzer for Java).

Many popular static analysis tools can produce PMD output directly or via add-on reporters, for example: PHPMD, OCLint, etc.

Visual Studio FxCop

This is the XML report format of the Microsoft's .NET Managed code analysis tool.

Visual Studio PREfast

This is the XML report format of Microsoft's native C/C++ code analysis tool.

Text Log

GCC (aka "GNU Emacs") and/or Visual Studio compatible TEXT report format is an optional output for many static analysis tools.

To be recognized, a reported "lint" in a text log has to be formatted on a single line in one of the following ways:

  • GCC (aka "GNU Emacs")
file[:line[:column]]: type: message
  • Visual Studio
file[(line[,column])]: type: message

Generic Format

The Testspace generic XML format is defined to allow detailed reporting of any test results. Use the following reference when developing a converter to transform a foreign result format so that it can be uploaded to Testspace.

Reporting Entities

The entities in the Testspace result model are:

  • Folder
  • Test Suite
  • Test Case
  • Annotation
  • Comment
  • Custom Data
The Test Suite and Test Case entities are essential to any report. Other entities are not required, and can be used to supplement the report with additional data.

Folder

A Folder is used to group and organize Test Suites and other Folders. It has a name and an optional description.

Test Suite

A Test Suite serves to group similar Test Cases and optionally holds supplemental information via Annotations or Custom Data. It has a name and an optional description.

Each Test Suite also provides a roll-up subtotal of its children's pass/fail/error counts and durations, which allows you to better pinpoint where failures or other problems are occurring in your tests.

Test Case

The Test Case is the individual unit of pass/fail as optionally could contain supplemental information via Annotations or Custom Data. It has a name, a result status and an optional description.

Annotation

An Annotation can be attached to either a Test Suite or a Test Case to provide supplemental information. It has a name, a level (either "info", "warn" or "error") and optional description.

An Annotation can be used to hold text (typically using one or more Comments), and it can also be used to hold a URL link or to attach data from a file to the test result.

Comment

A Comment is used to attach additional text to an annotation.

Custom Data

Custom Data is used to include metrics data in your test report. The values supplied with Custom Data can be displayed graphically on the Testspace Reports tab.

Typical uses for this would be to track code coverage under test.

XML Elements

Here is a format definition followed by an XSD that you can use to validate candidate documents.

The required elements in a Testspace report are

  • reporter (root element)
  • test_suite
  • test_case

Optional elements are:

  • annotation
  • comment
  • custom_data

The diagram at the right shows how these items are related.


<reporter>

This is the root element of the document.

Allowed child elements are:

  • test_suite
  • annotation
reporter Attributes
Name Value Required? Comments
product_versionanything yes Typically, the name and version of the script used to convert from native format is put here. This value is for reference only, it's not used by Testspace.
schema_version"1.0" yes This value must be as shown.
Example
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<reporter product_version="my product" schema_version="1.0">
    <!-- report body goes here -->
</reporter>

<test_suite>

This element functions as either:

  • a Test Suite (container for Test Cases) with allowed child elements:
    • test_case
    • annotation
    • custom_data

or

  • a Folder (container of other Test Suites) with allowed child elements:
    • test_suite
    • annotation
    • custom_data
Note: If a test_suite element has a mix of test_suite and test_case elements as children, it will function as a Test Suite and any child test_suite elements will be ignored.
test_suite Attributes
Name Value Required? Comments
nameName of the test suite yes Must be unique compared to all sibling test_suites. Also when functions as a Folder the characters /&#? are not allowed.
descriptionDescription of the test suite no
start_timeDate and time this suite began executing no Required to be in ISO 8601 format.
durationTotal execution time of this test suite (milliseconds) no If missing, upon upload, this value is (re)calculated by Testspace out of the test_cases times under that suite.
Examples
<!-- outer test suite has one or more test_case as children, so functions as a Suite -->
    <test_suite name="Test-Suite-name" description="Test suite description">
        <!-- test cases go here -->
    </test_suite>
<!-- outer test suite has no test_case as children, so functions as a Folder -->
    <test_suite name="Test-Suite Used as Folder">
        <test_suite name="Test-Suite-name" description="Test suite description" start_time="2015-04-23T15:26:48" duration="1523">
            <!-- test cases go here -->
        </test_suite>
    </test_suite>
test_suite Attributes Calculated by Testspace

These attributes are roll-up values calculated by Testspace and will be seen in exported report data. They are not required in XML that is uploaded. If given, these attributes will be ignored by Testspace.

Name Value
failedTotal of all child test cases with failed status
in_progressTotal of all child test cases with in_progress status
not_applicableTotal of all child test cases with not_applicable status
passedTotal of all child test cases with passed status
unknownTotal of all child test cases with untested (aka unknown) status

<test_case>

This element represents a Test Case.

Allowed child elements are:

  • annotation
  • custom_data
test_case Attributes
Name Value Required? Comments
nameName of the test case yes Must be unique compared to all sibling test_cases
descriptionDescription of the test case no
statusResult of the test yes Must be one of five values. See Test Case Status Values
durationTime under test for this test case (milliseconds) yes Value can contain decimal point: "9.123456". If unknown set value to "0"
start_timeDate and time test case execution began no Required to be in ISO 8601 format
Test Case Status Values
"passed"Test passed
"failed"Test failed
"unknown"Test was not run. (Typically due to prior test crashing or hanging.)
"in_progress"Test crashed or experienced hang
"not_applicable"Test has no result (e.g. test case used to report metrics)
Example
    <test_suite name="Test-Suite-name" start_time="" description="Test suite description">
        <!-- test_case must be a child of test_suite element -->
        <test_case name="Test1" status="passed" duration="13.000000" start_time="2015-01-22T23:41:08" />
	<test_case name="Test2" description="Test case description" status="passed" duration="4.000000" start_time="2015-01-22T23:41:08" />
    </test_suite>

<annotation>

This element represents an Annotation and is used for applying supplemental information to a test_suite, a test_case, or the document root.

Annotations allow you to add text along with other data. The other data can be either a URL link or file data that is attached to the report.

Specifically, there are three types of annotation:

  • no-attachment
  • link-attached
  • data-attached
annotation Attributes

These attributes are common for all annotation types.

Name Value Required? Comments
name Name of the annotation yes The name is displayed in the left-most column of the annotation display.
level Error level of the annotation yes Value is either "info", "warn", or "error"
description Description of the annotation no Description displayed in Testspace

This attribute is additionally required for link-attached annotations

Name Value Required? Comments
link_file "true" yes
fileURL of the linked file yes This can be any valid URL
additional data-attached attributes

These attributes are additionally used for data-attached annotations

Name Value Required? Comments
file_nameName of the linked file yes Filename to be used when attached data is downloaded as a file from Testspace
mime_typeMime type of data no Defaults to "octet/stream"

The data attached to the annotation must be compressed using gzip and converted to text using Base64, then placed in a CDATA section as child text element.

Examples
    <!-- no-attachment annotations -->
    <annotation  name="My Info Annotation" level="info" description="demo of no-attachment info-level annotation"/>
    <annotation name="My Error Annotation" level="error" description="demo of no-attachment error-level annotation"/>
    <!-- link-attached annotation -->
    <annotation name="My File Link" level="info" description="demo of link-attached annotation"
	file="http://eudoxos.de/gtk/gtkdatabox/screenshots/grid.png" link_file="true"/>
    <!-- data-attached annotation -->
    <annotation name="My File Attach" description="demo of data-attached annotation" level="error"
	file_name="wbemcore.log" link_file="false">
	<![CDATA[H4sICLhdg0MAAHdtaXByb3YubG9nAJ2TP08DMQzFdyS+g0cYqGLHviTeKsHAUAaE1Lm6BnRSuUPX
	P5+ftE1PtB3IVcqQIe/n52fnYR6XMN1+ASKgUyS1DGQMT6x4g/IICvPnGdSL1Qr6uNn2bRLEvu96
	BU4P7+8uEGSU8IhA591tBM4m2Dgxe8J7rGOzS89edrHdXEtQRbJv9sGML2pZmXLRgGzdeARjOkdE
	MCFUfo94/f6J/bprF5uma+Fz0ayS5gmmdR3Xa1jGtonLRPrYRnjrdoACqRebEjB7kkwQq0BYYuYC
	gWp8RrCw2OsU/9EISqEmDBoXqtEa74rqWE2Xg4YcerYlkcy69oCg44hJMbfHlQtyC8KryS48cup2
	xIjPSWzT0uZ+KrESxpsRq+JPCIu+aGVPqRKBCYdU6YQIhf/9HCGpF5eH6WjkrmZEUBpcuMLZ/kGg
	UcPpxwzr4XyJi1+WXbTO/AQAAA==]]>
   </annotation>

<comment>

This element represents a Comment and is used for providing additional label-text pair with an annotation.

comment Attributes
Name Value Required? Comments
labelComment label yes

The text must be put in a CDATA section as child text element.

Example
    <annotation name="My Error Annotation" level="error" description="demo of no-attachment error-level annotation"/>
        <!-- comment must be a child of an annotation element -->
	<comment label="Comment Label">
            <![CDATA[Comment contents.]]>
        </comment>
    </annotation>

<custom_data>

This element represents a Custom Data and is used for including custom metrics into your report (i.e. test_suite or test_case element) so that you can track them in Testspace.1)

Custom data comprises a name and value. The values must be a comma separated list of numbers.

custom_data Attributes
Name Value Required? Comments
nameCustom data name yes

The value must be put in a CDATA section as child text element. You can specify one, two, or three numbers separated by commas. The numbers can be whole integers, floating point numbers, or a mix of the two.

Example
    <!-- custom_data with one number value -->
    <custom_data name="CustomValue1"><![CDATA[12]]></custom_data>
    <!-- custom_data with two floating point numbers value -->
    <custom_data name="CustomValue2"><![CDATA[12.1,12.456]]></custom_data>
    <!-- custom_data with three numbers value -->
    <custom_data name="CustomValue3"><![CDATA[12.25,1,13]]></custom_data>

Formal XML Schema Definition

TestspaceReport.xsd
<?xml version="1.0"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
 
<xs:complexType name="commentType">
    <xs:simpleContent>
        <xs:extension base="xs:string">
            <xs:attribute name="label" type="xs:string" use="required" />
         </xs:extension>
    </xs:simpleContent>
</xs:complexType>
 
<xs:complexType name="customdataType">
    <xs:simpleContent>
        <xs:extension base="xs:string">
            <xs:attribute name="name" type="xs:string" use="required" />
        </xs:extension>
    </xs:simpleContent>
</xs:complexType>
 
<xs:complexType name="annotationType" mixed="true">
    <xs:sequence minOccurs="0" maxOccurs="unbounded">
        <xs:element name="comment" type="commentType"/>
    </xs:sequence>
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="description" type="xs:string"/>
    <xs:attribute name="file" type="xs:string"/>
    <xs:attribute name="level" use="required">
        <xs:simpleType>
            <xs:restriction base="xs:string">
                <xs:enumeration value="info"/>
                <xs:enumeration value="warn"/>
                <xs:enumeration value="error"/>
            </xs:restriction>
        </xs:simpleType>
    </xs:attribute>
    <xs:attribute name="link_file" type="xs:boolean"/>
    <xs:attribute name="file_name" type="xs:string"/>
    <xs:attribute name="mime_type" type="xs:string"/>
</xs:complexType>
 
<xs:complexType name="testcaseType">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element name="custom_data" type="customdataType"/>
        <xs:element name="annotation" type="annotationType"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="description" type="xs:string"/>
    <xs:attribute name="start_time" type="xs:string" />
    <xs:attribute name="status" use="required">
        <xs:simpleType>
            <xs:restriction base="xs:string">
                <xs:enumeration value="passed"/>
                <xs:enumeration value="failed"/>
                <xs:enumeration value="unknown"/>
                <xs:enumeration value="in_progress"/>
                <xs:enumeration value="not_applicable"/>
                <xs:enumeration value="ignored"/>
            </xs:restriction>
        </xs:simpleType>
    </xs:attribute>
    <xs:attribute name="duration" type="xs:float"/>
</xs:complexType>
 
<xs:complexType name="testsuiteType">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element name="test_case" type="testcaseType"/>
        <xs:element name="test_suite" type="testsuiteType"/>
        <xs:element name="custom_data" type="customdataType"/>
        <xs:element name="annotation" type="annotationType"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="description" type="xs:string" />
    <xs:attribute name="start_time" type="xs:string" />
    <xs:attribute name="unknown" type="xs:unsignedInt" />
    <xs:attribute name="in_progress" type="xs:unsignedInt" />
    <xs:attribute name="passed" type="xs:unsignedInt" />
    <xs:attribute name="failed" type="xs:unsignedInt" />
    <xs:attribute name="not_applicable" type="xs:unsignedInt" />
    <xs:attribute name="duration" type="xs:float"/>
</xs:complexType>
 
<xs:element name="reporter">
    <xs:complexType>
        <xs:choice minOccurs="1" maxOccurs="unbounded">
            <xs:element name="test_suite" type="testsuiteType"/>
            <xs:element name="annotation" type="annotationType"/>
        </xs:choice>
        <xs:attribute name="product_version" type="xs:string"/>
        <xs:attribute name="schema_version" type="xs:string"/>
    </xs:complexType>
</xs:element>
 
</xs:schema>
1) Within Testspace, you can configure historical tracking and notification of custom metrics.

Page Tools