Skip to main content

CI/CD Support

Pushing data to Testspace from your Continuous Integration (CI) process is the best way to generate the data necessary for tracking test results and other ancillary data of interest. Adding the Testspace Client to your CI process is straightforward and should only require a small update to your CI script. There are three general steps to using the Testspace client in your CI:

  • Download the client
  • Configure the Testspace URL
  • Push the results to the server

Overview#

The specifics of configuring and pushing are dictated by the type of Testspace Project, connected or standalone.

Connected Projects provide Insights into the efficiency and effectiveness of your Git Forking and Branching workflows. Projects connect to GitHub, Bitbucket or GitLab repositories, with new Spaces automatically created for every new branch. Test results and other content are pushed to the Testspace server on every commit. Connected Projects work with all standard CI services, such as Travis, AppVeyor and Circle CI.

Standalone Projects and their Spaces – as the name suggests – stand-alone, unconnected from Git services, and can be used for all types of unit, integration and system-level testing. Results can be pushed manually on the console or from any type of automation including CI.

Connected#

Once services are connected with GitHub, Bitbucket or GitLab, a Testspace Connected Project will automatically map to the associated repository.

Download#

When using Online CI services we recommend adding the client download to the CI script directly to ensure that you're using the latest version. Refer to the client download documentation.

Configure#

The Testspace URL defines the destination for the results when pushed. Use of an access token for authentication, found in the User's settings, is required for private Repos. The token should be set as an environment variable named TESTSPACE_TOKEN, ideally stored securely to hide your access credentials.

$ testspace config url YOUR-SUBDOMAIN.testspace.com

Public Repos do NOT require an access token if using GitHub Actions, TravisCI, Circle CI, AppVeyor, Azure Pipelines, Bitbucket Pipelines, or GitLab CI.

The Testspace Client config command will determine the Project and Space names from your Git organization/team, repo, and branch for creating the url. The Testspace Client will also collect source code changes and provide a link back to the build results in your CI.

Push#

The push command (default) for the Testspace Client – aggregates all CI results data into a single record of results while pushing the content to the Testspace Server.

Matrix#

Many CIs (e.g. Travis CI) support matrix builds and/or parallel jobs to reduce testing time across different distributions and environments. When pushing test results from each build in a matrix, Testspace aggregates all test results into a single record of health. To help organize and manage all of the results content that can be produced from running matrix builds there are a few things to consider.

CI Matrix is only supported in Connected Projects

Each test case from each build in the matrix should be viewed as a unique PASS | FAIL result. Therefore, all test results from all builds in the matrix should be pushed.

But having metric badges and charts (static analysis, code coverage, custom) for every build (job) in the matrix adds unnecessary clutter and redundancy to the dashboard for your projects. Selecting one representative metric set from one of the builds (jobs) in the matrix makes it much easier to view and track metrics over time.

Use Folders to organize job results#

Folders provide a natural organization for content pushed from multiple jobs. Environment variables – both system and user-defined – can standardize the naming process. The destination path to create is defined using square braces, prefixed to the results file to push.

testspace [$JOB/$LANGUAGE/test]test*.xml

Test results (e.g. test*.xml) are uploaded for each version into a subfolder named Tests. This folder is used to organize the multiple folders and test suites that exist in most test result files (depending on language and test framework used).

env:
matrix:
- JOB=ONE
- JOB=TWO
after_script: |
if [ $JOB = "ONE" ]; then
testspace [$JOB]analysis.xml [$JOB/Tests]test*.xml [$JOB]coverage.xml [$JOB]metrics.txt{metrics.csv}
elif [ $JOB = "TWO" ]; then
testspace [$JOB/Tests]test*.xml
else
echo "Unknown job"
fi

Custom#

A custom CI requires specific setting when working with a Connected Project.

Connected Project#

Define the following environment variables:

CI=true # required
CI_BUILD_NUMBER=<unique identifier of the running build> # required
CI_BUILD_URL=<URL of the running build> # optional
CI_COMMIT_SHA=<commit sha being build> # optional
CI_REPO=<repo being build> # optional
CI_COMMIT_BRANCH=<branch being build> # optional
TESTSPACE_TOKEN=<secure access token to authenticate access to Testspace> # required

Download and configure the Testspace client (preferably in the beginning of your build):

mkdir -p $HOME/bin
curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
testspace config url subdomain.testspace.com

config is required before pushing content in connected projects.

Push results to the Testspace server:

testspace test*.xml ?finish

Standalone Project#

The following sections overview using a CI system with a Testspace Standalone Project. Standalone Projects require that the project name and the space name be provided when pushing data with the Testspace Client.

Download#

We recommend adding the client download to the CI script directly to ensure that you're using the latest version. Refer to the client download documentation.

Config#

The Testspace URL defines your access credentials – required to push data to the Testspace Server – and the destination for the results when pushed. Using an access token for authentication, found in the User's settings, is required. The token is best set as an environment variable, ideally stored securely to hide your access credentials.

testspace config url access-token:@subdomain.testspace.com/my-project/my-space
Push#

The push command – the default command for the Testspace Client – aggregates all CI results data into a single record of health while pushing the content to the Testspace Server:

testspace analysis.xml test*.xml coverage.xml --repo=git

To track your source code change the --repo option must be explicitly specified.

To link your results back to your CI build system the --build-url option must be explicitly specified.

Refer to the client for more details.

AppVeyor#

Examples of updates required in a .appveyor.yml for pushing results to Testspace:

install:
- appveyor DownloadFile https://testspace-client.s3.amazonaws.com/testspace-windows.zip
- 7z x -y testspace-windows.zip
- testspace config url subdomain.testspace.com

TESTSPACE_TOKEN must be defined for private projects as a secure environment variable with value equal to your Testspace access-token.1

after_test:
- testspace test*.xml

Azure Pipelines#

Examples of updates required in a azure-pipelines.yml for pushing results to Testspace:

steps:
# Add the following lines to the steps section before any tests or static analysis is performed.
# Linux
- script: |
curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | sudo tar -zxvf- -C /usr/local/bin
testspace config url subdomain.testspace.com
displayName: Install Testspace client
# Windows
- bash: |
curl -OsSL https://testspace-client.s3.amazonaws.com/testspace-windows.zip
unzip -q -o testspace-windows.zip
./testspace config url subdomain.testspace.com
displayName: Install Testspace client

TESTSPACE_TOKEN must be defined for private projects as a secret environment variable with value equal to your Testspace access-token.1

# Add the push command after all tests and static analysis have been done.
- script: |
testspace test*.xml
condition: always()
displayName: Push result to Testspace server

Bitbucket Pipelines#

Examples of updates required in a bitbucket-pipelines.yml for pushing results to Testspace as part of a single step example for Bitbucket:

script:
pipelines:
default:
- step:
name: test
script:
# Add the following lines to the script section before any tests or static analysis is performed.
- curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
- testspace config url subdomain.testspace.com
after-script:
- testspace test*.xml

TESTSPACE_TOKEN must be defined for private projects as a secured environment variable with value equal to your Testspace access-token.1

Circle CI#

Examples of updates required in a .circleci/config.yml for pushing results to Testspace:

steps:
- checkout
# Add the following lines to the steps section after "checkout" but before any tests or static analysis is performed.
- run:
name: Install Testspace client
command: |
curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | sudo tar -zxvf- -C /usr/local/bin
testspace config url subdomain.testspace.com

TESTSPACE_TOKEN must be defined for private projects as a secret environment variable with value equal to your Testspace access-token.1

# Add the push command after all tests and static analysis have been done.
- run:
name: Push result to Testspace server
command: testspace test*.xml
when: always

Codefresh#

Example of the update required in a codefresh.yml for pushing results to Testspace:

steps:
#...
# Add freestyle step after all tests and static analysis have been done.
testspace:
stage: test
type: freestyle
image: buildpack-deps:18.04-scm
commands:
- curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
- testspace config url subdomain.testspace.com
- testspace test*.xml
when:
condition:
all:
always: "true"

TESTSPACE_TOKEN must be defined for private projects as a encrypted variable with a value equal to your Testspace access-token.1

CodeShip#

Pro#

Changes in Dockerfile to install and configure Testspace.

ARG TESTSPACE_TOKEN
RUN curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
RUN testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com/

TESTSPACE_TOKEN build argument must be defined (and preferably encrypted) in the codeship-services.yml file with value equal to your Testspace access-token.1

app:
build:
dockerfile: Dockerfile
args:
TESTSPACE_TOKEN: access-token

In the codeship-steps.yml file for each step that generates results for pushing to Testspace wrap the existing test -command using the Testspace –command option.

steps:
- command: testspace test*.xml --command "run test command"

Note: Codeship has a bug that incorrectly interprets command line quotes and may result if errors like "Syntax error: Unterminated quoted string". If your "run test command" is a standalone script with no arguments (no white spaces) do not surround it with quotations.

Basic#

Add the following to Setup Commands box in Codeship settings Test tab to install and configure Testspace.

mkdir -p $HOME/bin
curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
testspace config url sub-domain.testspace.com/

TESTSPACE_TOKEN must be defined as an environment variable with value equal to your Testspace access-token.1

To the Configure Test Pipelines box in Test tab wrap your existing test command(s) in the Testspace –command. This allows for the results to be pushed in the event of a test failure.

testspace test*.xml --command="run test command"

Drone#

Example of the update required in a .drone.yml for pushing results to Testspace:

steps:
#...
# Add freestyle step after all tests and static analysis have been done.
- name: testspace
image: buildpack-deps:18.04-scm
commands:
- curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
- testspace config url subdomain.testspace.com
- testspace test*.xml
environment:
TESTSPACE_TOKEN:
from_secret: testspace_token
when:
status: [ failure, success ]

TESTSPACE_TOKEN must be defined for private projects as a secret with a value equal to your Testspace access-token.1

GitHub Actions#

Example updates required in a .github/workflows/ci.yml for pushing results to Testspace:

important

Note: Applicable to self hosted runners, for workflows triggered by pull_requests to be able to properly obtain the originating PR-number, Testspace requires the jq utility to be installed on your OS image.

steps:
- uses: actions/checkout@v2
- uses: testspace-com/setup-testspace@v1
with:
domain: YOUR-ORG-NAME
token: ${{ secrets.TESTSPACE_TOKEN }} # optional, only required for private repos

The token must be defined for private projects as a secret environment variable with value equal to your Testspace access-token.1.

# Add the `push` command after all tests and static analysis have been done.
- name: Push result to Testspace server
run: |
testspace test*.xml
if: always()

Matrix#

When using a matrix it is recommended to use a folder to store the results specific to each matric entry.

name: CI
on: push
jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
steps:
..
- name: Publish Results to Testspace
run: testspace "[ ${{ matrix.os }} ]results.xml"

GitLab CI#

Examples of updates required in a .gitlab-ci.yml for pushing results to Testspace as part of a single job example for GitLab:

before_script:
- curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
- testspace config url subdomain.testspace.com

TESTSPACE_TOKEN must be defined for private projects as a protected environment variable with value equal to your Testspace access-token.1

job1:
script:
- testspace test*.xml

Jenkins#

1.x#

Depending on your Jenkins setup, the three steps to using the Testspace client – download, configure and push – can be done differently. Here is an example using the EnvInject and PostBuildScript plugins.

Build Environment#
[X] Inject passwords to the build as environment variables
Job passwords:
Name: TESTSPACE_TOKEN
Password: access-token
[X] Inject environment variables to the build process
Properties Content:
Path=$Path:$HOME/bin
Script Content:
curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
testspace config url subdomain.testspace.com
Post-Build Actions#
Execute a set of scripts:
Build steps | Execute shell:
testspace test*.xml ?finish

2.x (Pipelines)#

Assuming the GitHub Branch Source Plugin is setup, here is an example of updates required in a Jenkinsfile for pushing results to Testspace:

pipeline {
...
stages {
stage('Test') {
environment {
TESTSPACE_TOKEN = credentials('testspace-token')
}
steps {
sh 'curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin'
sh 'testspace config url subdomain.testspace.com'
...
}
post {
always {
sh 'testspace test*.xml'
}
}
}
...
}
}

TESTSPACE_TOKEN must be defined as an "secret text" credentials with value equal to your Testspace access-token.1

Travis CI#

Examples of updates required in a .travis.yml for pushing results to Testspace:

before_install:
- mkdir -p $HOME/bin
- curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
- testspace config url subdomain.testspace.com

Note that Travis CI includes $HOME/bin in the environment $PATH by default, even though the directory does not exist.

TESTSPACE_TOKEN must be defined for private projects as an encrypted environment variable with value equal to your Testspace access-token.1

after_script:
- testspace test*.xml
important

In matrix builds, to allow all jobs to push results to Testspace before build status is set, fast_finish should not be enabled.


  1. You can generate and get an Access token from your User Settings