Skip to main content

CI/CD Support

Pushing data to Testspace from your Continuous Integration (CI) process is the best way to generate the data necessary for tracking test results and other ancillary data of interest. Adding the Testspace Client to your CI process is straightforward and should only require a small update to your CI script. There are three general steps to using the Testspace client in your CI:

  • Download the client
  • Configure the Testspace URL
  • Push the results to the server

Overview​

The specifics of configuring and pushing are dictated by the type of Testspace Project, connected or standalone.

Connected Projects provide Insights into the efficiency and effectiveness of your Git Forking and Branching workflows. Projects connect to GitHub, Bitbucket or GitLab repositories, with new Spaces automatically created for every new branch. Test results and other content are pushed to the Testspace server on every commit. Connected Projects work with all standard CI services, such as Travis, AppVeyor and Circle CI.

Standalone Projects and their Spaces – as the name suggests – stand-alone, unconnected from Git services, and can be used for all types of unit, integration and system-level testing. Results can be pushed manually on the console or from any type of automation including CI.

Connected​

Once services are connected with GitHub, Bitbucket or GitLab, a Testspace Connected Project will automatically map to the associated repository.

Download​

When using Online CI services we recommend adding the client download to the CI script directly to ensure that you're using the latest version. Refer to the client download documentation.

Configure​

The Testspace URL defines the destination for the results when pushed. Use of an token for authentication, found in the Project's settings (push token) or in the User's settings (access token), is required for private Repos. The token should be set as an environment variable named TESTSPACE_TOKEN, ideally stored securely to hide your access credentials.

$ testspace config url YOUR-SUBDOMAIN.testspace.com
tip

Public Repos do NOT require a token if using GitHub Actions, TravisCI, Circle CI, AppVeyor, Azure Pipelines, Bitbucket Pipelines, or GitLab CI.

The Testspace Client config command will determine the Project and Space names from your Git organization/team, repo, and branch for creating the url. The Testspace Client will also collect source code changes and provide a link back to the build results in your CI.

Push​

The push command (default) for the Testspace Client – aggregates all CI results data into a single record of results while pushing the content to the Testspace Server.

Matrix​

Many CIs (e.g. Travis CI) support matrix builds and/or parallel jobs to reduce testing time across different distributions and environments. When pushing test results from each build in a matrix, Testspace aggregates all test results into a single record of health. To help organize and manage all of the results content that can be produced from running matrix builds there are a few things to consider.

CI Matrix is only supported in Connected Projects

Each test case from each build in the matrix should be viewed as a unique PASS | FAIL result. Therefore, all test results from all builds in the matrix should be pushed.

But having metric badges and charts (static analysis, code coverage, custom) for every build (job) in the matrix adds unnecessary clutter and redundancy to the dashboard for your projects. Selecting one representative metric set from one of the builds (jobs) in the matrix makes it much easier to view and track metrics over time.

Use Folders to organize job results​

Folders provide a natural organization for content pushed from multiple jobs. Environment variables – both system and user-defined – can standardize the naming process. The destination path to create is defined using square braces, prefixed to the results file to push.

testspace [$JOB/$LANGUAGE/test]test*.xml

Test results (e.g. test*.xml) are uploaded for each version into a subfolder named Tests. This folder is used to organize the multiple folders and test suites that exist in most test result files (depending on language and test framework used).

env:
  matrix:
   - JOB=ONE
   - JOB=TWO
after_script: |
  if [ $JOB = "ONE" ]; then
    testspace [$JOB]analysis.xml [$JOB/Tests]test*.xml [$JOB]coverage.xml [$JOB]metrics.txt{metrics.csv}
  elif [ $JOB = "TWO" ]; then
    testspace [$JOB/Tests]test*.xml
  else
    echo "Unknown job"
  fi

Custom​

A custom CI requires specific setting when working with a Connected Project.

Connected Project​

Define the following environment variables:

CI=true # required
CI_BUILD_NUMBER=<unique identifier of the running build> # required
CI_BUILD_URL=<URL of the running build> # optional
CI_COMMIT_SHA=<commit sha being build> # optional
CI_REPO=<repo being build> # optional
CI_COMMIT_BRANCH=<branch being build> # optional
TESTSPACE_TOKEN=<secure access token to authenticate access to Testspace> # required

Download and configure the Testspace client (preferably in the beginning of your build):

mkdir -p $HOME/bin
curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
testspace config url subdomain.testspace.com

config is required before pushing content in connected projects.

Push results to the Testspace server:

testspace test*.xml ?finish

Standalone Project​

The following sections overview using a CI system with a Testspace Standalone Project. Standalone Projects require that the project name and the space name be provided when pushing data with the Testspace Client.

Download​

We recommend adding the client download to the CI script directly to ensure that you're using the latest version. Refer to the client download documentation.

Config​

The Testspace URL defines your access credentials – required to push data to the Testspace Server – and the destination for the results when pushed.

Use of an token for authentication, found in the Project's settings push token or in the User's settings access token, is required. The token is best set as an environment variable, ideally stored securely to hide your access credentials.

testspace config url access-token:@subdomain.testspace.com/my-project/my-space
Push​

The push command – the default command for the Testspace Client – aggregates all CI results data into a single record of health while pushing the content to the Testspace Server:

testspace analysis.xml test*.xml coverage.xml --repo=git

To track your source code change the --repo option must be explicitly specified.

To link your results back to your CI build system the --build-url option must be explicitly specified.

Refer to the client for more details.

AppVeyor​

Examples of updates required in a .appveyor.yml for pushing results to Testspace:

install:
  - appveyor DownloadFile https://testspace-client.s3.amazonaws.com/testspace-windows.zip
  - 7z x -y testspace-windows.zip
  - testspace config url subdomain.testspace.com

TESTSPACE_TOKEN must be defined for private projects as a secure environment variable with value equal to your Testspace access-token.1

after_test:
  - testspace test*.xml

Azure Pipelines​

Examples of updates required in a azure-pipelines.yml for pushing results to Testspace:

steps:
# Add the following lines to the steps section before any tests or static analysis is performed.
# Linux
  - script: |
      curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | sudo tar -zxvf- -C /usr/local/bin
      testspace config url subdomain.testspace.com
    displayName: Install Testspace client

# Windows
  - bash: |
      curl -OsSL https://testspace-client.s3.amazonaws.com/testspace-windows.zip
      unzip -q -o testspace-windows.zip
      ./testspace config url subdomain.testspace.com
    displayName: Install Testspace client

TESTSPACE_TOKEN must be defined for private projects as a secret environment variable with value equal to your Testspace access-token.1

# Add the push command after all tests and static analysis have been done.
  - script: |
      testspace test*.xml
    condition: always()
    displayName: Push result to Testspace server

Bitbucket Pipelines​

Examples of updates required in a bitbucket-pipelines.yml for pushing results to Testspace as part of a single step example for Bitbucket:

script:
pipelines:
  default:
    - step:
        name: test
        script:
# Add the following lines to the script section before any tests or static analysis is performed.
          - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
          - testspace config url subdomain.testspace.com
        after-script:
          - testspace test*.xml

TESTSPACE_TOKEN must be defined for private projects as a secured environment variable with value equal to your Testspace access-token.1

Buildkite​

Examples of updates required in a .buildkite/pipeline.yml for pushing results to Testspace:

steps:
  - commands:
    - "test-script.sh"
    - |
      # Add the push command after all tests and static analysis have been done.
      curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | sudo tar -zxvf- -C /usr/local/bin
      testspace config url subdomain.testspace.com
      testspace test*.xml

TESTSPACE_TOKEN must be defined as a secret environment variable with value equal to your Testspace access-token.1

Circle CI​

Examples of updates required in a .circleci/config.yml for pushing results to Testspace:

steps:
  - checkout
# Add the following lines to the steps section after "checkout" but before any tests or static analysis is performed.
  - run:
      name: Install Testspace client
      command: |
        curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | sudo tar -zxvf- -C /usr/local/bin
        testspace config url subdomain.testspace.com

TESTSPACE_TOKEN must be defined for private projects as a secret environment variable with value equal to your Testspace access-token.1

# Add the push command after all tests and static analysis have been done.
  - run:
      name: Push result to Testspace server
      command: testspace test*.xml
      when: always

Cirrus CI​

Examples of updates required in a .cirrus.yml for pushing results to Testspace:

setup_task_template: &SETUP_TASK_TEMPLATE
  environment:
    TESTSPACE_TOKEN: ENCRYPTED[testspace-token-var-id]
  setup_testspace_script:
    - curl -s https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
    - testspace config url subdomain.testspace.com

TESTSPACE_TOKEN must be defined for private projects as a encrypted variable with value equal to your Testspace access-token.1

test_task:
  << : *SETUP_TASK_TEMPLATE

# Add the push command after all tests and static analysis have been done.
  always:
    push_testspace_script:
      - testspace test*.xml

Codefresh​

Example of the update required in a codefresh.yml for pushing results to Testspace:

steps:
#...
# Add freestyle step after all tests and static analysis have been done.
  testspace:
    stage: test
    type: freestyle
    image: buildpack-deps:18.04-scm
    commands:
      - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
      - testspace config url subdomain.testspace.com
      - testspace test*.xml
    when:
      condition:
        all:
          always: "true"

TESTSPACE_TOKEN must be defined for private projects as a encrypted variable with a value equal to your Testspace access-token.1

Drone​

Example of the update required in a .drone.yml for pushing results to Testspace:

steps:
#...
# Add freestyle step after all tests and static analysis have been done.
- name: testspace
  image: buildpack-deps:18.04-scm
  commands:
    - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
    - testspace config url subdomain.testspace.com
    - testspace test*.xml
  environment:
    TESTSPACE_TOKEN:
      from_secret: testspace_token
  when:
    status: [ failure, success ]

TESTSPACE_TOKEN must be defined for private projects as a secret with a value equal to your Testspace access-token.1

GitHub Actions​

Example updates required in a .github/workflows/ci.yml for pushing results to Testspace:

NOTE

Applicable to self hosted Windows runners, for workflow bash shell steps to properly behave, the "Git for Windows" installation path (e.g. C:\Program Files\Git\bin) need to added to the system PATH. If WSL is installed, the "Git for Windows" installation path needs to be placed before C:\Windows\System32 - see https://github.com/actions/runner/issues/1328.

NOTE

Applicable to self hosted runners, for workflows triggered by pull_requests to be able to properly obtain the originating PR-number, Testspace requires the jq utility to be installed on your OS image (named jq/jq.exe and on the system PATH) .

steps:
- uses: actions/checkout@v3
- uses: testspace-com/setup-testspace@v1
  with:
    domain: YOUR-ORG-NAME # usually set to just ${{github.repository_owner}}
    token: ${{ secrets.TESTSPACE_TOKEN }} # optional, only required for private repos

The token must be defined for private projects as a secret environment variable with value equal to your Testspace access-token.1.

Note that the actions/checkout step is required before the testspace setup action for the GitHub checks to be included.

# Add the `push` command after all tests and static analysis have been done.
  - name: Push result to Testspace server
    run: |
      testspace test*.xml
    if: always()

Matrix​

When using a matrix it is recommended to use a folder to store the results specific to each matric entry.

name: CI
on: push
jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    steps:
      ..
      - name: Publish Results to Testspace
        run: testspace "[ ${{ matrix.os }} ]results.xml"

GitLab CI​

Examples of updates required in a .gitlab-ci.yml for pushing results to Testspace as part of a single job example for GitLab:

before_script:
  - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
  - testspace config url subdomain.testspace.com

TESTSPACE_TOKEN must be defined for private projects as a protected environment variable with value equal to your Testspace access-token.1

job1:
  script:
    - testspace test*.xml

Jenkins​

1.x​

Depending on your Jenkins setup, the three steps to using the Testspace client – download, configure and push – can be done differently. Here is an example using the EnvInject and PostBuildScript plugins.

Build Environment​
[X] Inject passwords to the build as environment variables
  Job passwords:
    Name: TESTSPACE_TOKEN
    Password: access-token
[X] Inject environment variables to the build process
  Properties Content:
    Path=$Path:$HOME/bin
  Script Content:
    curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
    testspace config url subdomain.testspace.com
Post-Build Actions​
Execute a set of scripts:
  Build steps | Execute shell:
    testspace test*.xml ?finish

2.x (Pipelines)​

Assuming the GitHub Branch Source Plugin is setup, here is an example of updates required in a Jenkinsfile for pushing results to Testspace:

pipeline {
...
  stages {
    stage('Test') {
      environment {
        TESTSPACE_TOKEN = credentials('testspace-token')
      }
      steps {
        sh 'curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin'
        sh 'testspace config url subdomain.testspace.com'
...
      }
      post {
        always {
          sh 'testspace test*.xml'
        }
      }
    }
...
  }
}

TESTSPACE_TOKEN must be defined as a "secret text" credentials with value equal to your Testspace access-token.1

Semaphore CI​

Examples of updates required in a .semaphore/semaphore.yml for pushing results to Testspace:

blocks:
  - task:
      prologue:
        commands:
          - checkout
          - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | sudo tar -zxvf- -C /usr/local/bin
          - testspace config url subdomain.testspace.com

TESTSPACE_TOKEN must be defined as a secret environment variable with value equal to your Testspace access-token.1

      epilogue:
        always:
          commands:
            - testspace test*.xml
NOTE

For auto result completion to properly work, status checks needs to be configured on pipeline-level.

Travis CI​

Examples of updates required in a .travis.yml for pushing results to Testspace:

before_install:
  - mkdir -p $HOME/bin
  - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
  - testspace config url subdomain.testspace.com

Note that Travis CI includes $HOME/bin in the environment $PATH by default, even though the directory does not exist.

TESTSPACE_TOKEN must be defined for private projects as an encrypted environment variable with value equal to your Testspace access-token.1

after_script:
  - testspace test*.xml
info

In matrix builds, to allow all jobs to push results to Testspace before build status is set, fast_finish should not be enabled.

  1. You can use a User Access token or use a specific Project Push token.↩