User Tools



Add To CI

Pushing data to Testspace from your Continuous Integration (CI) process is the best way to generate the data necessary for tracking test results and other ancillary data of interest. Adding the Testspace Client to your CI process is straightforward and should only require a small update to your CI script. There are three general steps to using the Testspace client in your CI:

  1. Download the client
  2. Configure the Testspace URL
  3. Push the results to the server

The specifics of configuring and pushing are dictated by the type of Testspace Project, connected or standalone.

Connected Project

Connected Projects provide Insights into the efficiency and effectiveness of your Git Forking and Branching workflows. Projects connect to GitHub, Bitbucket or GitLab repositories, with new Spaces automatically created for every new branch. Test results and other content are pushed to the Testspace server on every commit. Connected Projects work with all standard CI services, such as Travis, AppVeyor and Circle CI.

Standalone Project

Standalone Projects and their Spaces – as the name suggests – stand alone, unconnected from Git services, and can be used for all types of unit, integration and system level testing. Results can be pushed manually on the console or from any type of automation including CI.


Connected Project

The following sections overview using a CI system with a Testspace Connected Project. Once services are connected with GitHub, Bitbucket or GitLab, a Testspace Connected Project will automatically map to the associated repository.

Download the Client

When using Online CI services we recommend adding the client download to the CI script directly to ensure that you're using the latest version.

Linux

curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin

Windows PowerShell5

Invoke-WebRequest https://testspace-client.s3.amazonaws.com/testspace-windows.zip -outfile testspace-windows.zip
Expand-Archive -LiteralPath testspace-windows.zip -DestinationPath ./

OSX

curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-darwin.tgz | tar -zxvf- -C $HOME/bin

Configure the Client

Public Repos do NOT require an access token if using TravisCI, Shippable, AppVeyor, Bitbucket Pipelines or Circle CI.

The Testspace URL defines your access credentials – required to push data to the Testspace Server – and the destination for the results when pushed. Use of an access token for authentication, found in the User's settings, is recommended. The token is best set as an environment variable, ideally stored securely to hide your access credentials.

testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com

Public Repo example (regarding Travis, Shippable, AppVeyor, Circle CI, and Bitbucket Pipelines)

testspace config url subdomain.testspace.com

The Testspace Client config command will determine the Project and Space names from your Git organization/team, repo, and branch for creating the url. The Testspace Client will also collect source code changes and provide a link back to the build results in your CI.

Push Results to the Server

The CI commit status is checked to make sure the CI is finished to ensure results were properly gathered and fully reported.

The push command – the default command for the Testspace Client – aggregates all CI results data into a single record of health while pushing the content to the Testspace Server.

testspace analysis.xml test*.xml coverage.xml 

Matrix Support

CI Matrix is only supported in Connected Projects

Many CIs (e.g. Travis CI) support matrix builds to reduce testing time across different distributions and environments. When pushing test results from each build in a matrix, Testspace aggregates all test results into a single record of health. To help organize and manage all of the results content that can be produced from running matrix builds there are a few things to consider.

Each test case from each build in the matrix should be viewed as a unique PASS | FAIL result. Therefore, all test results from all builds in the matrix should be pushed.
Having metric badges and charts (static analysis, code coverage, custom) for every build in the matrix adds unnecessary clutter and redundancy to the database and dashboards for your projects. Selecting one representative metric set from one of the builds in the matrix makes it much easier to view and track metrics over time.

Use Folders to organize results

Use Folders to represent Jobs

Folders provide a natural organization for content pushed from multiple builds. Environment variables – both system and user-defined – can standardize the naming process. The destination path to create is defined using square braces, prefixed to the results file to push.

testspace [$BUILD/$LANGUAGE/test]test*.xml

Use Conditional Build Steps

Many CI systems allow for you to use conditional build steps to selectively choose when and which files to push, with this example being from Travis CI.

env:
  matrix:
   - JOB=ONE
   - JOB=TWO
after_script:
  |  
  if [ $JOB = "ONE" ]; then
    testspace [$JOB]test*.xml 
  elif [ $JOB = "TWO" ]; then
    testspace [$JOB]test*.xml 
  else
    echo "Unknown job" 
  fi

Test results (test*.xml) are uploaded for each version into a subfolder named Tests. This folder is used to organize the multiple folders and test suites that exist in most test result files (depending on language and test framework used).

For more information on creating folders refer here.


Travis CI

Examples of updates required in a .travis.yml for pushing results to Testspace:

before_install:
  - mkdir -p $HOME/bin
  - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
  - testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com
:!: Where TESTSPACE_TOKEN must be defined for private projects as an encrypted environment variable in .travisci.yml or encrypted in settings with value equal to your Testspace access-token.
after_script:
  - testspace test*.xml 

Note that Travis CI includes $HOME/bin in the environment $PATH by default, even though the directory does not exist.

Matrix Examples

Multiple Language Versions Matrix

An example .travis.yml file that builds with two versions of Node.js. The conditional push commands are controlled by the Travis specific TRAVIS_NODE_VERSION environment variable.

  language: node_js
  node_js:
    - "4"
    - "6"

  # fast_finish must be in the default false state to allow all jobs to push results to Testspace before build status is set.
  matrix:
    fast_finish: false
  after_script:
    - export FOLDER=node_js_.$TRAVIS_NODE_VERSION
    - |
      if [ "${TRAVIS_NODE_VERSION}" = 6 ]; then
        testspace [$FOLDER]analysis.xml [$FOLDER/Tests]test*.xml [$FOLDER]coverage.xml [$FOLDER]metrics.txt{metrics.csv}
      else 
        testspace [$FOLDER/Tests]test*.xml 
      fi

Multiple Language Versions With Environment Variables Matrix

An example using explicit inclusion to define a PUSH_ALL_RESULTS environment variable for one version only. Each job in matrix.include must also have the python value defined:

  language: python
  python:
    - '3.5'
    - '3.4'
    - '2.7'

  matrix:
    include:
      - python: '3.5'
        env: PUSH_ALL_RESULTS=true
      - python: '3.4'
      - python: '2.7'
  script: ./test.py $TEST_SUITE

  after_script: 
  - export FOLDER=python_.$TRAVIS_PYTHON_VERSION
  - | 
    if [ "${PUSH_ALL_RESULTS}" ]; then
      testspace [$FOLDER]analysis.xml [$FOLDER/Tests]test*.xml [$FOLDER]coverage.xml [$FOLDER]metrics.txt{metrics.csv}
    else 
      testspace [$FOLDER/Tests]test*.xml 
    fi

[n x n] Matrix

The following example defines a 3 x 5 matrix of Ruby and Rails versions.

  language: ruby
  rvm:
    - 2.1
    - 2.2.3
    - 2.3.3

  gemfile:
    - Gemfile.rails32
    - Gemfile.rails40
    - Gemfile.rails41
    - Gemfile.rails42
    - Gemfile.rails50
  after_script:
  - export RAILS_VER=$(echo $BUNDLE_GEMFILE | sed -r 's/.*Gemfile.(\S+)/\1/g')
  - export FOLDER=ruby_${TRAVIS_RUBY_VERSION}.$RAILS_VER
  - |
    if [[ "$RAILS_VER" = "rails42" && "$TRAVIS_RUBY_VERSION" = "2.3.3" ]]; then
      bundle exec rubocop --require rubocop/formatter/checkstyle_formatter --format RuboCop::Formatter::CheckstyleFormatter  --out tmp/checkstyle.xml || true 
      testspace [$FOLDER/test]$PWD/test/reports/test*.xml  [$FOLDER]tmp/checkstyle.xml  [$FOLDER]coverage/coverage.xml
    else
      testspace [$FOLDER/test]$PWD/test/reports/test*.xml
    fi

Circle CI

1.0

Examples of updates required in a circle.ymlfor pushing results to Testspace:

dependencies:
  pre:
    - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
    - testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com
:!: Where TESTSPACE_TOKEN must be defined for private projects as an environment variable (and preferably encrypted) with value equal to your Testspace access-token.
test:
  post:
    - testspace test*.xml 

Note that Circle CI creates the $HOME/bin directory and adds it to the environment $PATH by default.

2.0

Examples of updates required in a .circleci/config.ymlfor pushing results to Testspace:

steps:
...
# Add the following lines to the steps section after "checkout" but before any tests or static analysis is performed.
  - run: |
      mkdir -p $HOME/bin
      echo 'export PATH=$PATH:$HOME/bin' >> $BASH_ENV
      curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
  - run: testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com
:!: Where TESTSPACE_TOKEN must be defined for private projects as a secure environment variable with value equal to your Testspace access-token.
# Add the push command after all tests and static analysis has been done.
  - run:
      command: testspace test*.xml 
      when: always

Parallelism

TBD


AppVeyor

Examples of updates required in a .appveyor.yml for pushing results to Testspace:

install:
  - appveyor DownloadFile https://testspace-client.s3.amazonaws.com/testspace-windows.zip
  - 7z x -y testspace-windows.zip
  - testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com
:!: Where TESTSPACE_TOKEN must be defined for private projects as an secure environment variable with value equal to your Testspace access-token.
after_test:
  - testspace test*.xml 

Matrix Examples

Pushing Subset of Jobs

An example appveyor.yml file filtering on a subset of Jobs to push results.

environment:
  matrix:
   - TOXENV: "coverage"
   - TOXENV: "linting"
   - TOXENV: "py26"
   - TOXENV: "py27"
   - TOXENV: "py33"
   - TOXENV: "py34"
   - TOXENV: "py35"
   - TOXENV: "py36"
   - TOXENV: "pypy"
after_test:
  - ps: | 
      if (${env:TOXENV} -match 'py' -or ${env:TOXENV} -eq 'coverage')
      {
        .\testspace "[$env:TOXENV]test*.xml"
      }

Shippable

Examples of updates required in a shippable.yml for pushing results to Testspace:

build:
  ci: 
    - mkdir -p $HOME/bin
    - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
    - testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com
:!: Where TESTSPACE_TOKEN must be defined for private projects as an environment variable (and preferably encrypted) with value equal to your Testspace access-token.

The push command must be added to both on_success and on_failure sections as post_ci only runs on successful builds.

  on_success:
    - testspace test*.xml 
  on_failure:
    - testspace test*.xml 

Note that Shippable CI includes $HOME/bin in the environment $PATH by default, even though the directory does not exist.


Codeship

Pro

Changes in Dockerfile to install and configure Testspace.

ARG TESTSPACE_TOKEN

RUN curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
RUN testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com/
:!: Where TESTSPACE_TOKEN build argument must be defined (and preferably encrypted) in the codeship-services.yml file with value equal to your Testspace access-token.
app:
  build:
    dockerfile: Dockerfile
    args: 
      TESTSPACE_TOKEN: access-token

In the codeship-steps.yml file for each step that generates results for pushing to Testspace wrap the existing test command using the Testspace –command option.

steps:
  - command: testspace test*.xml --command "run test command"
:!: Note: Codeship has a bug that incorrectly interprets command line quotes and may result if errors like "Syntax error: Unterminated quoted string". If your "run test command" is a standalone script with no arguments (no white spaces) do not surround it with quotations.

Basic

Add the following to Setup Commands box in Codeship settings Test tab to install and configure Testspace.

mkdir -p $HOME/bin
curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
testspace config url ${TESTSPACE_TOKEN}:@sub-domain.testspace.com/
:!: Where TESTSPACE_TOKEN must be defined as an environment variable with value equal to your Testspace access-token.

To the Configure Test Pipelines box in Test tab wrap your existing test command(s) in the Testspace –command. This allows for the results to be pushed in the event of a test failure.

testspace test*.xml --command="run test command"

GitLab CI

Examples of updates required in a .gitlab-ci.yml for pushing results to Testspace as part of a single job example for GitLab:

before_script:
  - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
  - testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com
:!: Where TESTSPACE_TOKEN must be defined for private projects as an environment variable (and preferably encrypted) with value equal to your Testspace access-token.
job1:
  script:
    - testspace test*.xml 

Multiple Jobs

The following is a simple example showing multiple Jobs, using the Job Names as Folder Names.

before_script:
  - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
  - testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com
stages:
  - test

job1:
  stage: test
  script:
    - testspace [$CI_JOB_NAME]test*.xml

job2:
   stage: test
   script: 
     - testspace [$CI_JOB_NAME]test*.xml 

Bitbucket Pipelines

Examples of updates required in a bitbucket-pipelines.yml for pushing results to Testspace as part of a single step example for Bitbucket:

script:
pipelines:
  default:
    - step:
        name: test
        script: 
          - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
          - testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com
          - testspace test*.xml --command="run test command"
:!: Where TESTSPACE_TOKEN must be defined as an environment variable (and preferably as a secured variable) with value equal to your Testspace access-token.

Multiple Jobs

The following is a simple example showing multiple Steps, using the step name as Folder Names.

pipelines:
  default:
    - step:
        name: test_a
        script: 
          - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
          - testspace config url ${TESTSPACE_TOKEN}:@samples.testspace.com
          - testspace [test_a]test*.xml --command="run test command"
    - step:
        name: test_b
        script: 
          - curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C /usr/local/bin
          - testspace config url ${TESTSPACE_TOKEN}:@samples.testspace.com
          - testspace [test_b]test*.xml --command="run test command"

Jenkins

1.x

Depending on your Jenkins setup, the three steps to using the Testspace client – download, configure and push – can be done differently. Here is an example using the EnvInject and PostBuildScript plugins.

Build Environment

[X] Inject passwords to the build as environment variables
  Job passwords:
    Name: TESTSPACE_TOKEN
    Password: access-token
[X] Inject environment variables to the build process
  Properties Content:
    Path=$Path:$HOME/bin
  Script Content:
    curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin  
    testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com

Post-Build Actions

Execute a set of scripts:
  Build steps | Execute shell:
    testspace test*.xml ?finish

2.x (Pipelines)

Examples of updates required in a Jenkinsfile for pushing results to Testspace:

pipeline {
...
  stages {
    stage('Test') {
      environment {
        TESTSPACE_TOKEN = 'access-token'
      }
      steps {
        sh 'curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin'
        sh 'testspace config url ${TESTSPACE_TOKEN}:@subdomain.testspace.com'
...        
      }
      post {
        always {
          sh 'testspace test*.xml ?finish'
        }
      }
    }
...
  }
}

Custom

Config is required before pushing content in connected projects.

A custom CI requires specific setting when working with a Connected Project. The Testspace Client requires CI=true to be defined when using config option. Also –repo=git is required to enable source code tracking.

mkdir -p $HOME/bin
curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin
CI=true testspace config url access-token:@subdomain.testspace.com
testspace test*.xml --repo=git

Standalone Project

Standalone Projects also work with Online CIs. See below for examples.

The following sections overview using a CI system with a Testspace Standalone Project. Standalone Projects require that the project name and the space name be provided when pushing data with the Testspace Client.

Download the Client

There are several methods to install the Testspace Client, depending on your CI machine's environment. When using Online CI services we recommend adding the client download to the CI script directly to ensure that you're using the latest version.

Linux

curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-linux.tgz | tar -zxvf- -C $HOME/bin

Windows PowerShell

Invoke-WebRequest https://testspace-client.s3.amazonaws.com/testspace-windows.zip -outfile testspace-windows.zip
Expand-Archive -LiteralPath testspace-windows.zip -DestinationPath ./

OSX

curl -fsSL https://testspace-client.s3.amazonaws.com/testspace-darwin.tgz | tar -zxvf- -C $HOME/bin

Configure Testspace URL

The Testspace URL defines your access credentials – required to push data to the Testspace Server – and the destination for the results when pushed. Using an access token for authentication, found in the User's settings, is required. The token is best set as an environment variable, ideally stored securely to hide your access credentials.

testspace config url access-token:@subdomain.testspace.com/my-project/my-space

For more information on configuring the client refer here.

Push Results to the Server

Add source code change tracking using the repo option

testspace test*.xml --repo=git

The push command – the default command for the Testspace Client – aggregates all CI results data into a single record of health while pushing the content to the Testspace Server:

testspace analysis.xml test*.xml coverage.xml --repo=git

How To


Page Tools