Skip to main content

Specification

Manual tests, known as test specs, are written using Markdown syntax. The markdown files (*.md) are maintained in the code section of the repo, by default in a folder named specs. To enable manual testing a, .testspace.yml configuration file is required to be placed at the root of your repo with the following minimum content:

.testspace.yml
manual:
important

Refer to .testspace.yml for details on how Testspace is configured to discover and manage testing sessions.

The following is a simple example directory structure containing 2 test specs:

root
├─ README.md
├─ .testspace.yml
└─ specs
└─ myspec1.md
└─ myspec2.md
..
..

The test specification model has largely been influenced by the Gauge automation framework, GitHub Flavored Markdown, and the Jekyll static websites framework.

tip

See Desktop Preview for developing and reviewing Test Specs on your desktop leveraging Jekyll.

Test Syntax

A manual test spec is a single markdown file consisting of:

  • A Front Matter header block
  • A test spec name
  • At least one test case
  • Optional test steps
/specs/test1.md
  ---
testspace:
---
# Test 1
## Case 1
- do this
- do that
## Case 2
..
tip

See the Markdown Cheat Sheet for helpful hints on leveraging markdown to capture test instructions.

Front Matter

To uniquely identify a markdown file as a Testspace spec file, a metadata block is required. This metadata is called front matter, denoted by the triple dashes (---) at the start and end of the block. It must be the first thing in the file, YAML formatted, containing the testspace: identifier.

---
testspace:
---

Disable Spec

To disable a test spec set the testspace token to false:

---
testspace: false
---

This setting prevents the spec from being selected on new test sessions. The existing status for the test spec will be maintained.

Remove Spec

To remove a test spec, without deleting it from the repo, comment out (#) the testspace token:

---
#testspace:
---

The existing status will be removed on the subsequent result.

Spec

Each spec also is required to contain a spec name written in the <H1> markdown syntax, following the front matter block. Any optional text after the spec name is the description of the spec.

---
testspace:
---

# My Spec

Spec description text

Case

A spec requires at least one test case. A case name is denoted by a markdown <H2> heading. Any optional text following the heading is the test case description.

# My Spec

## My Case
Case description text

## My Other Case

Step

Steps are optional instructions to be executed for a test case. Steps are written using a markdown (ordered or unordered) list inside a test case.

## My Case

* My step 1 instructions
* My step 2 instructions

Steps can also exist outside a test case, in which case they are meant to represent a repeated context that is applicable to each test case in the spec.

Fixtures

Test case Fixtures implement steps/conditions that are required for executing each case within the spec. They are defined using the same markdown syntax but named in a special way.

important

A fixture can be defined anywhere in the file, but only one time.

Setup is a fixture, defined as a set of instructions, to be presented before each case during execution.

A Setup fixture is defined using the ## [setup] syntax:

## [setup]
This is my setup fixture:

- First do this
- Next do that

The Setup fixture can be defined anywhere in the file, but only one time.

Teardown is a fixture, defined as a set of instructions, to be presented after each case during execution.

A Teardown fixture is defined using the ## [teardown]:

## [teardown]
This is my teardown fixture:

- Now do this
- And do that

Template Language

Testspace supports the template language called Liquid. Test specs are handled as template files, meaning they get preprocessed before being rendered. This functionality enables a test spec file to use variables, include files (i.e. subroutines) along with passing parameters, and even the ability to implement conditional logic.

tip

Testspace supports the template language called Liquid. Refer to this blog article for a Liquid overview.

Variables

There are two types of variables supported by Testspace: custom and global. Testspace traverses your spec files and processes variables.

Custom

User-defined "custom" variables are defined in the front matter block and are referenceable throughout the spec.

---
testspace:
var1: Hey there
---

When referencing a variable, the following syntax is required: {{ spec.VAR }}.

## My Case  

* My first step uses this {{ spec.var1 }}
* My Other Step

Global

The following global variables are supported: {{ variable }}.

VariableDescription
project.nameThe name of the Testspace project
project.idThe assigned project id
space.nameThe name of the space
space.idThe assigned space id
spec.filenameThe test spec file name, excluding the path
spec.filepathThe test spec file path relative to the repo's root (e.g. "specs")
spec.idThe assigned test spec id
repo.branchThe branch name (same as the space.name)
repo.urlThe branch URL (e.g. "https://github.com/org/repo")

The following global variables are instantiated and only valid in the context of an open test session:

VariableDescription
spec.run_idThe assign ed run id
user.nameThe name of the user executing the spec (if assigned)
user.idThe user's assigned id ( if assigned)
session.nameThe session's name entered via the user
session.idThe session assigned id

The following is a simple example of referencing a global variable:

## My Case  
Enter the session name: {{ session.name }} in the comments below

* Check this
* Check that

Includes

The include tag allows using content from another file stored in the /specs/_includes folder. Include files are required to be located based on the configured specs folder under _includes.

The following syntax is required when referencing an include file:

{% include file.md %}

Both custom and global variables are referenceable within an include file using the {{ }} syntax.

important

Include files can not include self or any other "include" file.

# My Spec

* Step One

{% include morestuff.md %}
{% include sub-folder/otherstuff.md %}

Parameters

Parameters can also be passed to the included file using var="string":

{% include morestuff.md domain="testspace.com" %}

When referencing a passed "parameter" via the include tag, the following syntax is required: {{ include.VAR }}.

Logic

Logic and control flow for test specs is accomplished using Liquid tags. The curly-brace percentage delimiters {% %} and the text that they surround, does not produce any visible output when the test spec is rendered. The tags enable assigning variables and creating conditions and loops without showing any of the Liquid logic on the page.

if-else

{% if spec.OS == 'Windows' %}
- check for Windows stuff.
{% elsif spec.OS == 'Linux' %}
- check for Linux stuff
{% else %}
- check for Mac stuff
{% endif %}

for-loop

{% for i in (1..5) %}
{{ i | plus: 100}}
{% endfor %}
tip

For more information on logic and control flow using Liquid refer here.


Examples

The following examples can be used with Jekyll for local desktop reviewing. Refer to Desktop Preview for details.

Example using an include

/specs/_includes/inc.md
## Case 2
My description here
- Check this subdomain: {{ spec.subdomain }}
- Click here: https://{{ spec.subdomain }}.{{ include.domain }}

Example of a test spec including and passing parameters:

/specs/myspec.md
---
testspace: true
title: Vars and Include
subdomain: s2
---

{% if page %} {% assign spec = page %} {% endif %}

# {{ spec.title }}
Some description

## Case 1
- Do this
- Do that

{% include inc.md domain="testspace.com" %}

Example of myspec.md preview:

Template example

Example using data driven testing

Using front matter data to drive testing:

/specs/os-systems-test.md
---
testspace:
title: OS Systems
matrix: # test different OS systems
- name: Windows
timeout: 27 seconds
reqs: "[Windows details](https://staging7.newco.com/windows)"
- name: Linux
timeout: 14 seconds
reqs: "[Linux details](https://staging7.newco.com/linus)"
---

{% if page %} {% assign spec = page %} {% endif %}

# {{ spec.title }}
Matrix being tested.

Name | Timeout | Info
-----| --------| -----
{%- for os in spec.matrix %}
{{ os.name }} | {{ os.timeout}} | {{os.reqs -}}
{% endfor %}

{% for os in spec.matrix %}
## Test {{ os.name }}
* check for correct timeout: {{ os.timeout}}
* check on requirements: {{ os.reqs }}
{% endfor %}

Example of os-systems-test.md preview:

Template example

Automated Fixtures

A spec fixture is serverless automation (aka function), using GitHub Workflows or AWS Lambdas, that executes before and optionally after a spec. A fixture is used to ensure that the spec's execution environment is in a well-known state (e.g. acquire necessary resources), before the manual instructions are executed, and when applicable clean up the environment (e.g. release resources) after the spec is complete.

tip

Testspace fixtures enable testing that leverages a hybrid of automation and manual verification.

To define a fixture for a spec, the front matter is used:

---
testspace:
before:
name: github::setup
payload:
on: "setup is on"
---

There are two types of fixtures - before and after. Both before and after automation are described using the following properties:

ParameterValueDescription
namestringUnique identifier:
- GitHub: github:[org/repo][@branch]:function-name. Defaults to current "org", "repo", and "branch".
- AWS: aws:region:function-name.
description (optional)stringUser friendly description to present at execution time
payload (optional)YAML,
JSON file
YAML - in-placed embedded YAML. The content is serialized as JSON.

JSON File - reference to an existing JSON file using a @ prefix - @path/to/file.json. The path is relative to the spec file.

The Payload content is serialized as a single JSON string.

tip

Refer to https://www.json2yaml.com/ for some helpful tips on using JSON and YAML.

Before

A fixture before automation requires to be successfuly performed before the spec's test cases can be executed.

YAML Example

---
testspace:
before:
name: {name}
payload:
p1: "1st parameter"
p2: 42
p3:
- one
- two
- three
---
Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload, ..) {

var input = JSON.parse(payload);
console.log("p1:", input.p1);
console.log("p2:", input.p2);
console.log("p3:", input.p3);
}

JSON File Example

Example using a JSON File, requiring the @ prefix:

---
testspace:
before:
name: github::hello
description: Hello - using a single JSON file as input values
payload: "@file.json"
---

Where file.json contains:

{
"stuff": ["one", "two", "three"],
"more": {
"this": "one",
"that": "other one"
}
}
Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload, ..) {

var input = JSON.parse(payload);
console.log("payload:", input);
}

Advanced File Reference Example

Within either YAML or a JSON file, any property's value could be an arbitrary file reference - @path/to/file (any content, not just JSON). The path is relative to the referrer's location. The file content will be encoded as base64.

---
testspace:
before:
name: {name}
payload:
p1: "1st parameter"
p2: 42
p3:
- one
- two
- three
p4: "@file.txt"
---
Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload, ..) {

var input = JSON.parse(payload);
console.log("p1:", input.p1);
console.log("p2:", input.p2);
console.log("p3:", input.p3);
var p4 = Buffer.from(input.p4, 'base64');
// process the 'blob' content as needed
console.log("p4:", p4.toString());
}
important

For property values that are defined with a file reference (i.e. @path/to/file) they are encoded as base64 and required to be unencoded.

After

The fixture after automation is performed after the spec's test cases are executed, but only if the before automation was successful.

---
testspace:
before:
..
after:
name: github::goodbye
description: Goodbye - clean up stuff.
---

Context

A Testspace context is available for Fixtures during runtime. The context contains the global variables and the following fixture information.

VariableDescription
project.urlThe project URL (e.g. "https://newco.testspace.com/projects/42")
space.urlThe space URL
session.urlThe session URL (e.g. "https://newco.testspace.com/spaces/789/test_sessions/123")
spec.nameThe test spec name assigned in the file (i.e. # Name)
fixture.nameThe name defined in the before/after section of the spec
fixture.typebefore or after
fixture.descriptionThe friendly description (optional)

The context is serialized into a single JSON-string.

The following is an example of a context that has been deserialized:

{
user: { id: 173, name: 'Joe Smith' },
project: {
id: 6977,
name: 's2technologies:testspace.test.repo',
url: 'https://s2.testspace.com/projects/6977'
},
space: {
id: 22701,
name: 'github.fixture.v2a',
url: 'https://s2.testspace.com/spaces/22701'
},
session: {
id: 64952,
name: "Run 'gh.fixture.normal' spec",
url: 'https://s2.testspace.com/spaces/22701/test_sessions/64952'
},
spec: {
id: 10387858,
name: 'gh.fixture.normal',
filename: 'gh.fixture.normal.md',
filepath: 'specs',
run_id: '846be03d-d2d4-4f12-becb-472483c09950'
},
fixture: {
type: 'before',
name: 'github::normal',
description: 'before - simple input',
timeout: 300
},
repo: {
url: 'https://github.com/s2technologies/testspace.test.repo',
branch: 'github.fixture.v2a'
}
}

GitHub Workflows

GitHub has built-in CI/CD support. Testspace enables leveraging of this functionality in the context of executing a manual spec. To invoke a workflow as a automated fixture using GitHub a testspace.yml file is required in the repo.

Testspace uses a Workflow Dispatch event to trigger the workflow, using the required inputs definition.

name: Testspace
on:
workflow_dispatch:
inputs:
name:
description: 'Function name'
required: true
payload:
description: 'Function input-payload'
required: true
context:
description: 'Function execution-context'
required: true
...
important

A test spec uses the name field to invoke a specific job defined in the workflow.

---
testspace:
before:
name: github::fn-name
---
...

A name is defined in the test spec, which is used to execute a corresponding job.

jobs:
fn-name:
if: github.events.inputs.name == 'fn-name'
..
tip

Use job names to represent unique function calls required by different test specs.

Template

The following is an example workflow yml file.

name: Testspace
on:
workflow_dispatch:
inputs:
name:
description: 'Function name'
required: true
payload:
description: 'Function input-payload'
required: true
context:
description: 'Function execution-context'
required: true
env:
IN_NAME: ${{ github.event.inputs.name }}
IN_PAYLOAD: ${{ github.event.inputs.payload }}
IN_CONTEXT: ${{ github.event.inputs.context }}
jobs:
name1:
if: github.event.inputs.name == 'fn-name1'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Echo test spec function name
run: echo "running ${IN_NAME} function"
important

When calling a handler script always use environment variables when referencing IN_NAME, IN_PAYLOAD, and IN_CONTEXT.

For example, using a JavaScript executed from a workflow:

function handler(payload) {
...
}
handler(JSON.parse(process.env.IN_PAYLOAD));

The following example test spec works with the workflow above (i.e. name: fn-name1):

---
testspace:
before:
name: github::fn-name1
---

# TEST1
## Case 1
- do this
- do that
note

When a workflow fails do not re-run the failed job using the GitHub Actions UI. Testspace has no way of tracking the execution.

AWS Lambdas

AWS has built-in serverless function execution support using AWS Lambdas. This functionality fits very well with Testspace automated fixturing. To enable Testspace invoking Lambdas refer here for setup information.

Handler

Lambda handlers can be written in any language supported by AWS. The following is a Lambda function for Node.js:

exports.handler = async (event, context) => {

console.log("Input:", event);
// TO DO

const response = {
statusCode: 200,
body: JSON.stringify('Hello from Lambda!'),
};
return response;
};

The ASW Lambda runtime automatically converts the JSON Object to the corresponding language object type.

Logging

Testspace captures the input parameters sent to the Lambda and the log stream generated by the function as an annotation for the corresponding Result Suite.


Markdown Cheat Sheet

The following markdown syntax is supported.

Lists

List are used within steps and fixtures, but can also be nested.

Example:

  • First Step
    • sub-step 1
    • sub-step 2
  • Second Step
# My Spec

## My Case

* First Step
* sub-step 1
* sub-step 2
* Second Step

Liquid Cheat Sheet

WIP

The following are common liquid syntax examples.

Variables