Specification

Manual tests, known as test specs, are written using Markdown syntax. The markdown files (*.md) are maintained in the code section of the repo, by default in a folder named specs. To enable manual testing a, .testspace.yml configuration file is required to be placed at the root of your repo with the following minimum content:

.testspace.yml

manual:

info

Refer to .testspace.yml for details on how Testspace is configured to discover and manage testing sessions.

The following is a simple example directory structure containing 2 test specs:

root
├─ README.md
├─ .testspace.yml
└─ specs
     └─ myspec1.md
     └─ myspec2.md
     ..
..

The test specification model has largely been influenced by the Gauge automation framework, GitHub Flavored Markdown, and the Jekyll static websites framework.

tip

See Desktop Preview for developing and reviewing Test Specs on your desktop leveraging Jekyll.

Test Syntax

A manual test spec is a single markdown file consisting of:

A Front Matter header block
A test spec name
At least one test case
Optional test steps

/specs/test1.md
  ---
  testspace:
  ---
  # Test 1
  ## Case 1
   - do this
   - do that
  ## Case 2
    ..

tip

See the Markdown Cheat Sheet for helpful hints on leveraging markdown to capture test instructions.

Front Matter

To uniquely identify a markdown file as a Testspace spec file, a metadata block is required. This metadata is called front matter, denoted by the triple dashes (---) at the start and end of the block. It must be the first thing in the file, YAML formatted, containing the testspace: identifier.

---
testspace:
---

Disable Spec

To disable a test spec set the testspace token to false:

---
testspace: false
---

This setting prevents the spec from being selected on new test sessions. The existing status for the test spec will be maintained.

Remove Spec

To remove a test spec, without deleting it from the repo, comment out (#) the testspace token:

---
#testspace:
---

The existing status will be removed on the subsequent result.

Spec

Each spec also is required to contain a spec name written in the <H1> markdown syntax, following the front matter block. Any optional text after the spec name is the description of the spec.

---
testspace:
---

# My Spec

Spec description text

Case

A spec requires at least one test case. A case name is denoted by a markdown <H2> heading. Any optional text following the heading is the test case description.

# My Spec

## My Case
Case description text

## My Other Case

Step

Steps are optional instructions to be executed for a test case. Steps are written using a markdown (ordered or unordered) list inside a test case.

## My Case

* My step 1 instructions
* My step 2 instructions

Steps can also exist outside a test case, in which case they are meant to represent a repeated context that is applicable to each test case in the spec.

Fixtures

Test case Fixtures implement steps/conditions that are required for executing each case within the spec. They are defined using the same markdown syntax but named in a special way.

info

A fixture can be defined anywhere in the file, but only one time.

Setup is a fixture, defined as a set of instructions, to be presented before each case during execution.

A Setup fixture is defined using the ## [setup] syntax:

## [setup]
This is my setup fixture:

- First do this
- Next do that

The Setup fixture can be defined anywhere in the file, but only one time.

Teardown is a fixture, defined as a set of instructions, to be presented after each case during execution.

A Teardown fixture is defined using the ## [teardown]:

## [teardown]
This is my teardown fixture:

- Now do this
- And do that

Template Language

Testspace supports the template language called Liquid. Test specs are handled as template files, meaning they get preprocessed before being rendered. This functionality enables a test spec file to use variables, include files (i.e. subroutines) along with passing parameters, and even the ability to implement conditional logic.

tip

Testspace supports the template language called Liquid. Refer to this blog article for a Liquid overview.

Variables

There are two types of variables supported by Testspace: custom and global. Testspace traverses your spec files and processes variables.

Custom

User-defined "custom" variables are defined in the front matter block and are referenceable throughout the spec.

---
testspace:
var1: Hey there
---

When referencing a variable, the following syntax is required: {{ spec.VAR }}.

## My Case

* My first step uses this {{ spec.var1 }}
* My Other Step

Global

The following global variables are supported: {{ variable }}.

Variable	Description
`project.name`	The name of the Testspace project
`project.id`	The assigned project id
`space.name`	The name of the space
`space.id`	The assigned space id
`spec.filename`	The test spec file name, excluding the path
`spec.filepath`	The test spec file path relative to the repo's root (e.g. "specs")
`spec.id`	The assigned test spec id
`repo.branch`	The branch name (same as the `space.name`)
`repo.url`	The branch URL (e.g. "https://github.com/org/repo")

The following global variables are instantiated and only valid in the context of an open test session:

Variable	Description
`spec.run_id`	The assign ed run id
`user.name`	The name of the user executing the spec (if assigned)
`user.id`	The user's assigned id ( if assigned)
`session.name`	The session's name entered via the user
`session.id`	The session assigned id

The following is a simple example of referencing a global variable:

## My Case
Enter the session name: {{ session.name }} in the comments below

* Check this
* Check that

Includes

The include tag allows using content from another file stored in the /specs/_includes folder. Include files are required to be located based on the configured specs folder under _includes.

The following syntax is required when referencing an include file:

{% include file.md %}

Both custom and global variables are referenceable within an include file using the {{ }} syntax.

info

Include files can not include self or any other "include" file.

# My Spec

* Step One

{% include morestuff.md %}
{% include sub-folder/otherstuff.md %}

Parameters

Parameters can also be passed to the included file using var="string":

{% include morestuff.md domain="testspace.com" %}

When referencing a passed "parameter" via the include tag, the following syntax is required: {{ include.VAR }}.

Logic

Logic and control flow for test specs is accomplished using Liquid tags. The curly-brace percentage delimiters {% %} and the text that they surround, does not produce any visible output when the test spec is rendered. The tags enable assigning variables and creating conditions and loops without showing any of the Liquid logic on the page.

if-else

{% if spec.OS == 'Windows' %}
  - check for Windows stuff.
{% elsif spec.OS == 'Linux' %}
  - check for Linux stuff
{% else %}
  - check for Mac stuff
{% endif %}

for-loop

{% for i in (1..5) %}
 {{ i | plus: 100}}
{% endfor %}

tip

For more information on logic and control flow using Liquid refer here.

Examples

The following examples can be used with Jekyll for local desktop reviewing. Refer to Desktop Preview for details.

Example using an include

/specs/_includes/inc.md
## Case 2
My description here
- Check this subdomain: {{ spec.subdomain }}
- Click here: https://{{ spec.subdomain }}.{{ include.domain }}

Example of a test spec including and passing parameters:

/specs/myspec.md
---
testspace: true
title: Vars and Include
subdomain: s2
---

{% if page %} {% assign spec = page %} {% endif %}

# {{ spec.title }}
Some description

## Case 1
- Do this
- Do that

{% include inc.md domain="testspace.com" %}

Example of myspec.md preview:

Template example

Example using data driven testing

Using front matter data to drive testing:

/specs/os-systems-test.md
---
testspace:
title: OS Systems
matrix: # test different OS systems
  - name: Windows
    timeout: 27 seconds
    reqs: "[Windows details](https://staging7.newco.com/windows)"
  - name: Linux
    timeout: 14 seconds
    reqs: "[Linux details](https://staging7.newco.com/linus)"
---

{% if page %} {% assign spec = page %} {% endif %}

# {{ spec.title }}
Matrix being tested.

Name | Timeout | Info
-----| --------| -----
{%- for os in spec.matrix %}
  {{ os.name }} | {{ os.timeout}} | {{os.reqs -}}
{% endfor %}

{% for os in spec.matrix %}
## Test  {{ os.name }}
* check for correct timeout: {{ os.timeout}}
* check on requirements: {{ os.reqs }}
{% endfor %}

Example of os-systems-test.md preview:

Template example

Automated Fixtures

A spec fixture is serverless automation (aka function), using GitHub Workflows or AWS Lambdas, that executes before and optionally after a spec. A fixture is used to ensure that the spec's execution environment is in a well-known state (e.g. acquire necessary resources), before the manual instructions are executed, and when applicable clean up the environment (e.g. release resources) after the spec is complete.

tip

Testspace fixtures enable testing that leverages a hybrid of automation and manual verification.

To define a fixture for a spec, the front matter is used:

---
testspace:
before:
  name: github::setup
  payload:
    on: "setup is on"
---

There are two types of fixtures - before and after. Both before and after automation are described using the following properties:

Parameter	Value	Description
name	string	Unique identifier: - GitHub: `github:[org/repo][@branch]:function-name`. Defaults to current "org", "repo", and "branch". - AWS: `aws:region:function-name`.
description (optional)	string	User friendly description to present at execution time
payload (optional)	YAML, JSON file	YAML - in-placed embedded YAML. The content is serialized as JSON. JSON File - reference to an existing JSON file using a `@` prefix - `@path/to/file.json`. The path is relative to the spec file.

The Payload content is serialized as a single JSON string.

tip

Refer to https://www.json2yaml.com/ for some helpful tips on using JSON and YAML.

Before

A fixture before automation requires to be successfuly performed before the spec's test cases can be executed.

YAML Example

---
testspace:
before:
  name: {name}
  payload:
    p1: "1st parameter"
    p2: 42
    p3:
      - one
      - two
      - three
---

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload, ..) {

    var input = JSON.parse(payload);
    console.log("p1:", input.p1);
    console.log("p2:", input.p2);
    console.log("p3:", input.p3);
}

JSON File Example

Example using a JSON File, requiring the @ prefix:

---
testspace:
before:
  name: github::hello
  description: Hello - using a single JSON file as input values
  payload: "@file.json"
---

Where file.json contains:

{
    "stuff": ["one", "two", "three"],
    "more": {
        "this": "one",
        "that": "other one"
    }
}

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload, ..) {

    var input = JSON.parse(payload);
    console.log("payload:", input);
}

Advanced File Reference Example

Within either YAML or a JSON file, any property's value could be an arbitrary file reference - @path/to/file (any content, not just JSON). The path is relative to the referrer's location. The file content will be encoded as base64.

---
testspace:
before:
  name: {name}
  payload:
    p1: "1st parameter"
    p2: 42
    p3:
      - one
      - two
      - three
    p4: "@file.txt"
---

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload, ..) {

    var input = JSON.parse(payload);
    console.log("p1:", input.p1);
    console.log("p2:", input.p2);
    console.log("p3:", input.p3);
    var p4 = Buffer.from(input.p4, 'base64');
    // process the 'blob' content as needed
    console.log("p4:", p4.toString());
}

info

For property values that are defined with a file reference (i.e. @path/to/file) they are encoded as base64 and required to be unencoded.

After

The fixture after automation is performed after the spec's test cases are executed, but only if the before automation was successful.

---
testspace:
before:
  ..
after:
  name: github::goodbye
  description: Goodbye - clean up stuff.
---

Context

A Testspace context is available for Fixtures during runtime. The context contains the global variables and the following fixture information.

Variable	Description
`project.url`	The project URL (e.g. "https://newco.testspace.com/projects/42")
`space.url`	The space URL
`session.url`	The session URL (e.g. "https://newco.testspace.com/spaces/789/test_sessions/123")
`spec.name`	The test spec name assigned in the file (i.e. `# Name`)
`fixture.name`	The `name` defined in the before/after section of the spec
`fixture.type`	`before` or `after`
`fixture.description`	The friendly description (optional)

The context is serialized into a single JSON-string.

The following is an example of a context that has been deserialized:

{
  user: { id: 173, name: 'Joe Smith' },
  project: {
    id: 6977,
    name: 's2technologies:testspace.test.repo',
    url: 'https://s2.testspace.com/projects/6977'
  },
  space: {
    id: 22701,
    name: 'github.fixture.v2a',
    url: 'https://s2.testspace.com/spaces/22701'
  },
  session: {
    id: 64952,
    name: "Run 'gh.fixture.normal' spec",
    url: 'https://s2.testspace.com/spaces/22701/test_sessions/64952'
  },
  spec: {
    id: 10387858,
    name: 'gh.fixture.normal',
    filename: 'gh.fixture.normal.md',
    filepath: 'specs',
    run_id: '846be03d-d2d4-4f12-becb-472483c09950'
  },
  fixture: {
    type: 'before',
    name: 'github::normal',
    description: 'before - simple input',
    timeout: 300
  },
  repo: {
    url: 'https://github.com/s2technologies/testspace.test.repo',
    branch: 'github.fixture.v2a'
  }
}

GitHub Workflows

GitHub has built-in CI/CD support. Testspace enables leveraging of this functionality in the context of executing a manual spec. To invoke a workflow as a automated fixture using GitHub a testspace.yml file is required in the repo.

Testspace uses a Workflow Dispatch event to trigger the workflow, using the required inputs definition.

name: Testspace
run-name: ${{ fromJSON(inputs.context).spec.name }}
on:
  workflow_dispatch:
    inputs:
      name:
        description: 'Function name'
        required: true
      payload:
        description: 'Function input-payload'
        required: true
      context:
        description: 'Function execution-context'
        required: true
...

info

A test spec uses the name field to invoke a specific job defined in the workflow.

---
testspace:
before:
  name: github::fn-name
---
...

A name is defined in the test spec, which is used to execute a corresponding job.

jobs:
  fn-name:
    if: inputs.name == 'fn-name'
  ..

tip

Use job names to represent unique function calls required by different test specs.

Template

The following is an example workflow yml file.

name: Testspace
run-name: ${{ fromJSON(inputs.context).spec.name }}
on:
  workflow_dispatch:
    inputs:
      name:
        description: 'Function name'
        required: true
      payload:
        description: 'Function input-payload'
        required: true
      context:
        description: 'Function execution-context'
        required: true
env:
  IN_NAME:    ${{ inputs.name }}
  IN_PAYLOAD: ${{ inputs.payload }}
  IN_CONTEXT: ${{ inputs.context }}
jobs:
  name1:
    if: inputs.name == 'fn-name1'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Echo test spec function name
        run: echo "running ${IN_NAME} function"

info

When calling a handler script always use environment variables when referencing IN_NAME, IN_PAYLOAD, and IN_CONTEXT.

For example, using a JavaScript executed from a workflow:

function handler(payload) {
  ...
}
handler(JSON.parse(process.env.IN_PAYLOAD));

The following example test spec works with the workflow above (i.e. name: fn-name1):

---
testspace:
before:
  name: github::fn-name1
---

# TEST1
## Case 1
- do this
- do that

note

When a workflow fails do not re-run the failed job using the GitHub Actions UI. Testspace has no way of tracking the execution.

AWS Lambdas

AWS has built-in serverless function execution support using AWS Lambdas. This functionality fits very well with Testspace automated fixturing. To enable Testspace invoking Lambdas refer here for setup information.

Handler

Lambda handlers can be written in any language supported by AWS. The following is a Lambda function for Node.js:

exports.handler = async (event, context) => {

    console.log("Input:", event);
    // TO DO

    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

The ASW Lambda runtime automatically converts the JSON Object to the corresponding language object type.

Logging

Testspace captures the input parameters sent to the Lambda and the log stream generated by the function as an annotation for the corresponding Result Suite.

Markdown Cheat Sheet

The following markdown syntax is supported.

tip

Refer here - https://guides.github.com/features/mastering-markdown/

Lists

List are used within steps and fixtures, but can also be nested.

Example:

First Step
- sub-step 1
- sub-step 2
Second Step

# My Spec

## My Case

* First Step
  * sub-step 1
  * sub-step 2
* Second Step

Emphasis

The following emphasis applied to text:

bold
Italic
~~strikethrough~~
Bold and italic

**bold**
*Italic*
~~strikethrough~~
**Bold** and _italic_

Tables

Tables for rendering a matrix.

First Header	Second Header
Content from cell 1	Content from cell 2
Content in the first column	Content in the second column

 First Header | Second Header
------------ | -------------
Content from cell 1 | Content from cell 2
Content in the first column | Content in the second column

And an alternative format

| First Header | Second Header|
|------------ | -------------|
| Content from cell 1 | Content from cell 2|
| Content in the first column | Content in the second column|

Images

Images are supported and are displayed during test execution.

My Image

Image using an absolute path in repo

![My Image](/images/myimage.png "Image here")

or a relative path

![My Image](../images/myimage.png "Image here")

Links to images is also supported:

My Image

![My Image](https://upload.wikimedia.org/wikipedia/commons/3/3b/Check_icon.svg "Is this a check")

Code block

Using code blocks requires surronding the code with ```:

$ testspace results.xml

Inline code also supported:

Example - I think you should use an <addr> element here instead.

I think you should use an
`<addr>` element here instead.

Links

Link support:

https://testspace.com Testspace

https://testspace.com - automatic!
[Testspace](https://testspace.com)

Quoting text

Important statements:

Beware of this and that

> Beware of this and that

Emoji

Emoji support:

😊 👀 👍

:relaxed:
:eyes:
:+1:

To add comment in your spec file that will not be redendered use the following syntax:

[your comment]: # (content being removed)

If no your comment use [//].

[//]: # (content being removed)

https://stackoverflow.com/questions/4823468/comments-in-markdown

Liquid Cheat Sheet

WIP

The following are common liquid syntax examples.

tip

Refer here - https://shopify.github.io/liquid/

Variables
Comments
Raw
White space

Variables

White space

In Liquid, you can include a hyphen in your tag syntax {{-, -}}, {%-, and -%} to strip whitespace from the left or right side of a rendered tag.

Allows to leave *un-rendered* code and/or *un-executed* code.

{% assign xyz = "hey" %}
{% comments %}
{% assign xyz = "NO" %}
{% endcomments %}

Refer to https://shopify.github.io/liquid/tags/template/

Temporarily disables tag processing. Applicable when conflict with markdown syntax. For example,

{% raw %}
steps:
  - uses: testspace-com/setup-testspace@v1
    with:
      domain: ${{ github.repository_owner }}
      token:  ${{ secrets.TESTSPACE_TOKEN }} # optional, only required for private repos
  ...
{% endraw %}

Refer to https://shopify.github.io/liquid/tags/template/

Test Syntax​

Front Matter​

Disable Spec​

Remove Spec​

Spec​

Case​

Step​

Fixtures​

Template Language​

Variables​

Custom​

Global​

Includes​

Parameters​

Logic​

Examples​

Example using an include​

Example using data driven testing​

Automated Fixtures​

Before​

YAML Example​

Handler Example​

JSON File Example​

Handler Example​

Advanced File Reference Example​

Handler Example​

After​

Context​

GitHub Workflows​

Template​

AWS Lambdas​

Handler​

Logging​

Markdown Cheat Sheet​

Lists​

Emphasis​

Tables​

Images​

Code block​

Links​

Quoting text​

Emoji​

Liquid Cheat Sheet​

Test Syntax

Front Matter

Disable Spec

Remove Spec

Spec

Case

Step

Fixtures

Template Language

Variables

Custom

Global

Includes

Parameters

Logic

Examples

Example using an include

Example using data driven testing

Automated Fixtures

Before

YAML Example

Handler Example

JSON File Example

Handler Example

Advanced File Reference Example

Handler Example

After

Context

GitHub Workflows

Template

AWS Lambdas

Handler

Logging

Markdown Cheat Sheet

Lists

Emphasis

Tables

Images

Code block

Links

Quoting text

Emoji

Liquid Cheat Sheet