Skip to main content

Specification

Manual tests, known as specs, are written using Markdown syntax. The markdown files (*.md) are maintained in the code section of the repo, by default in the specs folder.

root
โ””โ”€ specs
โ””โ”€ mytest1.md
โ””โ”€ mytest2.md
..

A manual test file consists of different sections; some of which are mandatory:

  • Header
  • Case
  • Step
  • Context
  • Fixture
  • Variable
  • Include

The test specification model has largely been influenced by the Gauge automation framework, GitHub Flavored Markdown, and the Jekyll static websites framework.

tip

See Spec Preview for developing and reviewing Specs on your desktop leveraging Jekyll.

Header#

To uniquely identify a markdown file as a Testspace spec file, a metadata block is required. This metadata is called front matter, denoted by the triple dashes (---) at the start and end of the block. It must be the first thing in the file, YAML formatted, containing the testspace: identifier.

---
testspace:
---

Each spec also is required to contain a spec name written in the <H1> markdown syntax, following the front matter block. Any optional text after the spec name is the description of the spec.

---
testspace:
---
# My Spec
Spec description text

Case#

A spec requires at least one test case. A case name is denoted by a markdown <H2> heading. Any optional text following the heading is the test case description.

# My Spec
## My Case
Case description text
## My Other Case

Step#

Steps are optional instructions to be executed for a test case. Steps are written using a markdown (ordered or unordered) list inside a test case.

## My Case
* My step 1 instructions
* My step 2 instructions

Steps can also exist outside a test case, in which case they are meant to represent a repeated context that is applicable to each test case in the spec.

Context#

Spec context steps are defined in a spec before and/or after all of the cases. These steps allow you to specify a set of conditions that are necessary for executing the cases. If there are multiple cases, context steps are presented before and/or after every case in the specification.

Context steps are presented before and/or after every case

Setup#

A Setup context includes steps listed before the test cases:

# My Spec
* Setup Step One
* Setup Step Two
## First test case
...

Teardown#

A Teardown context includes steps listed after the last case, separated via a horizontal rule (i.e. using 3 or more dashes):

# My Spec
...
## Last test case
* Step One
---
* Teardown Step One
* Teardown Step Two

Fixture#

A spec fixture is a serverless automation (aka function), using GitHub Workflows or AWS Lambdas, that executes before and optionally after a spec. A fixture is used to ensure that the spec's execution environment is in a well-known state (e.g. acquire necessary resources) before the manual instructions are executed, and when applicable clean up the environment (e.g. release resources) after the spec is complete.

Refer to the Automation section for more details.

A fixture is used to automate the set up of a Spec, ensuring a well-known state before testing

To define a fixture for a spec, the front matter is used:

---
testspace:
before:
name: github::setup
input:
on: "setup is on"
---

There are two types of fixtures - before and after. Both before and after automation are described using the following properties:

ParameterValueDescription
namestringUnique identifier:
- GitHub: github:[org/][repo@branch]:workflow-name. Defaults to current "org", "repo", and "branch".
- AWS: aws:region:function-name.
descriptionstring(optional) User friendly description to present at execution time
inputstring,
number,
YAML,
file
(optional) list of up to 10 top-level name/value input parameters, specifying the value using a string/number, embedded YAML, or a file reference.

Parameters defined as embedded YAML will be serialized.

Parameters defined as a file reference require the @prefix - @path/to/file.json, the path is relative to the spec file. The file content will be encoded as base64.

Note that input, depending on the service provider, can also be called: inputs, payload, body, and data.

When referring to serverless automation hosted as a service, the term function is often used.

Before#

A fixture before automation requires to be successfuly performed before the spec's test cases can be executed.

---
testspace:
before:
name: github::hello
description: Hello - using strings as input values
input:
p1: do this
p2: do that
---

Example using an input file, requiring the @ prefix:

---
testspace:
before:
name: github::hello
description: Hello - using associate array and list, and a json file as input values
inputs:
a:
- one
- two
- three
b:
this: one
that: other one
c: "@file.json"
---

Where file.json contains:

{
"stuff": ["one", "two", "three"],
"more": {
"this": "one",
"that": "other one"
}
}

After#

The fixture after automation is performed after the spec's test cases are executed, but only if the before automation was successful.

---
testspace:
before:
..
after:
name: github::goodbye
description: Goodbye - using json array and list as input values
input:
myarr:
- one
- two
- three
mylist:
this: one
that: other one
---

Note that each value is a separate (individual) JSON object.

Variables#

There are two types of variables supported by Testspace: custom and global. Testspace traverses your spec files and processes variables.

Custom#

User-defined "custom" variables are defined in the front matter block and can be referenced throughout the spec.

---
testspace:
var1: Hey there
---

When referencing a variable, the following syntax is required: {{ spec.VAR }}.

## My Case
* My first step uses this {{ spec.var1 }}
* My Other Step

Global#

The following global variables are supported:

  • Project - project.id, project.name, and project.url
  • Space - space.id, space.name, and space.url
  • Spec - spec.id, spec.name and spec.filename (file-path relative to the repo's root)
  • Fixture - fixture.type, fixture.name, fixture.description and fixture.timeout
  • Repo - repo.url (e.g. "https://github.com/org/repo") and repo.branch (i.e. same as "space.name")

These global variables only provide context during a session:

  • User - user.id, user.name
  • Cycle - cycle.id, cycle.name, cycle.url and cycle.issue_id (i.e. GitHub's issue-id)
  • Session - session.id, session.name and , session.url
## My Case
* My first step uses this {{ spec.var1 }}
* My Other Step uses this {{ project.name }}

Includes#

Includes allow using content from another file stored in the /specs/_includes folder. The following syntax is required when referencing an include file:

{% include file.md %}

Include files are required to be located within the configured specs folder under _includes.

# My Spec
* Step One
{% include morestuff.md %}

Both global and custom variables can be referenced within an include file.

Parameters#

Parameters can be passed to includes using var="string":

{% include morestuff.md domain="testspace.com" %}

When referencing a parameter, the following syntax is required: {{ include.VAR }}.

Example#

The following example can be used with Jekyll for local desktop reviewing. Refer to Spec Preview for details.

Example inc.md:

## Case 2
My description here
- Check this subdomain: {{ spec.subdomain }}
- Click here: https://{{ spec.subdomain }}.{{ include.domain }}

Example spec.md:

---
testspace: true
title: Vars and Include
subdomain: s2
---
{% if page %} {% assign spec = page %} {% endif %}
# {{ spec.title }}
Some description
## Case 1
- Do this
- Do that
{% include inc.md domain="testspace.com" %}

Example Spec Preview:


Vars and Include#

Some description

Case 1#
  • Do this
  • Do that
Case 2#

My description here


Syntax#

The following markdown syntax is supported.

Emphasis#

The following emphasis applied to text:

  • bold
  • Italic
  • strikethrough
  • Bold and italic
**bold**
*Italic*
~~strikethrough~~
**Bold** and _italic_

Tables#

Tables for rendering a matrix.

First HeaderSecond Header
Content from cell 1Content from cell 2
Content in the first columnContent in the second column
First Header | Second Header
------------ | -------------
Content from cell 1 | Content from cell 2
Content in the first column | Content in the second column

And an alternative format

| First Header | Second Header|
|------------ | -------------|
| Content from cell 1 | Content from cell 2|
| Content in the first column | Content in the second column|

Images#

Images are supported and are displayed during test execution.

My Image

Image using an absolute path in repo

![My Image](/images/myimage.png "Image here")

or a relative path

![My Image](../images/myimage.png "Image here")

Links to images is also supported:

My Image

![My Image](https://upload.wikimedia.org/wikipedia/commons/3/3b/Check_icon.svg "Is this a check")

Links#

Link support:

https://testspace.com
Testspace

https://testspace.com - automatic!
[Testspace](https://testspace.com)

Code block#

Using code blocks requires surronding the code with ```:

$ testspace results.xml

Inline code also supported:

Example - I think you should use an <addr> element here instead.

I think you should use an
`<addr>` element here instead.

Quoting text#

Important statements:

Beware of this and that

> Beware of this and that

Lists#

List are used within steps and fixtures, but can also be nested.

Example:

  • First Step
    • sub-step 1
    • sub-step 2
  • Second Step
# My Spec
## My Case
* First Step
* sub-step 1
* sub-step 2
* Second Step

Emoji#

Emoji support:

๐Ÿ˜Š ๐Ÿ‘€ ๐Ÿ‘

:relaxed:
:eyes:
:+1:

Spec Preview#

The Testspace Spec model is compatible with the GitHub Jekyll model, where a separate parsing and static HTML generation step occurs. As a result of this approach, it is easy to leverage Jekyll for local Spec development and previewing.

note

Use Jekyll for local spec development and previewing.

We are recommending using the Jekyll documentation theme called Just The Docs for reviewing specs. Note that other Themes are also available, or you can develop your own.

Setup#

To leverage Jekyll on the desktop, a few one-time steps are required. It is recommended to set up Jekyll in the same folder as your test specs (i.e. cd specs).

The first step is to install the Jekyll gem:

gem install jekyll bundler

There are 5 files required to be added to your repo under the specs folder:

specs
โ””โ”€ _config.yml
โ””โ”€ .gitignore
โ””โ”€ Gemfile
โ””โ”€ Gemfile.lock
โ””โ”€ index.md
..

Jekyll requires a configuration file:

/specs/_config.yml
title: Spec Preview
defaults:
-
values:
layout: "default"
theme: just-the-docs
remote_theme: pmarsceill/just-the-docs

The .gitignore file should be added, ignoring the local Jekyll generated file:

# Jekyll local building artifacts
_site
.jekyll-cache

The following Gemfile file is required:

/specs/Gemfile
source "https://rubygems.org"
gem "jekyll", "~> 4.2.0"
gem "minima", "~> 2.5"
gem "just-the-docs"
group :jekyll_plugins do
gem "jekyll-feed", "~> 0.12"
end
platforms :mingw, :x64_mingw, :mswin, :jruby do
gem "tzinfo", "~> 1.2"
gem "tzinfo-data"
end
# Performance-booster for watching directories on Windows
gem "wdm", "~> 0.1.1", :platforms => [:mingw, :x64_mingw, :mswin]

Note that the Gemfile.lock will be generated on the install step.

And last create an index markdown file:

/specs/index.md
---
layout: home
---
This is were the Manual Test `Specs` live.

Install#

Once the 5 files have been added, or if a repo has been cloned/forked, run the bundler to install:

cd specs
bundle install

Run Jekyll to re-confirm setup: http://127.0.0.1:4000/

bundle exec jekyll serve

Specs#

For a Spec to be listed using Jekyll, the following additions are required.

Front Matter#

Add the following variable:

title: Spec Name

Assignment#

The spec object requires assignment to the page object. Add this 1-line statement after the front matter header block:

{% if page %} {% assign spec = page %} {% endif %}

Spec Name#

To synchronize the title name and spec name, it is recommended to use the following global variable for the spec name:

# {{ spec.title }}
Spec optional description.

Example#

The following is a template example:

specs/hello.md
---
testspace:
title: Hello
---
{% if page %} {% assign spec = page %} {% endif %}
# {{ spec.title }}
Spec description text .. blah blah
## My First Case
- Do this
- Do that