Automated Fixtures

A spec fixture is serverless automation (aka function), using GitHub Workflows or AWS Lambdas, that executes before and optionally after a spec. A fixture is used to ensure that the spec's execution environment is in a well-known state (e.g. acquire necessary resources), before the manual instructions are executed, and when applicable clean up the environment (e.g. release resources) after the spec is complete.

tip

Testspace fixtures enable testing that leverages a hybrid of automation and manual verification.

A fixture is defined in the spec's front matter. In a Standalone Project, type it in the editor's Variables box (the editor adds the testspace: line):

---
testspace:
before:
  name: github::setup
  payload:
    on: "setup is on"
---

There are two types of fixtures - before and after. Both before and after automation are described using the following properties:

Parameter	Value	Description
name	string	Unique identifier: - GitHub: `github:[org/repo][@branch]:function-name`. Defaults to current "org", "repo", and "branch" (Standalone: see Standalone Differences). The function name is a job in the repo's `.github/workflows/testspace.yml`. - AWS: `aws:region:function-name`.
description (optional)	string	User friendly description to present at execution time. Defaults to "Prepare test environment." (`before`) or "Clean up test environment." (`after`).
payload (optional)	YAML, JSON file	YAML - in-placed embedded YAML. The content is serialized as JSON. JSON File - reference to an existing JSON file using a `@` prefix - `@path/to/file.json`. The path is relative to the spec file (Connected only).
timeout (optional)	number	`before` only: seconds a fixture can run before the tester is asked whether to keep waiting. Default `300`; values are limited to 60–300.

The Payload content is serialized as a single JSON string.

tip

Refer to https://www.json2yaml.com/ for some helpful tips on using JSON and YAML.

Before

A fixture before automation must be successfully performed before the spec's test cases can be executed. It runs each time a tester clicks START, in a session or an ad hoc run.

YAML Example

---
testspace:
before:
  name: github::my-function
  payload:
    p1: "1st parameter"
    p2: 42
    p3:
      - one
      - two
      - three
---

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload) {

    var input = JSON.parse(payload);
    console.log("p1:", input.p1);
    console.log("p2:", input.p2);
    console.log("p3:", input.p3);
}

JSON File Example

Example using a JSON File, requiring the @ prefix:

---
testspace:
before:
  name: github::hello
  description: Hello - using a single JSON file as input values
  payload: "@file.json"
---

Where file.json contains:

{
    "stuff": ["one", "two", "three"],
    "more": {
        "this": "one",
        "that": "other one"
    }
}

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload) {

    var input = JSON.parse(payload);
    console.log("payload:", input);
}

Advanced File Reference Example

Within either YAML or a JSON file, any property's value could be an arbitrary file reference - @path/to/file (any content, not just JSON). The path is relative to the referrer's location. The file content will be encoded as base64.

---
testspace:
before:
  name: github::my-function
  payload:
    p1: "1st parameter"
    p2: 42
    p3:
      - one
      - two
      - three
    p4: "@file.txt"
---

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload) {

    var input = JSON.parse(payload);
    console.log("p1:", input.p1);
    console.log("p2:", input.p2);
    console.log("p3:", input.p3);
    var p4 = Buffer.from(input.p4, 'base64');
    // process the 'blob' content as needed
    console.log("p4:", p4.toString());
}

info

For property values that are defined with a file reference (i.e. @path/to/file) the content is encoded as base64 — the receiving handler must decode it (as shown above) to get the original bytes.

After

The fixture after automation is performed after the spec's test cases are executed, but only if the before automation was successful.

---
testspace:
before:
  ..
after:
  name: github::goodbye
  description: Goodbye - clean up stuff.
---

Context

A Testspace context is available for Fixtures during runtime. The context contains the global variables and the following fixture information.

Variable	Description
`project.url`	The project URL (e.g. "https://newco.testspace.com/projects/42")
`space.url`	The space URL
`session.url`	The session URL (e.g. "https://newco.testspace.com/spaces/789/test_sessions/123")
`spec.name`	The spec name (i.e. `# Name`)
`spec.run_id`	The id of this fixture run
`fixture.name`	The `name` defined in the before/after section of the spec
`fixture.type`	`before` or `after`
`fixture.description`	The friendly description (optional)
`fixture.timeout`	The `timeout`, in seconds

The context is serialized into a single JSON-string.

The following is an example of a context that has been deserialized (shown as a JavaScript object, not strict JSON — object keys aren't quoted):

{
  user: { id: 173, name: 'Joe Smith' },
  project: {
    id: 6977,
    name: 's2technologies:testspace.test.repo',
    url: 'https://s2.testspace.com/projects/6977'
  },
  space: {
    id: 22701,
    name: 'github.fixture.v2a',
    url: 'https://s2.testspace.com/spaces/22701'
  },
  session: {
    id: 64952,
    name: "Run 'gh.fixture.normal' spec",
    url: 'https://s2.testspace.com/spaces/22701/test_sessions/64952'
  },
  spec: {
    id: 10387858,
    name: 'gh.fixture.normal',
    filename: 'gh.fixture.normal.md',
    filepath: 'specs',
    run_id: '846be03d-d2d4-4f12-becb-472483c09950'
  },
  fixture: {
    type: 'before',
    name: 'github::normal',
    description: 'before - simple input',
    timeout: 300
  },
  repo: {
    url: 'https://github.com/s2technologies/testspace.test.repo',
    branch: 'github.fixture.v2a'
  }
}

GitHub Workflows

GitHub has built-in CI/CD support. Testspace enables leveraging of this functionality in the context of executing a manual spec. To invoke a workflow as an automated fixture using GitHub, the repo needs a GitHub Actions workflow file named .github/workflows/testspace.yml — not to be confused with Testspace's own .testspace.yml. Testspace runs that file only; if it's missing, the fixture fails ("Workflow file specified in fixture's configuration cannot be found.").

Testspace uses a Workflow Dispatch event to trigger the workflow, using the required inputs definition.

name: Testspace
run-name: ${{ fromJSON(inputs.context).spec.name }}
on:
  workflow_dispatch:
    inputs:
      name:
        description: 'Function name'
        required: true
      payload:
        description: 'Function input-payload'
        required: true
      context:
        description: 'Function execution-context'
        required: true
...

info

A spec uses the name field to invoke a specific job defined in the workflow.

---
testspace:
before:
  name: github::fn-name
---
...

A name is defined in the spec, which is used to execute a corresponding job.

jobs:
  fn-name:
    if: inputs.name == 'fn-name'
  ..

tip

Use job names to represent unique function calls required by different specs.

Template

The following is an example .github/workflows/testspace.yml.

name: Testspace
run-name: ${{ fromJSON(inputs.context).spec.name }}
on:
  workflow_dispatch:
    inputs:
      name:
        description: 'Function name'
        required: true
      payload:
        description: 'Function input-payload'
        required: true
      context:
        description: 'Function execution-context'
        required: true
env:
  IN_NAME:    ${{ inputs.name }}
  IN_PAYLOAD: ${{ inputs.payload }}
  IN_CONTEXT: ${{ inputs.context }}
jobs:
  name1:
    if: inputs.name == 'fn-name1'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Echo spec function name
        run: echo "running ${IN_NAME} function"

info

When calling a handler script always use environment variables when referencing IN_NAME, IN_PAYLOAD, and IN_CONTEXT.

For example, using a JavaScript executed from a workflow:

function handler(payload) {
  ...
}
handler(JSON.parse(process.env.IN_PAYLOAD));

The following example spec works with the workflow above (i.e. name: fn-name1):

---
testspace:
before:
  name: github::fn-name1
---

# TEST1
## Case 1
- do this
- do that

note

When a workflow fails do not re-run the failed job using the GitHub Actions UI. Testspace has no way of tracking the execution.

AWS Lambdas

AWS has built-in serverless function execution support using AWS Lambdas. This functionality fits very well with Testspace automated fixturing. To enable Testspace invoking Lambdas refer here for setup information.

Handler

Lambda handlers can be written in any language supported by AWS. The following is a Lambda function for Node.js:

exports.handler = async (event, context) => {

    console.log("Input:", event);
    // TO DO

    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

The AWS Lambda runtime automatically converts the JSON Object to the corresponding language object type.

Logging

Testspace captures the input parameters sent to the Lambda and the log stream generated by the function as an annotation for the corresponding Result Suite.

Standalone Differences

Automated fixtures work on Standalone Projects too, with a few differences from Connected:

Prerequisite. A github:... fixture needs the Company connected to GitHub. For any fixture that omits the repository, or uses the short form (e.g. github:manual:fn-name1), the Project's Issues provider setting must also be set to a GitHub org/repo — that's where the default repository comes from.
Default-repo resolution differs from Connected. On Connected, a blank/short repository name defaults to the project's own connected repo. On Standalone — which has no repo of its own — it defaults to whatever repo is set as the project's Issues provider instead, and the branch defaults to that repo's default branch.
Not limited to the Issues-provider repo. A fixture can still target any org/repo the Company's GitHub App installation can reach, using the full form (e.g. github:acme/other-repo:fn-name1) — same as Connected. Issues provider only supplies the default, it isn't a restriction.
The repo.* fixture context is always blank. The metadata sent to the triggered GitHub Actions workflow or AWS Lambda (context.repo.url / context.repo.branch — see Context above) is always empty strings for Standalone, since there's no real repo/branch to report. This is the fixture-execution counterpart to the {{ repo.* }} Liquid variables in Writing a Spec, which behave the same way.

info

Known limitation: file references (payload: "@path/to/file.json", or @path/to/file as a property value) don't work on Standalone Projects, for both github:... and aws:... fixtures — they fetch the file from a connected repo, which a Standalone Project doesn't have. An inline YAML payload (as shown in Before above) works with no restrictions.

Execution

A spec with a before fixture, and optionally an after fixture, has extra execution behavior. A gray ⚙️ icon next to START indicates a required action; hover over it for the fixture's description:

Before Fixture

The before fixture is required to be successfully processed before test cases can be executed. It is triggered by clicking the blue START button, every time START is clicked. While it runs, the icon turns blue ("Preparing the environment..."); when it succeeds, it turns green ("Environment ready.") and the cases can be tested.

On failure, a red exclamation icon replaces it. Its tooltip gives the failure information. The spec's cases are set to errored (Required "before" fixture failed).

If the fixture configuration itself isn't valid (for example, a name that isn't github: or aws:, or an unknown AWS region), the red exclamation icon shows from the start and its tooltip names the problem. START then sets the cases to errored (Required "before" fixture is invalid) without running anything.

The before fixture has several constraints that govern the execution of the spec:

Closing the dialog while executing will cancel the execution
Clicking START again runs the fixture again
When the fixture runs longer than its timeout (5 minutes by default; see the properties above), the tester is asked whether to Wait or Cancel. Wait asks again after another timeout; Cancel stops the fixture and sets the cases to errored.

After Fixture

The after fixture is optional and can only exist if there is also a before fixture. If the spec has successfully executed, once the spec dialog is closed, the associated after fixture automation is executed, in the background, without any status available to the tester.

info

The after fixture will automatically execute when:

The user selects STOP or X within the spec run dialog
The browser tab or computer is shutdown
No user activity for 10 minutes (user prompted with 60-second countdown)

Before​

YAML Example​

Handler Example​

JSON File Example​

Handler Example​

Advanced File Reference Example​

Handler Example​

After​

Context​

GitHub Workflows​

Template​

AWS Lambdas​

Handler​

Logging​

Standalone Differences​

Execution​

Before Fixture​

After Fixture​

Before

YAML Example

Handler Example

JSON File Example

Handler Example

Advanced File Reference Example

Handler Example

After

Context

GitHub Workflows

Template

AWS Lambdas

Handler

Logging

Standalone Differences

Execution

Before Fixture

After Fixture