Automated Fixtures

A spec fixture is serverless automation (aka function), using GitHub Workflows or AWS Lambdas, that executes before and optionally after a spec. A fixture is used to ensure that the spec's execution environment is in a well-known state (e.g. acquire necessary resources), before the manual instructions are executed, and when applicable clean up the environment (e.g. release resources) after the spec is complete.

tip

Testspace fixtures enable testing that leverages a hybrid of automation and manual verification.

To define a fixture for a spec, the front matter is used:

---
testspace:
before:
  name: github::setup
  payload:
    on: "setup is on"
---

There are two types of fixtures - before and after. Both before and after automation are described using the following properties:

Parameter	Value	Description
name	string	Unique identifier: - GitHub: github:[org/repo][@branch]:function-name. Defaults to current "org", "repo", and "branch". - AWS: aws:region:function-name.
description (optional)	string	User friendly description to present at execution time
payload (optional)	YAML, JSON file	YAML - in-placed embedded YAML. The content is serialized as JSON. JSON File - reference to an existing JSON file using a @ prefix - @path/to/file.json. The path is relative to the spec file.

The Payload content is serialized as a single JSON string.

tip

Refer to https://www.json2yaml.com/ for some helpful tips on using JSON and YAML.

Before

A fixture before automation must be successfully performed before the spec's test cases can be executed.

YAML Example

---
testspace:
before:
  name: github::my-function
  payload:
    p1: "1st parameter"
    p2: 42
    p3:
      - one
      - two
      - three
---

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload) {

    var input = JSON.parse(payload);
    console.log("p1:", input.p1);
    console.log("p2:", input.p2);
    console.log("p3:", input.p3);
}

JSON File Example

Example using a JSON File, requiring the @ prefix:

---
testspace:
before:
  name: github::hello
  description: Hello - using a single JSON file as input values
  payload: "@file.json"
---

Where file.json contains:

{
    "stuff": ["one", "two", "three"],
    "more": {
        "this": "one",
        "that": "other one"
    }
}

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload) {

    var input = JSON.parse(payload);
    console.log("payload:", input);
}

Advanced File Reference Example

Within either YAML or a JSON file, any property's value could be an arbitrary file reference - @path/to/file (any content, not just JSON). The path is relative to the referrer's location. The file content will be encoded as base64.

---
testspace:
before:
  name: github::my-function
  payload:
    p1: "1st parameter"
    p2: 42
    p3:
      - one
      - two
      - three
    p4: "@file.txt"
---

Handler Example

The following JavaScript handler example of deserializing the payload JSON String based on the above definition.

function handler(payload) {

    var input = JSON.parse(payload);
    console.log("p1:", input.p1);
    console.log("p2:", input.p2);
    console.log("p3:", input.p3);
    var p4 = Buffer.from(input.p4, 'base64');
    // process the 'blob' content as needed
    console.log("p4:", p4.toString());
}

info

For property values that are defined with a file reference (i.e. @path/to/file) the content is encoded as base64 — the receiving handler must decode it (as shown above) to get the original bytes.

After

The fixture after automation is performed after the spec's test cases are executed, but only if the before automation was successful.

---
testspace:
before:
  ..
after:
  name: github::goodbye
  description: Goodbye - clean up stuff.
---

Context

A Testspace context is available for Fixtures during runtime. The context contains the global variables and the following fixture information.

Variable	Description
project.url	The project URL (e.g. "https://newco.testspace.com/projects/42")
space.url	The space URL
session.url	The session URL (e.g. "https://newco.testspace.com/spaces/789/test_sessions/123")
spec.name	The test spec name assigned in the file (i.e. # Name)
fixture.name	The name defined in the before/after section of the spec
fixture.type	before or after
fixture.description	The friendly description (optional)

The context is serialized into a single JSON-string.

The following is an example of a context that has been deserialized (shown as a JavaScript object, not strict JSON — object keys aren't quoted):

{
  user: { id: 173, name: 'Joe Smith' },
  project: {
    id: 6977,
    name: 's2technologies:testspace.test.repo',
    url: 'https://s2.testspace.com/projects/6977'
  },
  space: {
    id: 22701,
    name: 'github.fixture.v2a',
    url: 'https://s2.testspace.com/spaces/22701'
  },
  session: {
    id: 64952,
    name: "Run 'gh.fixture.normal' spec",
    url: 'https://s2.testspace.com/spaces/22701/test_sessions/64952'
  },
  spec: {
    id: 10387858,
    name: 'gh.fixture.normal',
    filename: 'gh.fixture.normal.md',
    filepath: 'specs',
    run_id: '846be03d-d2d4-4f12-becb-472483c09950'
  },
  fixture: {
    type: 'before',
    name: 'github::normal',
    description: 'before - simple input',
    timeout: 300
  },
  repo: {
    url: 'https://github.com/s2technologies/testspace.test.repo',
    branch: 'github.fixture.v2a'
  }
}

GitHub Workflows

GitHub has built-in CI/CD support. Testspace enables leveraging of this functionality in the context of executing a manual spec. To invoke a workflow as an automated fixture using GitHub, a GitHub Actions workflow file (under .github/workflows/) is required in the repo — not to be confused with Testspace's own .testspace.yml.

Testspace uses a Workflow Dispatch event to trigger the workflow, using the required inputs definition.

name: Testspace
run-name: ${{ fromJSON(inputs.context).spec.name }}
on:
  workflow_dispatch:
    inputs:
      name:
        description: 'Function name'
        required: true
      payload:
        description: 'Function input-payload'
        required: true
      context:
        description: 'Function execution-context'
        required: true
...

info

A test spec uses the name field to invoke a specific job defined in the workflow.

---
testspace:
before:
  name: github::fn-name
---
...

A name is defined in the test spec, which is used to execute a corresponding job.

jobs:
  fn-name:
    if: inputs.name == 'fn-name'
  ..

tip

Use job names to represent unique function calls required by different test specs.

Template

The following is an example workflow yml file.

name: Testspace
run-name: ${{ fromJSON(inputs.context).spec.name }}
on:
  workflow_dispatch:
    inputs:
      name:
        description: 'Function name'
        required: true
      payload:
        description: 'Function input-payload'
        required: true
      context:
        description: 'Function execution-context'
        required: true
env:
  IN_NAME:    ${{ inputs.name }}
  IN_PAYLOAD: ${{ inputs.payload }}
  IN_CONTEXT: ${{ inputs.context }}
jobs:
  name1:
    if: inputs.name == 'fn-name1'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Echo test spec function name
        run: echo "running ${IN_NAME} function"

info

When calling a handler script always use environment variables when referencing IN_NAME, IN_PAYLOAD, and IN_CONTEXT.

For example, using a JavaScript executed from a workflow:

function handler(payload) {
  ...
}
handler(JSON.parse(process.env.IN_PAYLOAD));

The following example test spec works with the workflow above (i.e. name: fn-name1):

---
testspace:
before:
  name: github::fn-name1
---

# TEST1
## Case 1
- do this
- do that

note

When a workflow fails do not re-run the failed job using the GitHub Actions UI. Testspace has no way of tracking the execution.

AWS Lambdas

AWS has built-in serverless function execution support using AWS Lambdas. This functionality fits very well with Testspace automated fixturing. To enable Testspace invoking Lambdas refer here for setup information.

Handler

Lambda handlers can be written in any language supported by AWS. The following is a Lambda function for Node.js:

exports.handler = async (event, context) => {

    console.log("Input:", event);
    // TO DO

    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

The AWS Lambda runtime automatically converts the JSON Object to the corresponding language object type.

Logging

Testspace captures the input parameters sent to the Lambda and the log stream generated by the function as an annotation for the corresponding Result Suite.

Standalone Differences

Automated fixtures work on Standalone projects too, with a few differences from Connected:

• Prerequisite. A github:... fixture needs the Company connected to GitHub. For any fixture that omits the repository, or uses the short form (e.g. github:manual:fn-name1), the Project's Issues provider setting must also be set to a GitHub org/repo — that's where the default repository comes from.
• Default-repo resolution differs from Connected. On Connected, a blank/short repository name defaults to the project's own connected repo. On Standalone — which has no repo of its own — it defaults to whatever repo is set as the project's Issues provider instead.
• Not limited to the Issues-provider repo. A fixture can still target any org/repo the Company's GitHub App installation can reach, using the full form (e.g. github:acme/other-repo:fn-name1) — same as Connected. Issues provider only supplies the default, it isn't a restriction.
• The repo.* fixture context is always blank. The metadata sent to the triggered GitHub Actions workflow or AWS Lambda (context.repo.url / context.repo.branch — see Context above) is always empty strings for Standalone, since there's no real repo/branch to report. This is the fixture-execution counterpart to the {{ repo.* }} Liquid variables in Writing a Spec, which behave the same way.

info

Known limitation: the payload: "@relative/path.json" file-reference form of a payload does not currently work on Standalone projects, for both github:... and aws:... fixtures — it depends on fetching the file from a connected repo, which a Standalone project doesn't have. A plain inline payload: {...} hash (as shown in Before above) works today with no restrictions.

Execution

A spec with a before fixture, and optionally an after fixture, has extra execution behavior. A ⚙️ icon next to START indicates a required action:

Before Fixture

The before fixture is required to be successfully processed before test cases can be executed. It is automatically triggered by clicking on the blue START button. While running a gray spinning icon would appear.

In case of failure, a red cross icon would indicate unsuccessful automation. The tooltip will provide additional failure information.

The before fixture has several constraints that govern the execution of the spec:

• Closing the dialog while executing will cancel the execution
• Closing and opening the Spec will require re-execution of the automation
• Executing longer than 5 minutes, the user will be prompted to Continue with execution

info

For Fixtures executing longer than 5 minutes, the user will be prompted to Continue with execution.

After Fixture

The after fixture is optional and can only exist if there is also a before fixture. If the Spec has successfully executed, once the Spec dialog is closed, the associated after fixture automation is executed, in the background, without any status available to the tester.

info

The after fixture will automatically execute when:

• The user selects STOP or X within the spec run dialog
• The browser tab or computer is shutdown
• No user activity for 10 minutes (user prompted with 60-second countdown)