Testspace

Automation

Testspace support for serverless automation using GitHub Actions and AWS Lambdas enables testing solutions that can leverage a hybrid of automation and manual verification.

See spec fixture section for details.

Implement more effective tests using a hybrid of automation and human observations.

For some of the values of integrating automation with manual execution:

  • Reduction in Manual Execution Time. Leverage automation for tedious and redundant setup/teardown requirements vs human execution.

  • More Effective Test Coverage. By leveraging a combination of automated fixturing and the power of human observations, more problems often can be discovered. Fully automating everything can have significant limits in verification and be time-consuming to implement and maintain.

  • Eliminated IT Setup for Human Tester. All testing, including automation, is executed in the context of a Browser. Automation is executed leveraging callable serverless functions and abstracted from the human tester. There are no dependencies on the computer being used by the Tester.

  • Test specifications and automation maintained together in a repo. All of the test artifacts are managed as code, leveraging the same development process as the application team.

Input

A Fixture's Input is defined within a calling specification's header metadata block. Refer to the spec fixture section for details.

---
testspace:
before:
  name: {name}
  input:
    p1: 1st parameter
    p2: 2nd parameter
    p3: 
      - one
      - two
      - three
    p4: 
      this: one
      that: other one
    ..
    p10: "@file.json"
---

There are up to 10 top-level parameters supported. The value of a parameter is set using one of the following types:

  • string/number
  • embedded YAML
  • File

Each parameter value defined as embedded YAML is serialized as a JSON-string and is required to be deserialized by the fixture handler.

Parameter values defined as embedded YAML are required to be deserialized

For parameter values defined with a file reference (i.e. @file.json) are encoded as base64 and required to be unencoded.

Example

The following JavaScript handler example is passed the input parameters as a JSON String.

function handler(input, ..) {
    var input = JSON.parse(input);
    console.log("Input: ", input);

    console.log("p1:", input.p1);
    console.log("p2:", input.p2);
    
    var p3 = JSON.parse(input.p3);
    console.log("p3", p3);

    var p4 = JSON.parse(input.p4);
    console.log("p4", p4);
    ..
    var p10 = Buffer.from(input.p10, 'base64').toString();
    console.log("p10", JSON.parse(p10));
}

GitHub Workflows

GitHub has built-in CI/CD support. Testspace enables leveraging of this functionality in the context of executing a manual spec.

Workflow

A GitHub workflow is used to specify the runtime environment where the serverless automation handler is going to be executed. As Testspace uses a Repository Dispatch event to trigger the workflow, it has to be configured to start on an external event, i.e. respository_dispatch event:

on:
  repository_dispatch:
    types:[workflow-name]

By default, all workflows within the repo will be triggered when configured for the respository_dispatch event. The types can be used for further filtering based on the identifier's name as defined in the corresponding spec's fixture header block:

spec.md:

---
testspace:
before:
  name: github::workflow-name
---

For the workflow file - myfixture.yml:

on:
  repository_dispatch:
    types: [myfixture]

Testspace sets the respository_dispatch event's type (aka event action) as the identifier's name defined in the Spec when executing a fixture. The workflow file name is required to match.

A step in the workflow specifies the way the serverless automation handler should be executed. For example, running a script (e.g. handler.js):

jobs:
  my-job:
    runs-on: ubuntu-latest
    steps:
      - name: Call handler
        run: node handler.js "$PAYLOAD"
        timeout-minutes: 5
        env:
          PAYLOAD: ${{ toJson(github.event.client_payload) }}

The github.event variable provides the Spec's fixture context being called:

  • github.event.action represents the spec's "workflow-name". Notice, it is the same as the respository_dispatch event's types that the workflow is filtered on.
  • github.event.client_payload is the "input". Notice, it needs to be serialized as JSON string before being passed to a script handler.
  • the fixture's input is constrained to up-to 10 top-level input parameters.

Note, on a workflow failure do not re-run the failed job using the GitHub Actions UI. Testspace has no way of tracking that execution.

Handler

The serverless automation handler could be implemted in a language of your choice. Here is a JavaScript example (e.g. handler.js, referenced above):

function handler(payload) {
  var input = JSON.parse(payload);
  console.log("Input: ", input.a, input.b);
}
handler(process.argv[2]);

AWS Lambdas

AWS has built-in serverless function execution support using AWS Lambdas. This functionality fits very well with Testspace automated fixturing. To enable Testspace invoking Lambdas, access keys are required to be set up for the account.

Handler

Lambda handlers can be written in any language supported by AWS. The following is a Lambda function for Node.js:

exports.handler = async (event, context) => {

    console.log("Input:", event);
    // TO DO
      
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

The ASW Lambda runtime automatically converts the JSON Object to the corresponding language object type.

Logging

Testspace captures the input parameters sent to the Lambda and the log stream generated by the function as an annotation for the corresponding Result Suite.

← ConfigurationSession →