User Tools



Metrics

Testspace supports two types of Metrics for viewing data of interest in a historical manner. One type is Standard Metrics for items such as test results, code coverage and static analysis. The second type is Custom Metrics which can be from any data source that is uploaded in a CSV file of a specified format. Metrics data can be viewed both as suites in an individual result set and in graphs with historical data in the Metrics tab.


Standard Metrics

Standard Metrics that are available on the Metrics tab include test results, code coverage and static analysis.1) These are populated when results of these types are published to the space.

The Metrics tab by default when a Space is created will have graphs for both Test Suites and Test Cases. Code Coverage and Static Analysis Metrics are automatically created for your Space when their content is published for the first time.

Schema Tab

Metrics Suites in the Schema are denoted with a graph icon (1) and the content of the Metrics is viewable by clicking on the icon.

Metrics Tab

The Metrics tab displays all Metrics graphs that have been created either by default or added by a user.

  1. Badges that have been elevated
  2. Time frame to display in graph
    • Recent is the default and will show up to the last 30 result sets.
    • For other time frames the individual plotted points (i.e. stacked columns) that will be shown are from a single result based on the following rules:
      • The result set with the highest number of case count
      • If case count the same then the result sets with highest pass percentage
  3. Button for adding additional Metrics
  4. Information for result set used in plot

For Code Coverage suites, the graphs that are created correspond to the data type in your coverage file. In addition to the graph, a badge is created for each type as well with an automatically created minimum threshold of 50%. For Static Analysis it is broken into three categories low, medium and high, with an informational badge that shows the sum of the medium and high issues.

To edit the Metrics name or Threshold choose the hamburger menu (1) next to the name and select edit. In the Edit Metric dialog box you can update the following:

  1. Name of the Metric seen in the Metrics tab
  2. Update the minimum threshold for the suite to pass. If there is no threshold value, then the Metric will not contribute to a Space's Health
  3. Choose whether you want a badge from this metrics and if you want it to contribute to a Space's Health
  4. Name seen on the badge
:!: Note: If a Space was created as the result of a new branch in a "connected" Project or by copying settings from an existing Space in a "standalone" Project then the last Result Set of the originating Space is shown as a baseline (1) - with dimmed-down color and a tooltip displaying the Space and Result Set name of where it was copied from.

:!: Note: If a Metrics path changes due to updates to published content or data that is missing due to other issues, this is denoted by the path turning red(1). In the case of the metric no longer being needed due to content changes it can be safely deleted in the edit panel for the metric.


Custom Metrics

Testspace allows for publishing Custom Metric data for use in graphing, creating thresholds and badges from such things as performance data, number of errors or warnings seen in test logs, bugs that are not covered by tests, etc. This can be used for tracking any numerical value that is of interest over time as well as be used to determine a Space's Health.

cProfile Example

In this example profiling data is created using Python cProfile on various Python math functions to demonstrate an example of how a log file with data of interest can be turned into Custom Metrics.

output.txt

         50000002 function calls in 10.118 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 10000000    2.685    0.000    2.685    0.000 {math.trunc}
 10000000    2.630    0.000    2.630    0.000 {math.cos}
 10000000    2.099    0.000    2.099    0.000 {math.frexp}
 10000000    1.573    0.000    1.573    0.000 {math.sqrt}
 10000000    0.987    0.000    0.987    0.000 {math.isnan}
        1    0.234    0.234    0.234    0.234 {range}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

This text file is used as the basis for the metric data to be published to Testspace in a CSV file as shown below.

cprofile.csv

Total Time, 10.118
Function Time, 2.685, 2.630, 2.099, 1.573, 0.897

Publish

testspace output.txt{cprofile.csv} "space-name"

The published data can be viewed in a result by choosing the suite corresponding to the name of the text file uploaded (1). In the dialog you are able to view the data from the CSV file (2) as well as the content of the text file (3).

Metrics Graphs from cProfile Data


Adding Metrics Graphs

Note that each Metric data source can only have one graph. To create a second graph duplicate the data (additional row) in the CSV file.

From the new metric button in the upper right corner of the Metrics tab on a Space page choose the Metric you would like to add (1) and click submit to add. If you would like to customize your Metric select the customize button (2).


Metric Dialog

Options for customizing your metric include:

  1. Changing name, including if none was set in CSV file
  2. Graph type, line or stacked column
  3. Max range of vertical axis
  4. Unit of measure
  5. Name of the key value


Metric Calculations

Testspace has the ability to do custom calculations on your metric data for use in either creating a new data point in the graph or for use in thresholds. Examples of this include summing the data to find a total and calculating an average for display on the graph.

Functions

For doing the calculations the mathematical functions listed below are available for use.

Math: +, -, *, /, %

Logic: <, >, <=, >=, <>, !=, =, AND, OR

Functions: IF, NOT, MIN, MAX, ROUND, ROUNDDOWN, ROUNDUP

String: LEFT, RIGHT, MID, LEN, FIND, SUBSTITUTE, CONCAT

Variables

The three variables defined in the customize metrics dialog are:

  • d - This is for the data from the input CSV file in order of the data in the file (d1, d2, etc…)
  • v - These are defined for each variable line (v1-v5) and are each graphed. Use these even if you have defined different keys for your graph.
  • t - Thresholds appear on the graph as smaller dots to mark bounds for your data.

cProfile Example

In this example there are three data points (d1, d2, d3) seen in the value's box(1). To show the sum of them on the graph they are added up in variable 4 (2) and labeled as total, which is now shown in the graph as well. These calculations can be added anywhere there is a Set Value Expression Here (3).

output.csv
Function Time, 2.640, 1.972, 1.390

Custom Badges

For some metrics a badge may be desired to elevate certain data to higher levels for increased visibility.

Options to set for the Badge include:

  1. Threshold value
  2. Enable Badge to be placed at top of Space view and in both Project and Organization listings.
  3. Badge label
  4. Value to be displayed in badge. If a unit is desired to be added to a numerical value used the concat function.
  5. Pass/Fail criteria for badge. If this expression does not return true then the Metric is set to fail and will cause the result to marked as unhealthy.

Spaces

1) See this article for of examples of how to push-data

Page Tools