Test Description

Learn more about specific features and configurations of the Skyramp test description.

Introduction

The test description in Skyramp simplifies the process of writing and running tests for complex distributed applications. It consists of three primary components:

  1. Test Configuration: This file, residing in the tests folder, defines the overall behavior of the test. It enables you to configure test patterns and load testing.

  2. Scenario Configuration: Found in the scenarios folder, these files define request behavior for specific methods or chains of requests and asserts in scenarios. They also allow you to configure payloads, dynamic requests, and set overrides and parameters.

  3. Endpoint Configuration: Located in the endpoints folder, these files specify details related to the service’s networking aspects, supporting gRPC, REST, JSON-RPC WebSocket, and JSON-RPC HTTP endpoints.

To get started, follow the steps outlined in the How to Test Services page. This guide will teach you how to dynamically generate a test description by providing service-level information. Alternatively, if you prefer to create a test definition from scratch, you can create .yaml files in the tests, scenarios, and endpoints directories of your project (e.g., my-test.yaml, my-scenario.yaml, and my-endpoint.yaml) and configure the necessary information by following the guidelines below.

Test Configuration

The test configuration serves as the central component of the test definition and defines the overall test behavior.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - startAt: 1
      scenarioName: scenario1
      atOnce: 2
      targetRPS: 3000
      duration: 10
      rampUp:
        duration: 3
        interval: 1
  override:
    mock:
      - endpointName: helloworld
        methodName: SayHello
        blob: |
          {
              "message": "myTest"
          }

In this example:

  • name: Specifies the name of your test.
  • testPattern: Enables the definition of various test scenarios.
  • override: Allows you to override mock behavior.

This example showcases advanced test capabilities, including:

  • Overrides: Customizing endpoint behaviors by specifying mocks.
  • Load Testing: Simulating heavy user traffic with features like target RPS and ramp-up controls.

Overrides

The override attribute in the test configuration allows you to customize specific endpoints within your tests, providing flexibility in your testing scenarios. By setting the override attribute, you can specify a mock to modify an endpoint defined elsewhere in the project folder. This customization enables you to simulate various scenarios and test your application’s robustness effectively.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - startAt: 1
      scenarioName: scenario1
  override:
    mock:
      - endpointName: routeguide_jMBp
        methodName: GetFeature
        blob: |-
          {
            "name": "fake",
            "location": {
              "latitude": 400,
              "longitude": 600
            }
          }

The override attribute is used to customize endpoints. The mock section in the example specifies the details of the override:

  • endpointName: Identifies the endpoint you want to override, in this case, routeguide_jMBp.
  • methodName: Specifies the specific method of the endpoint you wish to modify, here, GetFeature.
  • blob: Contains the customized data or response that will replace the original response from the endpoint.

By leveraging the override attribute, you can seamlessly adapt endpoint behaviors within your tests, allowing you to create various testing scenarios and evaluate your application’s performance under different conditions.

Load Testing

Load testing allows you to simulate heavy user traffic on your application by transforming functional tests into load tests. This can be achieved by incorporating specific load profile keywords into your tests.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - startAt: 1
      scenarioName: scenario1
      atOnce: 2
      targetRPS: 3000
      duration: 10
      rampUp:
        duration: 3
        interval: 1

The atOnce attribute in the testPattern signifies the concurrency of the scenario1. With atOnce set to 2, everything defined in scenario1 will run with a concurrency of 2, happening in parallel.

To control the load, you can utilize the following parameters:

  • targetRPS: Specifies the target Requests Per Second (RPS) your application should handle.
  • duration: Indicates the total duration of the load test in seconds.
  • rampUp: Allows you to gradually increase the load on an endpoint.
    • duration: Specifies how long it takes for the traffic ramp-up to occur.
    • interval: Signifies the rate at which traffic increases.

In the provided example, the load increases over 3 seconds, with increments every second until it reaches the target RPS of 3000. Modifying these values enables you to model traffic behavior for your application and test how it responds to varying levels of load. If targetRPS is not specified, Tester will attempt to send as many requests as possible within the system’s context.

Scenario Configuration

The scenario configuration file defines request behaviors for specific methods and creates scenarios as chains of defined requests and asserts.

Example Scenario Configuration:

version: v1
scenarios:
  - name: scenario1
    steps:
      - requestName: GetFeature_18GO
      - asserts: requests.GetFeature_18GO.res.name == "fake"
requests:
  - name: GGetFeature_18GO
    javascript: |-
      function handler(req) {
        // Your JavaScript logic here
        return {
          value: {
            latitude: x,
            longitude: y
          }
        };
      }      
  - name: ListFeatures_5P9V
    blob: |-
      {
        "hi": {
          "latitude": 0,
          "longitude": 0
        },
        "lo": {
          "      

latitude": 0,
          "longitude": 0
        }
      }
    endpointName: routeguide_jMBp
    methodName: ListFeatures
  # More request configurations...

In this example:

  • scenarios: Allows you to define different test scenarios, and this example includes a scenario named scenario1.
  • requests: Contains configurations for individual requests (to be referenced in scenarios by name).

Some advanced capabilities of the scenario configuration include:

Scenarios and Asserts

Scenarios are representations of end-user use cases that require testing, allowing you to define a sequence of actions to be performed, typically involving requests to specific endpoints. Each named scenario includes a steps attribute, listing these actions, which can be executed sequentially or concurrently, simulating various usage patterns, including load tests.

Example Scenario Configuration:

version: v1
scenarios:
  - name: scenario1
    steps:
      - requestName: GetFeature_18GO
      - asserts: requests.GetFeature_18GO.res.name == "fake"

requests:
  - name: GetFeature_18GO
    endpointName: routeguide_jMBp
    methodName: GetFeature
    javascript: |-
      function handler(req) {
        // Your JavaScript logic here
        return {
          value: {
            latitude: x,
            longitude: y
          }
        };
      }      
  # More request configurations...

In this example:

  • scenario1 is defined, containing two steps executed sequentially.
  • The first step utilizes the requestName attribute, referencing a request object previously defined in the configuration.
  • The second step is an assert statement, used to verify that the response from the request matches the expected value.

To use an assert, the asserts parameter is defined within the step, with a value in the format:

requests.<name of request>.res.message == "<expected value>"

Where <name of request> refers to the name of the previously defined request, and <expected value> is a string representing the expected return value from the request.

Note: Values returned by services are interpreted as JavaScript for evaluating assert statements. The type may not always be a string, but could be a boolean or a number, among other types. When working with a boolean, the <expected value> should be true or false and not "true" or "false" in the assert statement.

In scenarios, the associated steps are executed sequentially. To execute items in parallel, refer to the testPattern defined in the test configuration.

Dynamic Requests

Dynamic requests provide the flexibility to customize request handling logic in your scenario configurations. You can use different attributes to specify dynamic request behavior, such as python, pythonPath, javascript, or javascriptPath. Each attribute allows you to define custom request handling logic and return a JSON representation of the response value.

JavaScript Dynamic Requests

  1. Using the javascript Attribute

    To create JavaScript-based dynamic requests, employ the javascript attribute within the requestValue section of your request definition. Define a function called handler that takes the req parameter. Implement your custom JavaScript logic within the handler function, and return a JSON object representing the response value.

    Example Scenario Configuration:

    version: v1
requests:
  - name: GetFeature_18GO
    javascript: |-
      function handler(req) {
        // Your JavaScript logic here
        return {
          value: {
            latitude: x,
            longitude: y
          }
        };
      }      
    endpointName: routeguide_jMBp
    methodName: GetFeature
  # More request configurations...

  2. Using the javascriptPath Attribute

    Alternatively, you can use the javascriptPath attribute to specify the path to an external JavaScript script file that contains your custom request handling logic.

    Example Scenario Configuration:

    version: v1
requests:
  - name: GetFeature_18GO
    javascriptPath: scripts/getFeature.js
    endpointName: routeguide_jMBp
    methodName: GetFeature
  # More request configurations...

    The external JavaScript script file getFeature.js defines a handler function to process incoming requests and generate appropriate responses.

Installing NPM-Based Packages

If your JavaScript-based dynamic requests require NPM-based packages, follow these steps:

Specify the required packages in the npmPackages section of your test definition. The testing framework will automatically install these packages before running your test.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - requestName: GetFeature_18GO
      startAt: 1
    - requestName: ListFeatures_5P9V
      startAt: 1
    - requestName: RecordRoute_MD5D
      startAt: 1
    - requestName: RouteChat_QQEf
      startAt: 1
  npmPackages:
    - mathjs
    - chart

Python Dynamic Requests

Note: If your dynamic Python request is dependent on additional Python modules, refer to the Installing Worker with Python Modules section to learn how to build the custom Skyramp Worker image.

  1. Using the python Attribute

    You can use the python attribute within the requestValue section of your request definition. This attribute allows you to define a function called handler that takes the req parameter, representing the incoming request. Within the handler function, you can implement your custom Python logic. Finally, return a JSON representation of the response value using SkyrampValue.

    Example Scenario Configuration:

    version: v1
requests:
  - name: GetFeature_18GO
    python: |-
      def handler(req):
        // Your Python logic here
        return SkyrampValue(
          value={
            "latitude": x,
            "longitude": y
          }
        )      
    endpointName: routeguide_jMBp
    methodName: GetFeature
  # More request configurations...

  2. Using the pythonPath Attribute

    Alternatively, you can use the pythonPath attribute to specify the path to an external Python script file containing your custom request handling logic.

    Example Scenario Configuration:

    version: v1
requests:
  - name: GetFeature_18GO
    pythonPath: scripts/get_feature.py
    endpointName: routeguide_jMBp
    methodName: GetFeature
  # More request configurations...

    The external Python script file get_feature.py defines a handler function to process the request and generate the response.

Chaining and Overrides

Tester provides a powerful feature that allows you to chain values between sequential requests and override them.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - startAt: 1
      scenarioName: scenario1
  override:
    mock:
      - endpointName: routeguide_jMBp
        methodName: RouteChat
        javascript: |-
          function handler(req) {
            return {
              value: {
                message: req.value.message + "temp",
                location: {
                  latitude: req.value.latitude,
                  longitude: req.value.longitude
                }
              }
            }
          }

Example Scenario Configuration:

version: v1
scenarios:
  - name: scenario1
    steps:
      - requestName: RouteChat_QQEf
      - asserts: requests.RouteChat_QQEf.res.message == "message1temp"
      - requestName: RouteChat_QQEf
        override:
          message: requests.RouteChat_QQEf.res.message
      - asserts: requests.RouteChat_QQEf.res.message == "message1temp1temp"
      - requestName: RouteChat_QQEf
        override:
          message: requests.RouteChat_QQEf.res.message
      - asserts: requests.RouteChat_QQEf.res.message == "message1temp1temp1temp"

requests:
  - name: RouteChat_QQEf
    endpointName: routeguide_jMBp
    methodName: RouteChat
    vars:
      message: "message"
    javascript: |
      i = 0
      function handler() {
        i++
        return {
          value: {
            message: vars.message + i,
            location: {
              latitude: req.value.latitude,
              longitude: req.value.longitude
            }
          }
        }
      }      
  # More request configurations...

In this example:

  • The override attribute in the test section allows you to customize the behavior of an endpoint defined elsewhere in the project folder by specifying a mock.

  • Within the scenarios section, multiple steps are defined. Each step calls the RouteChat_QQEf request and includes an assert. What sets them apart is that in subsequent requestName calls, the message variable is overridden. The message variable takes on the value of the response returned by the request. This chaining is done multiple times to create a sequence of messages.

  • The requests attribute shows how the vars keyword is used to define a new variable called message in the RouteChat_QQEf request. This variable is utilized in the JavaScript snippet using vars.message. By overriding this variable in scenario1, you can modify the message content in the subsequent requests.

This feature allows you to create dynamic test scenarios where the output of one request influences the behavior of subsequent requests, making your testing more versatile and powerful.

Request Parameters and Headers

When making REST calls, requests often require headers, such as Basic Authentication information, and variables in the path. You can achieve this using the request object.

Example Scenario Configuration:

version: v1
requests:
  - name: addCartRequest
    endpointName: cart-service
    methodName: cart-service-post
    blob: |
      {
          "product_id": "OLJCESPC7Z",
          "quantity": 1
      }      
    headers:
      Authorization: "Basic YWxhZGRpbjpvcGVuc2VzYW1l"
    params:
      - name: user_id
        in: path
        value: abcde

In this example:

  1. Path Parameters: In the cart-service endpoint, if the path contains a path parameter (/cart/user_id/{user_id}), we can define a params attribute and set user_id to abcde. Importantly, because we set in to path, it’s treated as a path parameter. You can also set in to query to make it a REST query parameter.

  2. Headers: The headers attribute adds a header with the key Authorization and the value "Basic YWxhZGRpbjpvcGVuc2VzYW1l". It allows you to include headers in your requests, which is often required for authentication and other purposes.

Endpoint Configuration

The endpoint configuration file defines networking-level service details for an endpoint.

Example Endpoint Configuration:

version: v1
services:
  - name: routeguide
    port: 50051
    alias: routeguide
    protocol: grpc
endpoints:
  - name: routeguide_jMBp
    methods:
      - name: GetFeature
      - name: ListFeatures
      - name: RecordRoute
      - name: RouteChat
    defined:
      path: ./pb/route_guide.proto
      name: RouteGuide
    serviceName: routeguide

For configuring the endpoint file, we have a few key attributes:

  • services: This section lists the services available in your project. In this example, there is one service named routeguide.

  • endpoints: Under the endpoints section, you define individual endpoints, specifying the available methods, the service definition path, and the service name. In the example, we have an endpoint named routeguide_jMBp for the RouteGuide service.

  • methods: Within each endpoint, you list the available methods. In this case, we have methods like GetFeature, ListFeatures, RecordRoute, and RouteChat. This helps specify the details of each method and how it should behave.

  • defined: Here, you specify the service definition file’s path and the service name. The service definition file (route_guide.proto) outlines the structure of the service and its methods.

By configuring endpoints, you define the available services and methods within your project, facilitating the testing of your distributed application. We recommend dynamically generating a test by providing service-level information.