Tester

• 1: How to Test Services
• 2: Test Description

1 - How to Test Services

Tester simplifies writing and running tests for complicated distributed apps in four simple steps:

1. Create a test
   • Write a test from scratch or
   • Generate a test from user input
2. Configure the test
3. Run the test
4. Analyze test results

Prerequisites

Before using Tester, you must install the Terminal Client on your machine and the Skyramp Worker in the environment where you want to run tests.

1. Create a Test

To create a test configuration, you have two options:

Write a Test from Scratch

Use the init test command to create a template test description. This template file will require user input to supply information about the services and endpoints you would like to test.

Follow the protocol sub-commands below to generate a test template based on your API protocol: gRPC or REST.

• gRPC
• REST

  skyramp init test grpc <test name>
  

  skyramp init test rest <test name>
  

Generate a Test from User Input

Use the generate command to generate a default test configuration to use and start with. These tests will work out-of-the-box but are intended to be a starting point for more complicated testing logic in different applications.

Follow the protocol sub-commands below to generate configurations based on your API protocol: gRPC or REST.

• gRPC
• REST

  skyramp tester generate grpc \
    --api-schema <path to .proto file> \
    --alias <name Docker alias to test> \
    --port <port number for the service> \
    --service <name of proto service>
  

  skyramp tester generate rest \
    --api-schema <path to OpenAPI schema file or URL (URL support for OpenAPI 3.x only)> \
    --alias <name of Docker alias to test> \
    --port <port number for the service> \
    --tag <optional OpenAPI tag to filter on> \
    --paths <optional REST paths to filter on>  \
    --crud-scenarios <enable generating CRUD (Create, Read, Update, Delete) scenarios for REST endpoints>
  

This command performs the following actions:

• Creates a test configuration file in the tests folder for the specified service or alias.
• Creates a scenario configuration file in the scenarios folder for each method defined in the service.
• If the endpoint definition does not already exist, it creates an endpoint configuration file in the endpoints folder.

You can edit the generated .yaml files as explained in the next section.

2. Configure the Test

The test description created with the init test command serves as a template for a test. It prompts you to fill in the necessary information to configure specific details such as the endpoint, scenario, and overall test configuration.

On the other hand, the test description generated using the tester generate command is designed to be ready-to-use out-of-the-box. Despite its immediate usability, you still retain the flexibility to add or modify configuration details. This allows you to customize and enhance your tests according to your specific requirements.

Below are examples of endpoint, scenario, and test configurations:

Endpoint Configuration

Endpoint configuration files can be found in the endpoints folder. Here’s an example of an endpoint configuration for a gRPC service:

version: v1
services:
    - name: routeguide
      port: 50051
      alias: routeguide
      protocol: grpc
endpoints:
    - name: routeguide_jMBp
      methods:
        - name: GetFeature
        - name: ListFeatures
        - name: RecordRoute
        - name: RouteChat
      defined:
        path: ./pb/route_guide.proto
        name: RouteGuide
      serviceName: routeguide

The generated endpoint configuration contains networking-level service details, including its name, port, alias, and protocol. Additionally, it contains metadata related to various endpoints that a service can have, including the methods it supports and the associated proto file.

Scenario Configuration

To configure scenarios, edit the scenario configuration file in the scenarios folder. Here’s an example of a scenario configuration for a gRPC service:

version: v1
requests:
    - name: GetFeature_18GO
      blob: |-
        {
          "latitude": 0,
          "longitude": 0
        }        
      endpointName: routeguide_jMBp
      methodName: GetFeature
    - name: ListFeatures_5P9V
      blob: |-
        {
          "hi": {
            "latitude": 0,
            "longitude": 0
          },
          "lo": {
            "latitude": 0,
            "longitude": 0
          }
        }        
      endpointName: routeguide_jMBp
      methodName: ListFeatures
    # More request configurations...

In this example, you define request behavior for a specific method of the service and specify the endpoint and method name as defined in the endpoint configuration. You can customize the request payload using a static JSON blob. To create more complex requests, you can add advanced capabilities such as defining dynamic requests and adding overrides and parameters.

The scenario file generated using tester generate by default only lists the aforementioned requests. To add advanced capabilities like assertions and chaining, you can create additional scenario files in the scenarios folder. These files can reference the generated requests and provide more complex test scenarios.

You can read about all advanced capabilities in the Test Description page.

Test Configuration

To configure the overall test behavior, edit the test configuration file in the tests folder. Here’s an example of a test configuration for a gRPC service:

version: v1
test:
    name: routeguide
    testPattern:
        - requestName: GetFeature_18GO
          startAt: 1
        - requestName: ListFeatures_5P9V
          startAt: 1
        - requestName: RecordRoute_MD5D
          startAt: 1
        - requestName: RouteChat_QQEf
          startAt: 1

In this example, you configure the top-level test’s behavior, including setting a name for the test and specifying the test pattern by referencing requests and scenarios as defined in the scenario configuration. Advanced capabilities like configuring load testing available in the Test Description page.

3. Run the Test

Once the test description is in place, you can run the test. To run the test, you first need to reference the specific test file that will be used. To do this, refer to the tests directory. The name of the test will be the name of the respective file containing the test, without the file extension. For example, if you have a file located at tests/checkout-test.yaml, the test file name will be checkout-test.

  skyramp tester start <test file name> \
    -a <ip:port of the Skyramp worker>

After running the command above to start the test, there will be output on the progress/status of the test until it finishes. Note that if you interrupt the command (such as by sending a SIGINT), the test will continue to run in the background in the Skyramp worker. For an ongoing test, you can check the status by running skyramp tester status <test file name> and providing either the address or namespace, as previously described.

Stop the Test

To stop an ongoing test, run skyramp tester stop <test file name>, once again with either the address or namespace arguments.

Learn more about what you can configure in your test description on the Test Description page.

4. Analyze Test Results

There are several ways to view and retain test results:

1. Generate an HTML test report by running skyramp tester start with the optional --html-report flag. The report is saved in the “results” directory of your working directory.

2. Start a Skyramp dashboard by running skyramp dashboard up. Refer to the dashboard documentation to learn more about managing test results.

2 - Test Description

Introduction

The test description in Skyramp simplifies the process of writing and running tests for complex distributed applications. It consists of three primary components:

1. Test Configuration: This file, residing in the tests folder, defines the overall behavior of the test. It enables you to configure test patterns and load testing.

2. Scenario Configuration: Found in the scenarios folder, these files define request behavior for specific methods or chains of requests and asserts in scenarios. They also allow you to configure payloads, dynamic requests, and set overrides and parameters.

3. Endpoint Configuration: Located in the endpoints folder, these files specify details related to the service’s networking aspects, supporting gRPC, REST, JSON-RPC WebSocket, and JSON-RPC HTTP endpoints.

To get started, follow the steps outlined in the How to Test Services page. This guide will teach you how to dynamically generate a test description by providing service-level information. Alternatively, if you prefer to create a test definition from scratch, you can create .yaml files in the tests, scenarios, and endpoints directories of your project (e.g., my-test.yaml, my-scenario.yaml, and my-endpoint.yaml) and configure the necessary information by following the guidelines below.

Test Configuration

The test configuration serves as the central component of the test definition and defines the overall test behavior.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - startAt: 1
      scenarioName: scenario1
      atOnce: 2
      targetRPS: 3000
      duration: 10
      rampUp:
        duration: 3
        interval: 1
  override:
    mock:
      - endpointName: helloworld
        methodName: SayHello
        blob: |
          {
              "message": "myTest"
          }          

In this example:

• name: Specifies the name of your test.
• testPattern: Enables the definition of various test scenarios.
• override: Allows you to override mock behavior.

This example showcases advanced test capabilities, including:

• Overrides: Customizing endpoint behaviors by specifying mocks.
• Load Testing: Simulating heavy user traffic with features like target RPS and ramp-up controls.

Overrides

The override attribute in the test configuration allows you to customize specific endpoints within your tests, providing flexibility in your testing scenarios. By setting the override attribute, you can specify a mock to modify an endpoint defined elsewhere in the project folder. This customization enables you to simulate various scenarios and test your application’s robustness effectively.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - startAt: 1
      scenarioName: scenario1
  override:
    mock:
      - endpointName: routeguide_jMBp
        methodName: GetFeature
        blob: |-
          {
            "name": "fake",
            "location": {
              "latitude": 400,
              "longitude": 600
            }
          }          

The override attribute is used to customize endpoints. The mock section in the example specifies the details of the override:

• endpointName: Identifies the endpoint you want to override, in this case, routeguide_jMBp.
• methodName: Specifies the specific method of the endpoint you wish to modify, here, GetFeature.
• blob: Contains the customized data or response that will replace the original response from the endpoint.

By leveraging the override attribute, you can seamlessly adapt endpoint behaviors within your tests, allowing you to create various testing scenarios and evaluate your application’s performance under different conditions.

Load Testing

Load testing allows you to simulate heavy user traffic on your application by transforming functional tests into load tests. This can be achieved by incorporating specific load profile keywords into your tests.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - startAt: 1
      scenarioName: scenario1
      atOnce: 2
      targetRPS: 3000
      duration: 10
      rampUp:
        duration: 3
        interval: 1

The atOnce attribute in the testPattern signifies the concurrency of the scenario1. With atOnce set to 2, everything defined in scenario1 will run with a concurrency of 2, happening in parallel.

To control the load, you can utilize the following parameters:

• targetRPS: Specifies the target Requests Per Second (RPS) your application should handle.
• duration: Indicates the total duration of the load test in seconds.
• rampUp: Allows you to gradually increase the load on an endpoint.
  • duration: Specifies how long it takes for the traffic ramp-up to occur.
  • interval: Signifies the rate at which traffic increases.

In the provided example, the load increases over 3 seconds, with increments every second until it reaches the target RPS of 3000. Modifying these values enables you to model traffic behavior for your application and test how it responds to varying levels of load. If targetRPS is not specified, Tester will attempt to send as many requests as possible within the system’s context.

Scenario Configuration

The scenario configuration file defines request behaviors for specific methods and creates scenarios as chains of defined requests and asserts.

Example Scenario Configuration:

version: v1
scenarios:
  - name: scenario1
    steps:
      - requestName: GetFeature_18GO
      - asserts: requests.GetFeature_18GO.res.name == "fake"
requests:
  - name: GGetFeature_18GO
    javascript: |-
      function handler(req) {
        // Your JavaScript logic here
        return {
          value: {
            latitude: x,
            longitude: y
          }
        };
      }      
  - name: ListFeatures_5P9V
    blob: |-
      {
        "hi": {
          "latitude": 0,
          "longitude": 0
        },
        "lo": {
          "      

latitude": 0,
          "longitude": 0
        }
      }
    endpointName: routeguide_jMBp
    methodName: ListFeatures
  # More request configurations...

In this example:

• scenarios: Allows you to define different test scenarios, and this example includes a scenario named scenario1.
• requests: Contains configurations for individual requests (to be referenced in scenarios by name).

Some advanced capabilities of the scenario configuration include:

• Scenarios and Asserts: Creating end-user use cases and validating responses.
• Dynamic requests: Customize request handling logic.
• Chaining and overrides: Chain values between sequential requests and overriding them as needed.

Scenarios and Asserts

Scenarios are representations of end-user use cases that require testing, allowing you to define a sequence of actions to be performed, typically involving requests to specific endpoints. Each named scenario includes a steps attribute, listing these actions, which can be executed sequentially or concurrently, simulating various usage patterns, including load tests.

Example Scenario Configuration:

version: v1
scenarios:
  - name: scenario1
    steps:
      - requestName: GetFeature_18GO
      - asserts: requests.GetFeature_18GO.res.name == "fake"

requests:
  - name: GetFeature_18GO
    endpointName: routeguide_jMBp
    methodName: GetFeature
    javascript: |-
      function handler(req) {
        // Your JavaScript logic here
        return {
          value: {
            latitude: x,
            longitude: y
          }
        };
      }      
  # More request configurations...

In this example:

• scenario1 is defined, containing two steps executed sequentially.
• The first step utilizes the requestName attribute, referencing a request object previously defined in the configuration.
• The second step is an assert statement, used to verify that the response from the request matches the expected value.

To use an assert, the asserts parameter is defined within the step, with a value in the format:

requests.<name of request>.res.message == "<expected value>"

Where <name of request> refers to the name of the previously defined request, and <expected value> is a string representing the expected return value from the request.

Note: Values returned by services are interpreted as JavaScript for evaluating assert statements. The type may not always be a string, but could be a boolean or a number, among other types. When working with a boolean, the <expected value> should be true or false and not "true" or "false" in the assert statement.

In scenarios, the associated steps are executed sequentially. To execute items in parallel, refer to the testPattern defined in the test configuration.

Dynamic Requests

Dynamic requests provide the flexibility to customize request handling logic in your scenario configurations. You can use different attributes to specify dynamic request behavior, such as python, pythonPath, javascript, or javascriptPath. Each attribute allows you to define custom request handling logic and return a JSON representation of the response value.

JavaScript Dynamic Requests

1. Using the javascript Attribute

   To create JavaScript-based dynamic requests, employ the javascript attribute within the requestValue section of your request definition. Define a function called handler that takes the req parameter. Implement your custom JavaScript logic within the handler function, and return a JSON object representing the response value.

   Example Scenario Configuration:

   version: v1
   requests:
     - name: GetFeature_18GO
       javascript: |-
         function handler(req) {
           // Your JavaScript logic here
           return {
             value: {
               latitude: x,
               longitude: y
             }
           };
         }      
       endpointName: routeguide_jMBp
       methodName: GetFeature
     # More request configurations...
   

2. Using the javascriptPath Attribute

   Alternatively, you can use the javascriptPath attribute to specify the path to an external JavaScript script file that contains your custom request handling logic.

   Example Scenario Configuration:

   version: v1
   requests:
     - name: GetFeature_18GO
       javascriptPath: scripts/getFeature.js
       endpointName: routeguide_jMBp
       methodName: GetFeature
     # More request configurations...
   

   The external JavaScript script file getFeature.js defines a handler function to process incoming requests and generate appropriate responses.

Installing NPM-Based Packages

If your JavaScript-based dynamic requests require NPM-based packages, follow these steps:

Specify the required packages in the npmPackages section of your test definition. The testing framework will automatically install these packages before running your test.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - requestName: GetFeature_18GO
      startAt: 1
    - requestName: ListFeatures_5P9V
      startAt: 1
    - requestName: RecordRoute_MD5D
      startAt: 1
    - requestName: RouteChat_QQEf
      startAt: 1
  npmPackages:
    - mathjs
    - chart

Python Dynamic Requests

Note: If your dynamic Python request is dependent on additional Python modules, refer to the Installing Worker with Python Modules section to learn how to build the custom Skyramp Worker image.

1. Using the python Attribute

   You can use the python attribute within the requestValue section of your request definition. This attribute allows you to define a function called handler that takes the req parameter, representing the incoming request. Within the handler function, you can implement your custom Python logic. Finally, return a JSON representation of the response value using SkyrampValue.

   Example Scenario Configuration:

   version: v1
   requests:
     - name: GetFeature_18GO
       python: |-
         def handler(req):
           // Your Python logic here
           return SkyrampValue(
             value={
               "latitude": x,
               "longitude": y
             }
           )      
       endpointName: routeguide_jMBp
       methodName: GetFeature
     # More request configurations...
   

2. Using the pythonPath Attribute

   Alternatively, you can use the pythonPath attribute to specify the path to an external Python script file containing your custom request handling logic.

   Example Scenario Configuration:

   version: v1
   requests:
     - name: GetFeature_18GO
       pythonPath: scripts/get_feature.py
       endpointName: routeguide_jMBp
       methodName: GetFeature
     # More request configurations...
   

   The external Python script file get_feature.py defines a handler function to process the request and generate the response.

Chaining and Overrides

Tester provides a powerful feature that allows you to chain values between sequential requests and override them.

Example Test Configuration:

version: v1
test:
  name: routeguide
  testPattern:
    - startAt: 1
      scenarioName: scenario1
  override:
    mock:
      - endpointName: routeguide_jMBp
        methodName: RouteChat
        javascript: |-
          function handler(req) {
            return {
              value: {
                message: req.value.message + "temp",
                location: {
                  latitude: req.value.latitude,
                  longitude: req.value.longitude
                }
              }
            }
          }          

Example Scenario Configuration:

version: v1
scenarios:
  - name: scenario1
    steps:
      - requestName: RouteChat_QQEf
      - asserts: requests.RouteChat_QQEf.res.message == "message1temp"
      - requestName: RouteChat_QQEf
        override:
          message: requests.RouteChat_QQEf.res.message
      - asserts: requests.RouteChat_QQEf.res.message == "message1temp1temp"
      - requestName: RouteChat_QQEf
        override:
          message: requests.RouteChat_QQEf.res.message
      - asserts: requests.RouteChat_QQEf.res.message == "message1temp1temp1temp"

requests:
  - name: RouteChat_QQEf
    endpointName: routeguide_jMBp
    methodName: RouteChat
    vars:
      message: "message"
    javascript: |
      i = 0
      function handler() {
        i++
        return {
          value: {
            message: vars.message + i,
            location: {
              latitude: req.value.latitude,
              longitude: req.value.longitude
            }
          }
        }
      }      
  # More request configurations...

In this example:

• The override attribute in the test section allows you to customize the behavior of an endpoint defined elsewhere in the project folder by specifying a mock.

• Within the scenarios section, multiple steps are defined. Each step calls the RouteChat_QQEf request and includes an assert. What sets them apart is that in subsequent requestName calls, the message variable is overridden. The message variable takes on the value of the response returned by the request. This chaining is done multiple times to create a sequence of messages.

• The requests attribute shows how the vars keyword is used to define a new variable called message in the RouteChat_QQEf request. This variable is utilized in the JavaScript snippet using vars.message. By overriding this variable in scenario1, you can modify the message content in the subsequent requests.

This feature allows you to create dynamic test scenarios where the output of one request influences the behavior of subsequent requests, making your testing more versatile and powerful.

Request Parameters and Headers

When making REST calls, requests often require headers, such as Basic Authentication information, and variables in the path. You can achieve this using the request object.

Example Scenario Configuration:

version: v1
requests:
  - name: addCartRequest
    endpointName: cart-service
    methodName: cart-service-post
    blob: |
      {
          "product_id": "OLJCESPC7Z",
          "quantity": 1
      }      
    headers:
      Authorization: "Basic YWxhZGRpbjpvcGVuc2VzYW1l"
    params:
      - name: user_id
        in: path
        value: abcde

In this example:

1. Path Parameters: In the cart-service endpoint, if the path contains a path parameter (/cart/user_id/{user_id}), we can define a params attribute and set user_id to abcde. Importantly, because we set in to path, it’s treated as a path parameter. You can also set in to query to make it a REST query parameter.

2. Headers: The headers attribute adds a header with the key Authorization and the value "Basic YWxhZGRpbjpvcGVuc2VzYW1l". It allows you to include headers in your requests, which is often required for authentication and other purposes.

Endpoint Configuration

The endpoint configuration file defines networking-level service details for an endpoint.

Example Endpoint Configuration:

version: v1
services:
  - name: routeguide
    port: 50051
    alias: routeguide
    protocol: grpc
endpoints:
  - name: routeguide_jMBp
    methods:
      - name: GetFeature
      - name: ListFeatures
      - name: RecordRoute
      - name: RouteChat
    defined:
      path: ./pb/route_guide.proto
      name: RouteGuide
    serviceName: routeguide

For configuring the endpoint file, we have a few key attributes:

• services: This section lists the services available in your project. In this example, there is one service named routeguide.

• endpoints: Under the endpoints section, you define individual endpoints, specifying the available methods, the service definition path, and the service name. In the example, we have an endpoint named routeguide_jMBp for the RouteGuide service.

• methods: Within each endpoint, you list the available methods. In this case, we have methods like GetFeature, ListFeatures, RecordRoute, and RouteChat. This helps specify the details of each method and how it should behave.

• defined: Here, you specify the service definition file’s path and the service name. The service definition file (route_guide.proto) outlines the structure of the service and its methods.

By configuring endpoints, you define the available services and methods within your project, facilitating the testing of your distributed application. We recommend dynamically generating a test by providing service-level information.