Orchestration System for Automated Performance Testing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In cloud platforms, the disconnect between application development teams and performance testing teams leads to inefficient and inaccurate performance testing, often resulting in significant time expenditure and errors due to manual setup of test environments and lack of domain knowledge.

Innovation Solution

An orchestration system that automates performance testing by determining and creating test environments, deploying applications, and executing performance tests across multiple environments to identify bottlenecks and improve system performance, reducing the need for manual intervention and domain expertise.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If manual setup of test environments is performed, then flexibility in test configuration is improved, but time consumption and error rate increase

Engineering Contradiction:

Improvetest configuration flexibilityVSAvoidtest environment setup time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by automatically provisioning test environments with required configurations, applications, and dependencies before performance testing begins. This eliminates manual setup time while maintaining configuration flexibility through automated parameter specification.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system enables self-service by allowing performance testing to be executed automatically without manual intervention in environment setup. The automated system provisions resources, configures applications, and manages test execution independently, reducing both time consumption and human error.

Inventive Principle:

Principle #25Self-service

2Reliability

If performance testing is performed by a separate team without domain knowledge, then independence in testing is improved, but testing accuracy decreases

Engineering Contradiction:

Improvetesting independenceVSAvoidperformance testing accuracy

Core Design Contradiction:

ReliabilityVSMeasurement precision

Solution Approach 1:

The system introduces an intermediary layer of automated orchestration that bridges the gap between independent performance testing and application-specific domain knowledge. The automation framework acts as a mediator, encoding domain knowledge into reusable templates and scripts that guide the testing process without requiring human experts.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system manages parameter changes by automatically adjusting test configurations based on application-specific requirements encoded in the orchestration framework. This allows accurate performance testing without manual intervention, as the system dynamically modifies test parameters according to pre-defined application characteristics.

Inventive Principle:

Principle #35Parameter changes

3Adaptability or versatility

If multiple test environments are created manually, then comprehensive performance coverage is improved, but complexity of test setup increases

Engineering Contradiction:

Improveperformance test coverageVSAvoidtest environment complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system applies segmentation by dividing the complex task of creating multiple test environments into standardized, reusable templates. Each environment type is defined as a separate configuration module, allowing comprehensive coverage through composition of modular components rather than manual creation of each environment.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The orchestration framework provides universality by creating a multi-functional system that can automatically provision various types of test environments using a single automated process. This unified approach handles diverse environment requirements through parameterized configurations, reducing overall system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11762763B2Orchestration for automated performance testing

Publication Date: 2023.09.19 SALESFORCE INC

AI summary

Methods, systems, and devices supporting orchestration for automated performance testing are described. A server may orchestrate performance testing for software applications across multiple different test environments. The server may receive a performance test indicating an application to test and a set of test parameters. The server may determine a local or a non-local test environment for running the performance test. The server may deploy the application to the test environment, where the deploying involves deploying a first component of the performance test to a first test artifact in the test environment and deploying a second component of the performance test different from the first component to a second test artifact in the test environment. The server may execute the performance test to obtain a result set, where the executing involves executing multiple performance test components as well as orchestrating results across multiple test artifacts to obtain the result set.