Distributed Canary Testing With Artifact Caching

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In large-scale computing systems, automated canary testing faces inefficiencies due to the need for repeated acquisition of test artifacts, which consumes network bandwidth and delays test scheduling, especially when using serverless compute services where state is discarded between tests.

Innovation Solution

Implementing test artifact caching at clients, allowing reuse of cached artifacts between tests, reducing the need for reacquisition and improving scheduling precision, and using selective API testing and cross-region configurations to enhance reliability and performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If test artifacts are reacquired for each canary test, then test reliability is maintained, but network bandwidth consumption increases and test scheduling is delayed

Engineering Contradiction:

Improvetest reliabilityVSAvoidtest scheduling delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Test artifacts are acquired and cached in advance before canary tests are executed. The caching mechanism stores artifacts locally at test clients, allowing subsequent tests to reuse cached artifacts without reacquiring them, thereby eliminating scheduling delays while maintaining reliability through artifact validation

Inventive Principle:

Principle #10Preliminary action

2Productivity

If test artifacts are cached locally at clients, then test scheduling precision improves and network usage reduces, but system complexity increases

Engineering Contradiction:

Improvetest scheduling precisionVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

A caching layer is introduced as an intermediary between the artifact repository and test execution environment. This caching mechanism manages artifact storage, validation, and retrieval locally at test clients, improving scheduling precision and reducing network usage while the added complexity is managed through automated cache management and validation protocols

Inventive Principle:

Principle #24Intermediary (Mediator)

3Loss of energy

If test artifacts are cached, then network bandwidth consumption is reduced, but storage requirements at clients increase

Engineering Contradiction:

Improvenetwork bandwidth consumptionVSAvoidclient storage requirements

Core Design Contradiction:

Loss of energyVSVolume of stationary object

Solution Approach 1:

The caching implementation adopts a local quality approach where each test client maintains a localized cache of test artifacts appropriate to its specific testing needs. This allows selective caching of only necessary artifacts at each client location, reducing overall network bandwidth consumption while managing storage requirements through localized, targeted artifact retention rather than universal caching

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS11500763B1Distributed canary testing with test artifact caching

Publication Date: 2022.11.15 AMAZON TECH INC

AI summary

Methods, systems, and computer-readable media for distributed canary testing with test artifact caching are disclosed. Using one or more storage components, a test client stores one or more software artifacts for testing of a software product. The client initiates a first test of the software product using the software artifact(s) stored in the storage component(s). In the first test, the client sends a first set of requests to the software product at a first point in time. The client initiates a second test of the software product using the software artifact(s) stored in the storage component(s). In the second test, the client sends a second set of requests to the software product at a second point in time. The software artifact(s) are maintained in the storage component(s) between the first point in time and the second point in time.