Remote Debugging for Flaky Test Failures

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Flaky tests in autonomous vehicle software development are challenging to debug due to non-deterministic results, leading to inefficiencies in resource utilization on cluster infrastructure and increased developer time spent on debugging.

Innovation Solution

A remote debugging session system is implemented on cluster infrastructure, allowing developers to run flaky tests in parallel, pause failed executions, and access a debugging environment equipped with tools to inspect the runtime environment and source code, thereby identifying the cause of flakiness.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If flaky tests are run multiple times to reproduce failures, then the ability to identify the cause of flakiness improves, but the time and resources consumed increase

Engineering Contradiction:

Improveability to identify cause of flakinessVSAvoidtime spent on debugging

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary actions by automatically re-running flaky tests multiple times in parallel before the developer intervenes. The test infrastructure proactively identifies flaky tests, schedules repeated executions, and captures failure states, thereby preparing diagnostic information in advance and reducing the time developers would otherwise spend manually reproducing failures.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates copies of the test execution environment by running tests in parallel across multiple worker nodes. Each parallel execution is an independent copy that can fail independently, allowing the system to gather multiple failure instances without requiring the developer to manually reproduce the issue repeatedly.

Inventive Principle:

Principle #26Copying

2Productivity

If multiple test executions are run in parallel to improve debugging efficiency, then the productivity improves, but the resource consumption on cluster infrastructure increases

Engineering Contradiction:

Improvedebugging efficiencyVSAvoidresource consumption

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The system applies partial action by limiting the number of parallel test executions to a configurable threshold rather than running all possible tests simultaneously. This allows sufficient parallelism to improve debugging efficiency while preventing excessive resource consumption that would occur if all tests were executed in parallel without limits.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system implements feedback mechanisms where the test infrastructure monitors test results and automatically adjusts execution strategies. When flaky tests are detected, the system feeds back this information to schedule targeted re-executions only for affected tests, optimizing resource usage by focusing parallel executions only where needed rather than running all tests uniformly.

Inventive Principle:

Principle #23Feedback

3Difficulty of detecting and measuring

If developers manually reproduce flaky test failures, then the ability to diagnose the issue improves, but the developer time and cluster resources increase

Engineering Contradiction:

Improvediagnosis capabilityVSAvoiddeveloper time

Core Design Contradiction:

Difficulty of detecting and measuringVSLoss of time

Solution Approach 1:

The test infrastructure provides self-service by automatically detecting flaky tests, scheduling re-executions, capturing failure states, and presenting diagnostic information without requiring developer intervention. The system serves itself by monitoring test results, identifying patterns, and generating debugging information autonomously, thereby eliminating the need for developers to manually reproduce failures.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system introduces an intermediary layer between the developer and the test execution process. This intermediary is the automated test infrastructure that handles the complex tasks of scheduling, monitoring, and analyzing test failures, thereby mediating between the developer's diagnostic needs and the actual test execution, freeing developers from manual reproduction tasks.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12271290B2Remote debugging sessions for flaky tests

Publication Date: 2025.04.08 GM CRUISE HOLDINGS LLC

AI summary

Build, execution, and testing on software may be performed remotely on cluster infrastructure. Tests can be scheduled on workers in the cluster infrastructure, and results of the tests are reported to the developer. If a test fails, the developer may receive an exit code from the failed test. For tests that are flaky, an exit code provides little to no benefit to identify the cause of the flakiness. To make it easier for the developer to debug flaky tests, a test identified to be flaky can be run many times in parallel, one or more parallel executions that result in failure can be paused, and one or more remote debugging sessions can be created for the one or more parallel executions. A developer can access the remote debugging session to inspect the paused execution of the test equipped with debugging tools to determine the cause of the flakiness.