Firmware Test Case Selection via Binary Error Injection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Automated regression testing of firmware faces challenges in efficiently selecting test cases that cover code changes, often resulting in imprecise manual selection and resource-intensive coverage-driven approaches, particularly in resource-scarce environments.

Innovation Solution

The method involves identifying impacted functions by determining differences between firmware versions, modifying binary code to return an error value for these functions, and executing test cases to select target test cases that fail with this error, thereby focusing regression testing on critical areas while excluding previously associated test cases.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If code coverage tools are used to select test cases, then test case selection precision is improved, but CPU, memory, and storage resources are consumed

Engineering Contradiction:

Improvetest case selection precisionVSAvoidCPU, memory, and storage resources

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent extracts only the essential information needed for test case selection by analyzing firmware differences directly, rather than using comprehensive code coverage tools. The system identifies impacted functions by comparing firmware versions and selects test cases based on these differences, eliminating the need for resource-intensive coverage analysis while maintaining selection precision.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent uses lightweight, temporary data structures to represent firmware differences and impacted functions during the selection process. Instead of relying on heavy code coverage tools, the system creates minimal representations of firmware state changes that can be processed efficiently with limited resources, then discards them after use.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

2Use of energy by moving object

If manual test case selection is used, then resource consumption is reduced, but selection precision deteriorates

Engineering Contradiction:

Improveresource consumptionVSAvoidselection precision

Core Design Contradiction:

Use of energy by moving objectVSMeasurement precision

Solution Approach 1:

The system performs self-service by automatically analyzing firmware differences and identifying impacted functions without human intervention. The automated process compares firmware versions, determines which functions are affected, and selects appropriate test cases, eliminating the need for manual selection while consuming minimal resources.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical test case selection with an automated information-processing system. Instead of human analysts reviewing and selecting test cases, the system uses algorithmic comparison of firmware versions to automatically identify impacted functions and select test cases, improving precision while reducing resource consumption.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If all test cases are executed to ensure comprehensive coverage, then test reliability is improved, but test time increases

Engineering Contradiction:

Improvetest reliabilityVSAvoidtest time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the complete test suite into two categories: test cases that exercise impacted functions (selected for execution) and test cases that do not (excluded). By dividing the test set based on firmware differences, the system executes only the necessary portion of tests, maintaining reliability for affected areas while reducing overall test time.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial action by executing only the subset of test cases that are necessary to verify impacted functions, rather than running the complete test suite. This selective execution approach provides sufficient coverage for reliability while significantly reducing test time compared to exhaustive testing.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20240241815A1Method for efficiently improving test case coverage and robustness

Publication Date: 2024.07.18 DELL PROD LP

AI summary

Disclosed methods for selecting test cases determine differences between first and second versions of firmware. The first version may correspond to a version currently in use while the second version represent an update to the first version, including one or more bug fixes and/or additional features. Based at least in part on the differences between the two versions, one or more functions most likely to be impacted by the update are identified. Disclosed methods and system may then modify binary code for the second firmware version to cause the one or more impacted functions, when called, to return unconditionally a predetermined error value. With the firmware thus modified, a plurality of test cases are executed one time. Any test cases failing with the appropriate error value are identified as selected test cases and included within a group of target test cases for performing regression testing of the second version.