Compiler Feedback Regression Testing via Instrumentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional testing methodologies for compilers are limited in analysis and time-consuming, failing to identify unintended behavioral changes introduced during modifications, especially as compilers become increasingly complex, leading to potential issues with compiler integrity and functionality.

Innovation Solution

A feedback-driven regression testing method and system that compiles a code corpus using multiple versions of a compiler to generate and compare output, identifying behavioral regressions by analyzing internal decisions made during compilation, ensuring that each version behaves as intended and adhering to original specifications.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional testing methodologies are used for compilers, then the testing process is simple to implement, but the testing coverage and ability to identify behavioral regressions is insufficient

Engineering Contradiction:

Improvebehavioral regression detection accuracyVSAvoidtesting system complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The testing system is segmented into multiple independent components: a code corpus manager that handles test program selection, a compiler interface that manages compilation processes, and a regression analysis module that compares outputs. This segmentation allows each component to specialize in specific tasks, improving behavioral regression detection while keeping individual components manageable in complexity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary testing framework that sits between the compiler and the test programs. This framework captures compiler output, processes it through analysis rules, and generates regression reports. The intermediary layer abstracts the complexity of regression analysis, enabling accurate behavioral detection without requiring the entire system to become overly complex

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If conventional testing methods are used, then the testing process is quick to execute, but unintended behavioral changes are not identified

Engineering Contradiction:

Improvecompiler integrityVSAvoidtesting duration

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-compiling a comprehensive code corpus of test programs before actual regression testing begins. The code corpus is organized and prepared in advance with known expected behaviors. This preliminary preparation enables faster execution during actual testing while maintaining high reliability, as the test framework is already configured to detect behavioral changes

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The regression testing system establishes continuous testing workflows where compiler outputs are continuously analyzed against the code corpus. The system maintains continuous feedback loops that automatically detect behavioral regressions as they occur during compiler modifications, ensuring compiler integrity without requiring lengthy periodic testing cycles

Inventive Principle:

Principle #20Continuity of useful action

3Adaptability or versatility

If compilers are continuously modified to support expanding functionality, then the compiler capabilities are improved, but behavioral regressions are inadvertently introduced

Engineering Contradiction:

Improvecompiler functionalityVSAvoidcompiler behavioral consistency

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent implements feedback-driven regression testing where the system continuously monitors compiler behavior against the code corpus. When modifications are made to enhance compiler functionality, the feedback mechanism automatically compares new compiler outputs with expected behaviors from the code corpus, alerting developers to behavioral regressions. This feedback loop maintains reliability while allowing functional adaptations

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The code corpus serves as a universal test suite that can evaluate multiple compiler versions and functionality areas simultaneously. The same code corpus is used to test various compiler modifications, making the testing system versatile enough to handle expanding compiler capabilities while maintaining consistent behavioral standards across different functionality areas

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS8930765B2Systems and methods for feedback driven regression testing

Publication Date: 2015.01.06 ORACLE INT CORP

AI summary

Systems and methods for automatically testing one or more versions of a compiler of are disclosed. A compiler is instrumented to generated data exposing various internal decisions and/or actions made by the compiler. Subsequently, multiple distinct versions of the compiler are executed to compile a code corpus associated with a particular programming language. Output (including instrumentation output) from the compilation of the code corpus for each version of the compiler is obtained and compared to identify behavioral changes that may exist between the various versions of the compiler.