Software Debugging Benchmark Curation for Reproducible Failure Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software debugging technologies face challenges in collecting and maintaining realistic benchmark data for evaluating automatic debugging tools, particularly for identifying and fixing software failures, due to the complexity and variability of software failures, leading to potential data leakage issues that distort performance evaluations.
Innovation Solution
A software debugging benchmark system that monitors pull requests, applies predetermined criteria to identify potential failure components, verifies failure fixes in a virtual environment, and stores verified failure data in a database, while using a Docker container or virtual machine to ensure reproducibility and prevent data leakage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If software debugging benchmark data is collected from real software failures, then the realism and evaluative accuracy of the benchmark is improved, but the difficulty and effort required to collect and maintain the data increases significantly
Solution Approach 1:
The system automatically collects, verifies, and maintains benchmark data through self-service mechanisms. Automated scripts monitor repositories, execute verification processes, and update the benchmark database without requiring manual intervention, thereby maintaining high evaluative accuracy while reducing operational complexity
Solution Approach 2:
The system performs preliminary verification of potential benchmark data before inclusion. By pre-validating failure cases through automated testing and criteria checking before they are added to the benchmark, the system ensures data quality while streamlining the collection process
2Measurement precision
If benchmark data is collected from real software failures, then the realism of the benchmark is improved, but data leakage issues may occur that distort performance evaluations
Solution Approach 1:
The system introduces an intermediary verification layer between data collection and benchmark inclusion. This intermediary process validates that failures are genuine software defects rather than test artifacts or environmental issues, preventing data leakage while maintaining benchmark realism
Solution Approach 2:
The system implements feedback mechanisms where verification results from automated testing inform the inclusion decisions for benchmark data. This feedback loop ensures that only verified, non-leaking data is incorporated, maintaining both realism and evaluation reliability
3Measurement precision
If manual methods are used to collect and verify failure data, then the accuracy of failure verification is improved, but the time and resources required increase significantly
Solution Approach 1:
The system replaces manual mechanical verification processes with automated computational systems. Scripts automatically execute test cases, analyze failure logs, and verify benchmark data accuracy, maintaining verification precision while dramatically reducing the time and human resources required
Solution Approach 2:
The system enables continuous automated verification of benchmark data through scheduled execution of test suites and monitoring processes. This continuous action maintains high verification accuracy while operating efficiently over time, eliminating the need for repeated manual verification cycles
Data Source
AI summary
An operating method of a software debugging benchmark system includes: monitoring pull requests (PRs) created in a project and identifying, in a PR identified by the monitoring, a commit that adds a source code change and test code; determining whether the commit is a potential failure benchmark component by applying predetermined criteria to the commit; based on determining that the commit is a potential failure benchmark component, verifying, in a virtual environment, whether the commit has fixed an actual failure; and based on the verifying, storing a failure PR corresponding to the commit in a database as a failure detection test and failure fix commit.