Source Code Scanner Evaluation via Snippet Insertion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing source code scanners face challenges in accurately identifying security defects, leading to high false positive and false negative rates, which can be resource-intensive and credibility-damaging, and organizations struggle to select the most suitable scanner due to unreliable marketing information and varying performance across different projects.
Innovation Solution
A system and method for evaluating source code scanners by inserting good and bad code snippets into source code, generating an issue list, and comparing it to determine false positive and false negative rates, thereby producing metrics that help stakeholders select scanners based on quality and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If source code scanners are used to identify security defects, then security audit capability is improved, but false positive and false negative rates increase
Solution Approach 1:
The patent applies preliminary action by inserting known good and bad code snippets into the source code before scanning. This creates a controlled test environment where the expected outcomes are predetermined, allowing accurate measurement of the scanner's performance metrics including false positive and false negative rates.
Solution Approach 2:
The patent uses good and bad code snippets as intermediary elements between the source code scanner and the evaluation process. These snippets serve as reference standards that mediate the comparison between scanner output and actual security defect status, enabling precise measurement of scanner accuracy.
2Adaptability or versatility
If multiple source code scanners are evaluated, then scanner selection capability is improved, but evaluation complexity increases
Solution Approach 1:
The patent segments the evaluation process into distinct, manageable components: inserting good snippets, inserting bad snippets, scanning the modified code, comparing results, and generating metrics. This segmentation simplifies the overall evaluation complexity while maintaining comprehensive scanner assessment capability.
Solution Approach 2:
The patent uses copies of known good and bad code snippets as test inputs. These copies serve as standardized test cases that can be reused across multiple scanner evaluations, simplifying the evaluation process while enabling consistent comparison across different scanners.
3Measurement precision
If code snippets are inserted into source code for evaluation, then measurement accuracy is improved, but source code modification complexity increases
Solution Approach 1:
The system performs self-service by automatically inserting the good and bad code snippets into the source code without requiring manual intervention. This automation eliminates the complexity of manual code modification while maintaining measurement accuracy, as the insertion process is systematic and reproducible.
Data Source
AI summary
Apparatuses, methods, and non-transitory computer readable medium that evaluate a source code scanner are described. In one implementation, the method comprises obtaining source code. One or more good code snippets and one or more bad code snippets are inserted into the source code to obtain a modified source code. An issue list generated by the source code scanner upon scanning the modified source code is obtained. The issue list comprises code segments having security defects identified by the source code scanner, reasons for the security defects, and locations of the security defects in the modified source code. The code segments present in the issue list are compared with the one or more good code snippets and the one or more bad code snippets. A plurality of metrics, indicating quality of the source code scanner, are generated based on the comparison.