Code Duplication Detection via Abstract Syntax Trees
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large technical organizations face high maintenance costs and risks due to inflated codebases resulting from code duplication, which occupies space in cloud storage and requires significant developer time to identify and refactor duplicated code.
Innovation Solution
Systems and methods for detecting code duplication using abstract syntax trees, de-noising filters, dimension reduction techniques like Truncated Singular Value Decomposition (SVD) and Principal Component Analysis (PCA), and density-based clustering to identify near and exact duplicate code snippets, with the option to apply Natural Language Processing (NLP) for feature generation and a de-noising filter to remove irrelevant code.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If code duplication detection is performed using traditional methods, then exact duplicate detection is achieved, but near duplicate detection is insufficient
Solution Approach 1:
The patent changes the detection parameter from exact string matching to semantic similarity measurement using abstract syntax trees and machine learning models. This allows the system to detect near duplicates that have semantic equivalence but syntactic differences, resolving the contradiction between exact detection precision and near duplicate detection capability.
Solution Approach 2:
The patent replaces traditional mechanical string comparison methods with computational abstract syntax tree analysis and machine learning-based semantic similarity assessment. This substitution enables the system to understand code semantics rather than just comparing surface syntax, thereby detecting near duplicates that traditional methods miss.
2Reliability
If code duplication is detected to improve codebase quality, then maintenance cost reduces, but detection time and computational resources increase
Solution Approach 1:
The patent segments the codebase into individual code snippets or functions, processes each segment independently through abstract syntax tree construction and semantic similarity analysis, then aggregates the results. This segmentation allows parallel processing and reduces overall detection time while maintaining comprehensive codebase quality assessment.
Solution Approach 2:
The patent performs preliminary processing by converting code into abstract syntax trees and pre-computing semantic representations before actual duplication detection. This preliminary action optimizes the detection process and reduces real-time computational overhead, allowing faster identification of duplicates without sacrificing accuracy.
3Productivity
If comprehensive code duplication detection is implemented, then code reuse is promoted, but device complexity increases
Solution Approach 1:
The patent creates a universal detection system that handles multiple types of duplication (exact duplicates, near duplicates, semantic equivalents) using a unified approach based on abstract syntax trees and semantic similarity. This multi-functional system promotes code reuse across different scenarios without requiring separate complex detection mechanisms for each type.
Solution Approach 2:
The patent introduces abstract syntax trees as an intermediary representation between raw code and duplication detection algorithms. This intermediary layer simplifies the detection process by providing a standardized, normalized representation that can be efficiently compared using semantic similarity metrics, reducing the overall system complexity while maintaining comprehensive detection capability.
Data Source
AI summary
Systems and methods for detecting code duplication are disclosed. In one embodiment, a method for detecting exact code snippet duplicates may include: (1) representing, by a code duplication detection computer program, each of a plurality of code snippets in a codebase as an abstract syntax trees; (2) featurizing, by the code duplication detection computer program, the abstract syntax trees into corpus feature vectors by converting the abstract syntax tree into vector representations; (3) generating, by the code duplication detection computer program, dense feature vectors from the corpus feature vectors using a dimension reduction technique; (4) identifying, by the code duplication detection computer program, exact duplicate code snippet matches by apply density-based clustering to the dense feature vectors; and (5) tagging, by the code duplication detection computer program, the exact duplicate code snippets.