Delimiter Identification System Using Levenshtein Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems lack an efficient method to automatically identify delimiters in computer files, as different files use varying delimiters and data may misuse delimiter characters, making it difficult for programs to parse data correctly.
Innovation Solution
A system and method that copies a file portion, applies various delimiter schemas to create tokenized versions, identifies the most commonly repeating patterns, and uses Levenshtein distances to determine the best delimiter set, allowing for user selection or automatic parsing based on scoring and threshold criteria.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple delimiter schemas are applied to identify delimiters automatically, then the ability to handle varying delimiter usage is improved, but the computational complexity and processing time increase
Solution Approach 1:
The patent segments the delimiter identification process into multiple independent delimiter schemas, each targeting specific delimiter types (e.g., CSV delimiters, TSV delimiters, fixed-width delimiters). This segmentation allows the system to apply only relevant schemas to specific file types, reducing overall computational complexity while maintaining versatility.
Solution Approach 2:
The patent performs preliminary actions by pre-defining multiple delimiter schemas with common delimiter patterns before actual file processing. These schemas are prepared in advance with predefined delimiter sets and scoring criteria, enabling rapid matching during file analysis without requiring complex real-time computation.
2Adaptability or versatility
If multiple delimiter schemas are applied to identify delimiters automatically, then the ability to handle varying delimiter usage is improved, but the processing time increases
Solution Approach 1:
The patent implements partial action by applying only a subset of delimiter schemas based on file characteristics. The system analyzes file samples and applies scoring mechanisms to determine which schemas are most relevant, avoiding the computation of all possible schemas and thereby reducing processing time while maintaining high adaptability.
Solution Approach 2:
The patent uses feedback mechanisms through scoring systems that evaluate how well each delimiter schema matches the file content. Based on these scores, the system provides feedback to select the most appropriate schema, enabling adaptive processing that reduces time by avoiding unnecessary schema applications while maintaining versatility.
3Measurement precision
If data descriptions with preference are used to recognize delimiters as data instead of actual delimiters, then the accuracy of data recognition is improved, but the complexity of the delimiter schema increases
Solution Approach 1:
The patent applies local quality by implementing data descriptions with preference at specific locations within the delimiter schema where delimiter ambiguity is most likely to occur. Rather than making the entire schema complex, preference rules are applied locally to specific delimiter patterns that commonly appear within data fields, improving accuracy without overwhelming complexity.
4Ease of operation
If the system provides user selection option along with automatic parsing, then the ease of operation is improved, but the device complexity increases
Solution Approach 1:
The patent implements self-service by providing automatic delimiter identification that works without user intervention. The system autonomously analyzes files, applies scoring mechanisms, and selects the best delimiter schema automatically. The user interface simply presents options when needed, reducing operational complexity while maintaining ease of use through automated decision-making.
Data Source
AI summary
A system and method automatically identifies any or all of potential row, column and string delimiters in a file in which such delimiters are unknown to the program making such identification.