Structured Data Pattern Marking via Block Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large volumes of structured data, such as program code listings or CAN bus log files, is computationally intensive and requires significant hardware resources, making it impractical to search and display individual data units efficiently.
Innovation Solution
The method involves subdividing the data into processing blocks, generating partial result files for each block, and combining them into a result file, which is smaller and more manageable, allowing for efficient processing and visualization using standard PCs, even for data sizes exceeding storage or processing device limits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire structured data set is loaded and processed in memory, then complete search coverage is achieved, but hardware resource requirements increase significantly
Solution Approach 1:
The patent divides the large structured data set into multiple smaller blocks that can be processed sequentially in memory. Each block is loaded, searched for patterns, and the results are stored in a result file. This segmentation allows the system to maintain search completeness while keeping memory usage within available hardware limits.
2Quantity of substance
If the data set is divided into smaller processing blocks, then memory requirements are reduced, but processing time increases due to multiple passes
Solution Approach 1:
The patent performs preliminary actions by creating and storing a result file during the first pass that contains all necessary information about matched patterns and their locations. This result file is then used in subsequent operations to display and analyze matched data units without requiring repeated access to the original large data set, thereby reducing overall processing time.
3Loss of information
If the result file contains the entire data set, then all information is available for analysis, but the result file size becomes unmanageably large
Solution Approach 1:
The patent extracts only the essential information needed for analysis from the processed data blocks and stores it in a compact result file. Instead of copying the entire data set, the system stores references to matched data units, their locations, and relevant metadata. This extraction approach maintains information availability for analysis while keeping the result file size manageable.
4Ease of manufacture
If standard PCs are used for processing, then hardware cost is reduced, but processing capability is limited
Solution Approach 1:
The patent segments the large data set into smaller blocks that fit within the memory capacity of standard PCs. This allows inexpensive hardware to process terabyte-scale data sets by working with manageable chunks sequentially, eliminating the need for expensive high-memory systems while maintaining processing capability.
Data Source
Figure 1~2
AI summary
In a method for marking predefined patterns in a structured dataset stored in a digital data file (2) on a storage medium, a result file (9) is generated from the structured dataset. This result file contains the data units matching a search pattern criterion, along with data unit reference information. A pattern satisfying the search pattern criterion is then marked from the result file (9). The data file (2) is divided into several processing blocks (4), and a sub-result file (7) is generated for each processing block (4). The sub-result files (7) are then combined to form the result file (9). The search pattern criterion has multiple search pattern components and is satisfied if a data unit contains at least one search pattern component.It is also possible that the search pattern criterion contains at least one exclusion pattern component and is not met if a data unit contains at least one exclusion pattern component. The search pattern criterion can be met if a comparison of data unit information with the search pattern information yields a predefined information comparison value.