Inline Tree Data Structure for High-Speed Dataset Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional electronic data structures such as spreadsheets and databases become inefficient and slow when handling large datasets, making high-speed search and retrieval impractical due to increased size and overhead, especially when dealing with millions of records.
Innovation Solution
The implementation of an inline tree data structure that uses binary string indices, data clumping, and field masks to reduce storage size and improve search efficiency, allowing the entire dataset to be loaded into RAM for rapid processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional electronic data structures (spreadsheets and databases) are used to store large datasets, then data storage capability is improved, but search and retrieval speed deteriorates significantly
Solution Approach 1:
The patent segments the dataset into hierarchical groups and subsets, organizing data in a tree structure where the root represents the entire dataset, intermediate nodes represent groups, and leaf nodes represent individual records. This segmentation enables the search algorithm to process only relevant portions of the data at each level, dramatically improving search speed while maintaining the ability to store large quantities of data.
2Adaptability or versatility
If conventional electronic data structures are used, then flexibility in data arrangement is improved, but search efficiency deteriorates for very large datasets
Solution Approach 1:
The patent implements a dynamic search algorithm that adapts to the hierarchical structure of the data. The algorithm dynamically determines which branches of the tree to explore based on the search criteria, skipping irrelevant groups and subsets. This dynamic approach maintains the flexibility of data arrangement while achieving high search efficiency by processing only necessary data portions.
3Ease of operation
If datasets are stored in conventional formats, then ease of access and updating is improved, but search time increases exponentially with dataset size
Solution Approach 1:
The patent performs preliminary organization of data into a hierarchical tree structure before searching begins. Data is pre-grouped into meaningful categories and subsets, with metadata prepared in advance. This preliminary action enables the search algorithm to quickly navigate the structured data without performing extensive processing during the actual search, significantly reducing search time while maintaining ease of data access and updating.
Data Source
AI summary
A binary data file embodies an inline tree data structure storing fields of a hierarchical dataset. The inline tree comprises first-level binary string segments, each comprising substantially contiguous second-level binary string segments, corresponding to subranges of first and second subsets of data fields. Size is reduced by substituting: binary string indices for alphanumeric strings; a data clump index for a set of correlated/anticorrelated strings; field masks for unoccupied data fields. A dedicated conversion program generates the inline tree from conventional database formats, which is read entirely into RAM to be searched/filtered by a dedicated search/filter program. Small size (<2 bytes/field/record) and contiguous arrangement enables searching/filtering of >106 records (>100 data fields) in <500 nanoseconds/record/core. Recursive subdivision of selection field ranges can guide searches that include those selection fields. One example includes geographic searching/filtering of records that include latitude and longitude fields.