Hierarchical Value List Dataset Insertion Index Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Loading large datasets characterized by hierarchical value lists into database systems is computationally costly and time-consuming due to the resource-intensive process of transforming and associating values, especially when dealing with large tree structures, as conventional methods require constant traversal of the tree structure for each record.
Innovation Solution
The technique involves converting the dataset into in-memory index arrays corresponding to each level of the hierarchical value list, allowing for the insertion of entire levels or generations at once, rather than processing records individually, thereby reducing the number of steps and computational resources required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional methods are used to load large datasets into database systems, then the data can be inserted into the database, but the process becomes computationally costly and time-consuming due to constant traversal of the tree structure for each record
Solution Approach 1:
The patent segments the hierarchical value list into multiple levels or generations. Instead of processing the entire hierarchy for each record, the system divides the data into discrete levels and processes them separately using level arrays, significantly reducing the computational complexity from O(n*m) to O(n+k) where n is the number of records, m is the hierarchy depth, and k is the number of unique values across all levels.
Solution Approach 2:
The patent performs preliminary action by pre-processing the dataset to identify and extract unique values at each level of the hierarchy before actual database insertion. This allows the system to build level arrays in advance, so that during the loading phase, only simple lookups and insertions are needed without repeated tree traversals.
2Ease of operation
If the entire tree structure is maintained in memory for processing, then record insertion can be performed, but the memory requirements and computational overhead increase significantly
Solution Approach 1:
The patent extracts only the necessary information from the complete tree structure - specifically, the unique values at each level are extracted and stored in level arrays. This extraction eliminates the need to maintain the entire hierarchical tree structure in memory, reducing memory usage from O(total nodes) to O(unique values per level) while preserving all necessary functionality for efficient data loading.
3Reliability
If records are processed individually with full tree traversal, then data integrity is maintained, but the time required for dataset ingestion increases
Solution Approach 1:
The patent merges multiple individual record processing operations into batch operations at each level. By combining all values at a given level into a single level array and processing them together, the system maintains data integrity through systematic handling while reducing the total number of database operations from O(n*m) to O(k) where k is the total unique values across all levels.
Data Source
AI summary
A method includes obtaining a dataset comprising a plurality of records to be inserted into a database, and converting the dataset into a hierarchical value list, the hierarchical value list comprising a hierarchy with one or more levels. The method also includes generating a plurality of record arrays for the plurality of records to be inserted into the database, a given record array comprising a set of values for a given record at one or more index positions each corresponding to one of the one or more levels of the hierarchy. The method further includes building an index array comprising values for a given one of the index positions of the plurality of record arrays corresponding to a given one of the one or more levels of the hierarchy, and inserting the index array comprising the values for the given level of the hierarchy into the database.