Iterative Hierarchical Data Validation with Temporary Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data import processes, particularly for hierarchical data, face inefficiencies due to memory limitations when validating dependencies, as all data is loaded into memory, restricting the size of datasets that can be analyzed.
Innovation Solution
A method that iteratively processes hierarchical data by first persisting independent records in a datastore and then iteratively validating dependent records in temporary memory, reducing the memory burden by only loading necessary data for validation in each iteration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data is loaded into memory for validation, then dependency validation can be performed, but memory limitations restrict the size of datasets that can be analyzed
Solution Approach 1:
The patent segments the data validation process into multiple passes, where in each pass only a subset of records is loaded into memory for validation. The dataset is processed in chunks, with each chunk validated against previously validated records, allowing large datasets to be handled without requiring all data to reside in memory simultaneously.
Solution Approach 2:
The patent performs preliminary validation of independent records (lowest level of hierarchy) and stores them in the datastore before processing dependent records. This preliminary action establishes a foundation that enables subsequent validation passes to proceed with smaller memory footprints, as previously validated records are already persisted.
2Reliability
If all data is loaded into temporary memory for processing, then complete validation can be performed, but memory constraints limit the scope of analytics operations
Solution Approach 1:
The patent divides the validation process into multiple passes, where each pass processes a subset of records. In the first pass, independent records are validated and persisted. Subsequent passes process dependent records in chunks, validating each against previously persisted records, thereby enabling complete validation of large datasets without requiring all data to be in memory simultaneously.
Solution Approach 2:
The patent performs preliminary validation and persistence of independent records before processing dependent records. This preliminary action reduces the memory burden for subsequent validation passes, as the foundation of validated records is already in place, enabling expanded analytics scope.
3Productivity
If data is processed in a single pass, then processing speed is high, but memory limitations prevent handling of large datasets
Solution Approach 1:
The patent segments the data processing into multiple passes, where each pass processes a subset of records. While this reduces the per-pass processing speed compared to a single-pass approach, it enables the system to handle much larger datasets by processing them in manageable chunks, with each pass validating records against previously persisted data.
Solution Approach 2:
The patent performs preliminary validation and persistence of independent records before processing dependent records. This preliminary action establishes a foundation that enables subsequent passes to proceed more efficiently, as validation can leverage already-persisted records rather than requiring all data to be re-loaded into memory.
Data Source
AI summary
Techniques are provided for persisting validated hierarchical data in a datastore. In one example, a system can identify, in a dataset having a dependency hierarchy, records that are independent from any other records in the dataset and store the records in a datastore. The system can identify records in the dataset that are dependent on other records in the dataset and store the records in a temporary memory. The system can process the records using an iterative hierarchical processing scheme, in which each iteration of the scheme involves identifying a subset of records corresponding to a particular level of the dependency hierarchy. For each record in the subset of records, the system can obtain previously processed records on which the record depends, process each record using the previously processed records, store each processed record in the datastore, and remove the record from the temporary memory.