Data Generalization Apparatus for High-Speed Large-Scale Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques face challenges in performing generalization processing on large-scale databases efficiently, as they either require a primary storage device with sufficient capacity or rely on inefficient SQL operations when using a secondary storage device.
Innovation Solution
A data generalization apparatus that rearranges attribute values in a secondary storage device according to a generalization hierarchy, retrieves relevant values and hierarchy portions to a primary storage device for processing, and re-rearranges them back, utilizing lookahead processing and the LRU algorithm to minimize secondary storage access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If generalization processing is performed by placing the target database and generalization hierarchy in a primary storage device, then processing speed is high, but the capacity of the primary storage device becomes insufficient for large-scale databases
Solution Approach 1:
The patent divides the database and generalization hierarchy into segments that are loaded into the primary storage device only when needed. The rearrangement unit segments the database according to the generalization hierarchy structure, and the attribute value retrieval unit selectively loads only the necessary segments into primary storage, rather than loading the entire database at once.
Solution Approach 2:
The patent performs preliminary rearrangement of the database in the secondary storage device according to the generalization hierarchy order before processing. This preliminary organization allows the attribute value retrieval unit to efficiently predict and pre-load the next required attribute values and hierarchy portions into primary storage, minimizing access delays.
2Quantity of substance
If generalization processing is performed by placing the database in a secondary storage device using SQL operations, then storage capacity is sufficient, but processing speed is slow due to non-dedicated operations
Solution Approach 1:
The patent introduces a specialized data generalization apparatus as an intermediary layer between the secondary storage device and the processing unit. This apparatus includes a rearrangement unit, attribute value retrieval unit, generalization hierarchy retrieval unit, and generalization processing unit that work together to efficiently manage data flow and processing, avoiding the inefficiencies of general-purpose SQL operations.
Solution Approach 2:
The patent changes the operational parameters by using dedicated data structures and access patterns optimized for generalization processing. Instead of using standard SQL operations, the system employs custom retrieval algorithms that predict and pre-load data, changing how data is accessed and processed to achieve higher speeds while maintaining sufficient storage capacity.
3Quantity of substance
If frequent access to secondary storage is performed to retrieve attribute values and hierarchy portions, then processing can be done with small primary storage, but processing speed decreases
Solution Approach 1:
The patent performs preliminary rearrangement of the database in secondary storage according to the generalization hierarchy order. The attribute value retrieval unit uses this pre-organized structure to predict which attribute values and hierarchy portions will be needed next, allowing it to pre-load them into primary storage before they are actually required for processing.
Solution Approach 2:
The system implements a feedback mechanism where the attribute value retrieval unit monitors the processing progress and uses the generalization hierarchy structure to predict future data needs. This feedback loop allows the system to dynamically adjust what data to pre-load into primary storage, ensuring that the most needed data is available when required.
Data Source
AI summary
A data generalization apparatus that can perform generalization processing on large-scale data at high speed using only a primary storage device of a small capacity. Included is a rearrangement unit that rearranges the attribute values in a secondary storage device in accordance with an order of arrangement of the attribute values in a generalization hierarchy in the secondary storage device, an attribute value retrieval unit that retrieves some of the rearranged attribute values from the secondary storage device into a primary storage device, and a generalization hierarchy retrieval unit that retrieves a portion of the generalization hierarchy from the secondary storage device into the primary storage device. Further, there is a generalization processing unit that executes generalization processing based on the attribute values retrieved into the primary storage device and the generalization hierarchy retrieved into the primary storage device, and a re-rearrangement unit.