Parallel Block-Attribute Indexing for Dynamic Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional indexing techniques for block-based data stores are inefficient for dynamic datasets, as they prioritize one-time indexing over incremental updates, which is inadequate for systems that continuously receive and process large volumes of new data.
Innovation Solution
A multicore processor-based system that generates block-attribute pairs and performs parallel indexing using multiple indexing instances, allowing for efficient incremental indexing of dynamic datasets by distributing block-attribute pairs across multiple queues and processing them in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional one-time indexing is performed upon importation of a complete dataset, then query speed for complete static sets of data blocks is maximized, but indexing speed and efficiency deteriorate and incremental indexing of dynamic datasets is not enabled
Solution Approach 1:
The patent divides the indexing process into segments by creating multiple indexing instances that operate in parallel on different portions of the dataset. Each indexing instance processes a subset of block-attribute pairs independently, allowing the overall indexing task to be completed faster through concurrent execution across multiple processing cores.
Solution Approach 2:
The patent transitions from single-threaded sequential indexing to multi-threaded parallel indexing by utilizing multiple processing cores. This dimensional change from one-dimensional sequential processing to multi-dimensional parallel processing enables simultaneous execution of multiple indexing operations, dramatically improving indexing throughput and efficiency.
2Measurement precision
If conventional indexing prioritizes one-time indexing over incremental updates, then complete static dataset query performance is improved, but the ability to incrementally index dynamic datasets deteriorates
Solution Approach 1:
The patent implements dynamic indexing by enabling the indexing system to adapt to changing data conditions. Multiple indexing instances can be dynamically created, started, paused, and terminated based on data arrival rates and system load. This dynamic approach allows the system to perform incremental indexing on dynamic datasets while maintaining query performance on both static and dynamic portions of the data.
Solution Approach 2:
The patent ensures continuous indexing operation by maintaining multiple indexing instances that can process data blocks as they arrive. Rather than performing indexing as a single batch operation, the system continuously indexes incoming data blocks, ensuring that the index remains current with the data without requiring complete re-indexing operations.
Data Source
AI summary
Techniques for block-based indexing are described. In one embodiment, for example, an apparatus may comprise a multicore processor element, an assignment component for execution by the multicore processor element to generate a plurality of block-attribute pairs, each block-attribute pair corresponding to an attribute value and one of a plurality of data blocks, and an indexing component for execution by the multicore processor element to generate an index block for the plurality of data blocks based on the plurality of block-attribute pairs, the indexing component to perform parallel indexing of the plurality of block-attribute pairs using multiple indexing instances. Other embodiments are described and claimed.