Disk-Resident Streaming Dictionary Using Buffered B-Tree
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern databases struggle to keep pace with high-bandwidth data streams, leading to delayed indexing and storage of only a fraction of available data due to limitations in indexing and searching technologies.
Innovation Solution
A disk-resident streaming dictionary implementation using a buffered B-tree data structure that optimizes disk transfers and buffer management for efficient insertion, deletion, and lookup operations, allowing for faster data processing and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional database indexing technologies are used, then data can be stored and retrieved, but the system cannot keep pace with high-bandwidth data streams, resulting in delayed indexing and storage of only a fraction of available data
Solution Approach 1:
The patent segments the dictionary into multiple pages stored on disk, with each page containing a subset of key-value pairs. This segmentation allows parallel processing of different pages during insertion and lookup operations, enabling the system to handle high-bandwidth data streams by distributing the indexing load across multiple disk pages rather than processing all data sequentially through a single indexing structure.
2Quantity of substance
If more data is stored in the database, then data completeness improves, but the indexing and searching technology cannot keep pace, causing databases to be days to weeks out of date
Solution Approach 1:
The patent implements preliminary action by pre-allocating disk pages and organizing the dictionary structure in advance, allowing data to be inserted and indexed immediately upon arrival without waiting for batch processing. The buffered B-tree structure maintains ready-to-use pages that can accept new key-value pairs instantly, eliminating the delays associated with traditional batch indexing and ensuring the database remains current with incoming data streams.
3Productivity
If a disk-resident streaming dictionary is implemented with buffered B-tree, then data processing speed improves, but the complexity of buffer management and disk transfer optimization increases
Solution Approach 1:
The patent employs dynamics by implementing a buffered B-tree where buffers are dynamically allocated and managed based on the current state of the dictionary. The buffer pool automatically adjusts its contents, loading relevant pages from disk into memory buffers when needed and evicting less frequently accessed pages back to disk. This dynamic buffer management optimizes data processing rate by keeping frequently accessed data in fast memory while maintaining the ability to handle large volumes of data on disk.
Data Source
AI summary
A method, apparatus and computer program product for storing data in a disk storage system is presented. A dictionary data structure is defined and stored on the disk storage system. Key-value pairs can be inserted and deleted into the dictionary data structure, with full transactional semantics, at a rate that is faster than one insertion per disk-head movement. Keys can be looked up with only a logarithmic number of transfers, even for keys that have been recently inserted or deleted. Queries can be performed on ranges of key-value pairs, including recently inserted or deleted pairs, at a constant fraction of the bandwidth of the disk. The dictionary employs indirect logging for physical block logging.