Data Stream Storage Segmentation and Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream storage systems face inefficiencies in recording and accessing data streams with out-of-order arrival, leading to disk space reduction and increased access times due to inefficient disk usage and traversal methods.
Innovation Solution
A data stream storage system that records data into segments with a refresh region at each segment, using bidirectional links and an indexing table to enable traversal in position order, even when records arrive out of order, and includes a purging module for efficient disk space management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If records are appended to segments according to arrival order, then recording efficiency is improved, but access time increases due to out-of-order traversal
Solution Approach 1:
The patent introduces bidirectional links as an intermediary mechanism between records. These links enable traversal in position order by providing direct pointers to neighboring records, eliminating the need to traverse from the beginning of the data stream and thus reducing access time while maintaining efficient appending-based recording
2Quantity of substance
If segments cover different ranges of positions, then disk space utilization is improved, but device complexity increases due to segment management
Solution Approach 1:
The patent divides the data stream into multiple segments, each covering a specific range of positions. This segmentation allows efficient disk space utilization by organizing data into manageable chunks with defined boundaries, while the indexing table simplifies the management complexity by providing a clear mapping between positions and segments
3Reliability
If refresh regions are created at segment boundaries, then data completeness is improved, but storage overhead increases
Solution Approach 1:
The patent applies local quality by creating refresh regions specifically at segment boundaries where data completeness is most critical. These refresh regions contain essential data from adjacent segments, ensuring data integrity at transition points while minimizing storage overhead by limiting refresh regions to only where needed rather than throughout the entire data stream
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A device for recording at least part of a data stream including a recording module to record records divided among a plurality of segments, the records including a record for each instance of an event type of the data stream, each segment covering a range of the positions in the data stream such that different segments cover different ranges of positions in the stream, record the value of each instance, create a refresh region in each segment, the refresh region including data from the most recently recorded instance of each event type, create a plurality of links between the records to enable traversing the records in position order, and create an indexing table including an entry for each segment, the entry for each segment including the range of the positions covered by the records of the one segment. Related apparatus and method claims are also included.