Distributed Block Storage for Scalable Data Stream Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems for streams do not scale well in distributed environments, limiting their flexibility and efficiency in handling and querying data.
Innovation Solution
A block storage system that allows data elements from streams to be stored and retrieved efficiently across distributed devices, using globally unique block identifiers, token names, and index-based querying, enabling flexible and scalable data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data streams are stored in traditional storage systems, then data can be stored and retrieved, but the system does not scale well in distributed environments
Solution Approach 1:
The patent divides data streams into discrete data elements that can be independently stored and retrieved. Each data element is a self-contained unit with its own metadata, allowing the system to segment large data streams into manageable portions that can be distributed across multiple storage devices in a distributed environment.
Solution Approach 2:
The patent introduces a hierarchical indexing structure that adds dimensional organization to data storage. By creating indexes at multiple levels (stream-level indexes and element-level indexes), the system achieves scalable distributed storage by organizing data in additional dimensional layers that enable efficient distribution and retrieval across networked devices.
2Ease of operation
If data elements are stored individually, then retrieval flexibility is improved, but storage efficiency decreases
Solution Approach 1:
The patent merges multiple individual data elements into logical groups called data blocks. These blocks are stored together as units, improving storage efficiency by reducing redundant metadata and enabling batch operations. The merging is maintained through the indexing structure, which allows individual elements to be retrieved from their parent blocks when needed, thus combining storage efficiency with retrieval flexibility.
3Productivity
If indexing is implemented for efficient querying, then query performance improves, but system complexity increases
Solution Approach 1:
The patent extracts indexing functionality into separate, independent index structures that are distinct from the main data storage. By separating indexes from the data blocks themselves and storing them as independent metadata structures, the system achieves efficient querying while maintaining modular architecture that reduces overall complexity through clear separation of concerns.
4Reliability
If data blocks are read and written as single units, then data consistency is maintained, but access speed for individual elements decreases
Solution Approach 1:
The patent implements preliminary indexing actions that prepare data blocks for efficient access. By pre-organizing data elements within blocks and maintaining indexes that map elements to their block locations, the system enables rapid retrieval of individual elements without requiring full block reads. This preliminary organization maintains data consistency at the block level while enabling fast element-level access through index-guided retrieval.
Data Source
AI summary
Systems and methods for storing and retrieving data elements transmitted via data streams received from distributed devices connected via a network. The received data elements may be stored in block stores on the distributed devices. The stored data-elements may be allocated to data-blocks of a block-store that have assigned block-identifiers and further allocated to events of the data-blocks that have assigned token-names. Stream-schema of the received data-streams may comprise a list of token-names and an index-definition for each corresponding data-stream. Indices may be generated for the event-allocated data-elements. A query may be executed in order to retrieve data-elements of the received data-streams based on the indices.