Distributed Block Storage for Data Stream Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems for streams do not scale well in distributed environments, limiting their flexibility and efficiency in storing and retrieving data.
Innovation Solution
A block storage system that uses globally unique block-identifiers, allocates data-elements to logically grouped data-blocks, and generates indices based on stream-schemas for efficient querying and updating, allowing for concurrent readers and writers across distributed devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data storage systems use traditional centralized architectures, then data retrieval can be simple, but the system does not scale well in distributed environments
Solution Approach 1:
The patent divides the centralized storage system into distributed block stores across multiple devices. Each block store independently manages data blocks with globally unique identifiers, enabling the system to scale by adding more distributed devices without increasing central coordination complexity.
Solution Approach 2:
The patent introduces a mediator mechanism that coordinates between distributed block stores and data streams. The mediator assigns block identifiers, manages data allocation to blocks, and handles stream schema information, enabling scalable distribution while simplifying client interactions.
2Productivity
If data elements are stored without structured grouping, then storage is simpler, but querying and retrieving specific data becomes inefficient
Solution Approach 1:
The patent segments data into discrete data blocks with globally unique identifiers, where each block contains a specific subset of data elements. This segmentation enables efficient querying by allowing clients to request specific blocks rather than scanning entire datasets.
Solution Approach 2:
The patent performs preliminary organization of data elements into blocks with assigned identifiers before retrieval operations. The mediator pre-allocates blocks and assigns data elements to appropriate blocks based on stream schema, enabling fast subsequent queries without complex runtime organization.
3Adaptability or versatility
If the system supports concurrent readers and writers, then system flexibility improves, but data consistency and coordination become more difficult
Solution Approach 1:
The patent segments data access into block-level operations with unique identifiers. Each block can be independently read or written by different clients simultaneously, enabling concurrent operations. The mediator manages block allocation to ensure that concurrent writes to different blocks do not cause consistency issues.
Solution Approach 2:
The mediator acts as an intermediary that coordinates concurrent access to block stores. It manages the assignment of blocks to data streams, handles stream schema information, and ensures that concurrent readers and writers operate on appropriate blocks without causing data consistency problems.
4Reliability
If data blocks are read and written as single units, then data integrity is maintained, but storage space utilization may be less efficient
Solution Approach 1:
The patent segments data into blocks that can be read and written as atomic units, maintaining data integrity at the block level. The globally unique block identifiers enable precise tracking of block contents, allowing efficient atomic operations that preserve integrity while enabling flexible storage utilization.
Data Source
AI summary
Systems, methods and computer readable medium for storing data elements transmitted via data streams received from distributed devices connected via a network. The received data elements may be stored in block stores on the distributed devices. The stored data elements may be allocated to data blocks of a block store that have assigned block identifiers and further allocated to events of the data blocks. Stream schema of the received plurality of data streams may have the same stream schema, and indices may be generated indices based on the order of the event allocated data elements. Stream schema of the received data streams may comprise a list of token names. Token names may be assigned to the event allocated data elements. Indices may be generated for the event allocated data elements based on the stream schema.