Repeatable Stream Object for Concurrent Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream access methods do not allow multiple consumers to access the same data stream simultaneously without interfering with each other, as consuming data by one consumer renders it inaccessible to others, leading to issues in processing and storage.
Innovation Solution
Implementing a repeatable stream object through a streaming manager API that allows consumers to access a data stream by loading it into memory or disk, enabling concurrent access and real-time processing without requiring knowledge of the stream's size, and allowing configuration of in-memory buffer sizes for efficient resource management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple consumers access the same data stream simultaneously, then productivity is improved, but reliability deteriorates due to data interference and loss
Solution Approach 1:
The patent segments the data stream into discrete record batches that can be independently accessed and processed. Each consumer receives a copy of the same batch rather than competing for access to the same data structure, eliminating interference while maintaining data consistency.
Solution Approach 2:
The patent creates copies of stream data batches for distribution to multiple consumers. Instead of having consumers share a single stream instance, each consumer receives an independent copy, allowing simultaneous access without mutual exclusion or data loss.
2Ease of operation
If the entire stream is loaded into memory for random access, then ease of operation is improved, but volume of memory required increases
Solution Approach 1:
The patent divides the stream into manageable record batches that are loaded into memory only when needed. This segmentation allows the system to work with small, discrete units of data rather than requiring the entire stream to reside in memory simultaneously.
Solution Approach 2:
The patent implements partial loading of stream data into memory based on actual access requirements. Only the portions of the stream that need to be accessed are loaded, rather than pre-loading the entire stream, thus optimizing memory utilization while maintaining access flexibility.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for providing repeatable stream access by multiple consumers. A repeatable stream object is provided, which provides an in-memory buffer and disk storage for handling of a data stream, as well as independent, repeatable, and parallel access to the data stream by multiple consumers. Using the repeatable stream object, consuming of the stream by a first consumer causes the stream to also be placed into the in-memory buffer for later reuse, and does not impact the ability of a second consumer to also consume the stream.