Resource Server Compressed Stream Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream processing systems face challenges in efficiently serving and updating data resources to geographically distributed client systems, particularly in handling continuous data streams and performing analytics on these streams in a scalable and efficient manner.
Innovation Solution
A system and method that involves a stream processing system with an event publishing system, an event stream processing device, and a resource server, where streamed data is compressed and stored in pre-allocated memory blocks, and updates are managed using a tree map based on unique event identifiers, enabling efficient compression, storage, and delivery of data resources to client systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in traditional memory structures, then data access is simple, but data retrieval efficiency deteriorates when handling continuous data streams with frequent updates
Solution Approach 1:
The patent segments continuous data streams into discrete blocks that can be independently compressed and stored. Each block is processed and stored separately in pre-allocated memory blocks, allowing efficient retrieval without managing entire data streams as single units. This segmentation enables parallel processing and reduces memory management overhead.
Solution Approach 2:
The patent implements pre-allocation of memory blocks before data arrival. By preparing memory structures in advance and organizing them with tree map indexes, the system eliminates the need for dynamic memory allocation during data retrieval operations, significantly improving access speed and reducing latency.
2Quantity of substance
If compression is applied to reduce data size, then storage efficiency improves, but processing time increases
Solution Approach 1:
The patent performs compression operations on data blocks as they arrive and stores the compressed versions in pre-allocated memory. By compressing data upfront rather than on-demand, the system transforms time-consuming compression operations into background processing tasks, enabling rapid retrieval of already-compressed data without repeated compression overhead.
Solution Approach 2:
The patent creates and stores compressed copies of data blocks in pre-allocated memory regions. These compressed copies serve as efficient storage representations that can be quickly retrieved and decompressed when needed, rather than storing and processing full-size original data, thus reducing overall storage requirements and improving access efficiency.
3Speed
If pre-allocated memory blocks are used, then data access speed improves, but memory utilization efficiency deteriorates
Solution Approach 1:
The patent dynamically adjusts the parameters of pre-allocated memory blocks based on actual data characteristics and access patterns. By monitoring data arrival rates, block sizes, and access frequencies, the system optimizes memory allocation parameters to balance access speed with utilization efficiency, preventing both over-allocation and under-allocation of memory resources.
Solution Approach 2:
The patent implements feedback mechanisms that monitor memory block utilization and adjust allocation strategies accordingly. When memory blocks are underutilized, the system can reallocate or consolidate resources. When access patterns change, the feedback loop triggers reorganization of memory structures to maintain optimal performance, thus adapting to varying workloads without permanent resource waste.
Data Source
AI summary
A computing device is provided that supports receipt and processing of blocks of streamed data associated with an event and provision of pre-rendered and compressed events to event client systems with a high-throughput. The computing device includes a compression optimizer module that determines how to make new compressed blocks of the right level of compression as the blocks of streamed data are received from an event publishing system. The compression optimizer module utilizes a tree map and unique identifiers to selectively compress, read, and recompress the blocks of streamed data associated with each event. To prevent duplicate work in compressing the streamed data, compressed blocks are re-used where possible to support requests from the event client systems. Each new block includes a unique identifier defined by the event publishing system that generated the event, so that a previous version of the event in a compressed block can be replaced.