Real-Time Stream Processing with Map and Update Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Individuals face difficulties in locating relevant content in social networking services and other information sources due to the high volume and rapid pace of content updates.
Innovation Solution
The development of systems and methods for performing large-scale, long-running stream computations on real-time data streams using map and update operations, which process and transform data in real-time to generate new stream events and update static data structures stored persistently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If real-time data streams are processed at high speed to keep up with frequent content updates, then the speed of information delivery is improved, but the difficulty of locating relevant content increases due to sheer volume
Solution Approach 1:
The patent segments the overwhelming data stream into manageable chunks by processing events in discrete batches using map operations. Each map operation handles a specific segment of the data stream, transforming raw events into structured intermediate results that can be further processed. This segmentation makes the volume of data tractable while maintaining real-time processing capabilities.
Solution Approach 2:
The patent introduces intermediary data structures (slates) that act as mediators between the high-speed incoming data stream and the final output. These slates accumulate and organize intermediate results from map operations, providing a structured buffer that facilitates efficient processing and retrieval without losing the speed advantages of real-time streaming.
2Quantity of substance
If all stream events are processed and stored to ensure completeness of information, then the quantity of available information is improved, but the complexity of processing and storage increases
Solution Approach 1:
The patent applies local quality by allowing different portions of the data stream to be processed with different levels of detail and storage. Not all events require the same processing intensity - some are filtered out, some are summarized, and only relevant events are stored in detail. This differential processing reduces overall system complexity while preserving essential information.
Solution Approach 2:
The patent dynamically changes processing parameters based on the characteristics of the incoming data stream. Map operations can adjust their filtering, aggregation, and transformation parameters in real-time based on data volume, event types, and system load. This adaptability reduces processing complexity during high-volume periods while maintaining information completeness when feasible.
3Reliability
If stream computations are performed continuously to maintain real-time accuracy, then the reliability of information is improved, but the energy consumption increases
Solution Approach 1:
The patent implements periodic action by processing data streams in discrete batches or intervals rather than continuously. Map operations are triggered by events or time intervals, allowing the system to maintain real-time accuracy for critical events while reducing processing frequency for less time-sensitive data. This periodic processing maintains reliability where needed while reducing overall energy consumption.
Solution Approach 2:
The patent applies partial action by processing only the necessary portion of the data stream at any given time. Map operations selectively transform and filter events based on predefined criteria, focusing computational energy on relevant events while skipping or summarizing less important data. This selective processing maintains information reliability for key events while reducing total energy expenditure.
Data Source
AI summary
Exemplary embodiments allow performance of stream computations on real-time data streams using one or more map operations and/or one or more update operations. A map operation is a stream computation in which stream events in one or more real-time data streams are processed in a real-time manner to generate zero, one or more new stream events. An update operation is a stream computation in which stream events in one or more real-time data streams are processed in a real-time manner to create or update one or more static “slate” data structures that are stored in a durable manner.