Phase Algorithm for High-Volume Message Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in handling high message arrival rates, leading to either delayed processing or loss of data, as they struggle to keep up with the capacity to process individual messages efficiently, resulting in outdated or incomplete information display.
Innovation Solution
The implementation of programmable mathematical algorithms that tag message data with phase numbers, allowing the system to manage data by either delaying or losing messages, using the Phase Algorithm and Macro-Phase Algorithm to optimize processing and reduce errors during data overload conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the system processes messages at a high rate to maintain accuracy, then information display accuracy is improved, but the system capacity is exceeded causing data loss
Solution Approach 1:
The patent segments the continuous data stream into discrete phases, where each phase represents a time window for processing messages. This segmentation allows the system to manage overwhelming data by processing messages in manageable chunks rather than attempting to process everything simultaneously, thus maintaining accuracy while preventing capacity overload.
Solution Approach 2:
The patent implements a mechanism that skips or rushes through phases when the system detects it cannot process all messages within a phase. When the message arrival rate exceeds processing capacity, the system advances to the next phase without processing all accumulated messages, effectively skipping the incomplete processing cycle to maintain system responsiveness and prevent total data loss.
2Reliability
If the system delays processing to handle high message rates, then data loss is reduced, but information timeliness deteriorates
Solution Approach 1:
The patent employs periodic action by processing messages in repeated cycles of phases. Each phase operates as a periodic processing window where messages are handled in batches. This periodic structure allows the system to maintain data completeness by continuously cycling through processing phases while managing the trade-off between processing speed and data loss through controlled phase advancement.
3Reliability
If the system processes all messages to maintain completeness, then data loss is minimized, but processing speed decreases
Solution Approach 1:
The patent implements dynamic processing by adjusting the phase duration and processing depth based on real-time system conditions. When message arrival rates are high, the system dynamically shortens phase durations or reduces processing granularity to maintain speed. When rates are lower, the system can process more thoroughly within each phase, thus dynamically balancing completeness and speed according to actual workload conditions.
Data Source
AI summary
A method for processing data in the form of a stream of messages regarding, for example, stock price information, implemented using a computer system wherein the data may arrive at a rate faster than the computer system can process individual messages. Each message is tagged with a phase number as it arrives; only the data in the messages received at the end of a phase are stored in the computer system's database. Periodically, at the end of at least one phase and perhaps at the end of many phases depending on the rate messages are received, the computer queries its database to select information regarding the data. Algorithms, according to the present invention, allow the efficient selection of data by disregarding a portion of the information in some cases, and, in other cases, by finding the phase ranges wherein data messages overlap between phase ranges. In the former, data is lost; it the latter, data processing may be slowed. The selected information is then output.