Live Event Filter Split Testing for Reliable Logic Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing event processing systems face challenges in accurately testing software logic modifications due to the limitations of sandbox testing, which often uses historical data that does not reflect current phenomena, leading to unreliable comparisons with live data.
Innovation Solution
Implement a split-testing paradigm where live event data is divided into sub-streams, with one sub-stream processed by the current software logic and another by the modified logic, allowing direct comparison of their performance on real-time data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sandbox testing with historical data is used to test software logic modifications, then testing can be performed in isolation, but the testing accuracy and reliability deteriorate because historical data does not reflect current phenomena
Solution Approach 1:
The patent segments the event stream into multiple portions, with each portion assigned to different filter versions (current and modified). This allows parallel testing of software logic modifications on live data while maintaining system stability, resolving the contradiction between testing reliability and measurement precision by enabling accurate comparison of different filter versions on identical real-time data
Solution Approach 2:
The patent introduces an event routing mechanism as an intermediary that directs events to appropriate filter versions based on configuration parameters. This intermediary enables controlled comparison between current and modified filters on live data, improving testing accuracy while maintaining the reliability benefits of sandbox-like isolation
2Adaptability or versatility
If software logic is modified to process data differently, then adaptability improves, but the risk of introducing errors increases
Solution Approach 1:
The patent implements dynamic filter version management where the system can switch between current and modified filter versions based on configuration parameters and performance metrics. This dynamic approach allows the system to adapt to new data processing requirements while maintaining the ability to revert to proven reliable versions, thus improving adaptability without sacrificing reliability
Solution Approach 2:
The patent incorporates feedback mechanisms that monitor filter performance metrics and enable automated or manual comparison between current and modified filters. This feedback loop allows the system to detect errors in modified logic early and switch back to reliable versions if necessary, maintaining software reliability while enabling adaptive modifications
3Measurement precision
If live event data is split into multiple sub-streams for testing, then testing accuracy improves, but system complexity increases
Solution Approach 1:
The patent implements a universal event routing mechanism that handles multiple filter versions and configuration scenarios through a single unified system. This multi-functional approach enables accurate evaluation of different filter versions on split sub-streams while avoiding the need for separate testing infrastructure, thus improving measurement precision without proportionally increasing system complexity
Data Source
AI summary
During a live test of changes to a current set of event processing filters, a computer system may determine a selected set of event processing filters with which to evaluate a live event (either a current set of event processing filters or a new set of event processing filters in one example). The computer system evaluates the live event according to the selected set of event processing filters. Based on the evaluation, the computer system updates metrics for events evaluated during the live test. The metrics may include a first set of metrics for events evaluated during the live test using the current set of event processing filters and a second set of metrics for events evaluated using the new set of event processing filters. The metrics may be used to decide whether to promote the new set of event processing filters to the current set of event processing filters.