Event-Driven Network Flow Processing Framework

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Data centers face challenges in processing network traffic flow information in a timely manner due to the significant volume of data generated, often resulting in data loss and incomplete processing, especially when using a polling approach.

Innovation Solution

An event-driven approach is implemented to process network traffic flow information, where notifications are sent when data is available for processing, allowing for queued processing without loss, and enabling repeated attempts for incomplete data sets to ensure complete data processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a polling approach is used to process network traffic flow information, then the system can periodically check for data availability, but data loss and incomplete processing occur due to the significant volume of data generated

Engineering Contradiction:

Improvedata processing completenessVSAvoiddata loss

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

Instead of the system polling to check if data is available for processing, the patent inverts the approach by having the data storage component notify the processing component when data becomes available. This event-driven notification mechanism ensures that no data is missed and processing begins immediately when data arrives, eliminating data loss associated with polling intervals.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent implements a feedback mechanism where the data storage component provides notifications to the processing component whenever new network traffic flow information is stored. This feedback loop ensures that processing is triggered by actual data availability, maintaining complete and reliable data processing without the gaps inherent in polling approaches.

Inventive Principle:

Principle #23Feedback

2Productivity

If polling is used to check for data availability, then the system can operate with simple periodic checks, but processing efficiency and speed are reduced due to the high volume of data

Engineering Contradiction:

Improvedata processing speedVSAvoidprocessing delay

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent inverts the control flow by having the storage system actively notify the processing system rather than the processing system continuously polling for data. This eliminates wasted time in periodic checks and ensures processing begins immediately when data arrives, significantly improving processing speed and reducing delays.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The event-driven notification mechanism ensures that processing actions occur continuously and immediately when data is available, rather than being interrupted by idle polling intervals. This continuous action approach maximizes productivity by eliminating unnecessary waiting time and keeping the processing system actively engaged with data as it arrives.

Inventive Principle:

Principle #20Continuity of useful action

3Quantity of substance

If the system processes large volumes of network traffic flow data, then comprehensive monitoring is achieved, but the complexity of managing and processing the data increases significantly

Engineering Contradiction:

Improvevolume of data processedVSAvoidprocessing system complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent segments the data processing system into distinct functional components: a data capture component that collects network traffic flow information, a data storage component that stores the captured data, and a data processing component that processes the stored data. This segmentation allows each component to operate independently and efficiently, managing large volumes of data without overwhelming complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an event notification mechanism as an intermediary between the data storage component and the data processing component. This intermediary simplifies the interaction by providing a standardized notification interface that triggers processing when data is available, reducing the complexity of data management and coordination in high-volume environments.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Reliability

If polling intervals are shortened to reduce data loss, then data processing completeness improves, but the frequency of checks increases system resource consumption

Engineering Contradiction:

Improvedata processing completenessVSAvoidsystem resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent eliminates the need for frequent polling by inverting the approach: instead of the processing system continuously checking for data, the storage system notifies the processing system when data arrives. This achieves complete data processing without the increased resource consumption associated with shortened polling intervals.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The data storage component performs the function of notifying the processing component of data availability, essentially serving itself by triggering the next processing cycle. This self-service mechanism eliminates the need for external polling and reduces system resource consumption while maintaining complete data processing.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10764165B1Event-driven framework for filtering and processing network flows

Publication Date: 2020.09.01 AMAZON TECH INC

AI summary

Time-based groupings of network traffic flow data for virtualized computing resources are stored. Notifications that the time-based groupings are stored are sent, and in response to the notifications, the groupings are processed in accordance with the notifications. Network traffic flow data that is associated with users who have requested publication is published for associated virtualized computing resources.