Finite Automata Counter in Data Flow Graph Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics platforms face challenges in efficiently processing large volumes of data due to limitations in data stream processing architectures, particularly in applying data manipulation operations like regular expressions, filtering, and data retrieval.
Innovation Solution
The use of highly programmable data stream processing devices, referred to as data processing units (DPUs), which are specialized data-centric processors that leverage data flow graphs to efficiently apply data manipulation operations to streams of data units, utilizing hardware-based accelerators for regular expression processing and other analytical operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If general purpose processors are used to execute query predicates on large volumes of data, then programming flexibility is maintained, but processing speed and efficiency deteriorate
Solution Approach 1:
The system divides the data processing function into separate components: a data stream processing device for high-speed execution of query predicates using specialized hardware accelerators, and a general purpose processor for coordinating the overall analytics operation. This segmentation allows each component to be optimized independently, with the data stream processing device handling speed-critical operations while the general purpose processor maintains programming flexibility.
Solution Approach 2:
The patent introduces a data stream processing device as an intermediary between the general purpose processor and the data. This intermediary component receives query predicates from the general purpose processor, translates them into efficient operations for the hardware accelerators, and executes them at high speed on streaming data, thereby resolving the contradiction between flexibility and speed.
2Productivity
If hardware-based accelerators with specialized architectures are used, then processing efficiency improves, but adaptability to different analytical operations deteriorates
Solution Approach 1:
The hardware accelerator incorporates a state machine that can dynamically transition between different operational modes based on the type of analytical operation being performed. The state machine adapts its behavior according to the query predicate, allowing the same hardware architecture to efficiently handle various operations such as filtering, aggregation, and data transformation without requiring separate specialized hardware for each operation type.
Solution Approach 2:
The data stream processing device is designed with universal hardware accelerators that can perform multiple analytical operations through a single unified architecture. The hardware includes general-purpose processing cores capable of executing different query predicates and analytical operations, making the system adaptable to various data processing tasks while maintaining high processing efficiency through hardware acceleration.
3Speed
If data is processed in streaming mode through specialized devices, then processing speed improves, but data manipulation operation flexibility deteriorates
Solution Approach 1:
The system creates a virtualized representation of the data stream processing device that can be copied and instantiated multiple times. This virtualization layer provides a standardized interface for submitting data manipulation operations, abstracting the complexity of the underlying hardware accelerator. Users can submit operations through a consistent API while the system handles the speed-optimized streaming processing internally, maintaining both flexibility and speed.
Data Source
AI summary
System and methods for performing analytical operations are described. A hardware-based regular expression (RegEx) engine performs a regular expression operation on a stream of data units based on a finite automata (FA) graph. Performing includes configuring a regular expression engine of a hardware-based regular expression accelerator to, beginning at a root node in the plurality of nodes of the FA graph, step the regular expression engine through one or more nodes of the FA graph until the regular expression engine arrives at a skip node and to consume, at the skip node, two or more data units from the stream of data units before traversing one of the directional arcs to another node.