Database Event Partitioning for Low Latency Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems are inefficient in storing and retrieving irregular stream data, leading to slow query performance, especially in high-frequency trading and other applications where low latency and high volumes of data are critical.
Innovation Solution
A database system that processes data by partitioning incoming event instances into logical partitions, using doubly-linked lists and event wrappers to efficiently store and retrieve data, allowing for user-defined iteration methods and dynamic configuration, and employing a flow manager to manage data flow and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional database systems store irregular stream data, then data can be stored, but query performance becomes slow and latency increases
Solution Approach 1:
The database partitions incoming event instances into multiple logical data partitions based on user-defined iteration methods. Each partition is organized with efficient data structures (doubly-linked lists, hash tables) that enable rapid traversal and retrieval. This segmentation allows the system to process and query specific subsets of data quickly without scanning entire datasets, directly improving query performance and reducing latency.
Solution Approach 2:
The system pre-processes incoming stream data by assigning identifiers, partitioning events, and organizing them into optimized data structures before queries are executed. User-defined iteration methods are established in advance, and the database maintains pre-computed indexes and references (such as references to the last event instance in each partition) that enable immediate retrieval operations, eliminating the need for time-consuming data organization during query execution.
2Productivity
If data is partitioned into multiple logical partitions with user-defined iteration methods, then retrieval efficiency improves, but system complexity increases
Solution Approach 1:
The database employs a universal event instance structure that can represent multiple types of events (trades, quotes, orders, etc.) through a common data model. The partitioning mechanism and data structures are designed to handle diverse event types uniformly, allowing the same retrieval logic to work across different event categories. This multi-functionality simplifies the system by avoiding the need for separate handling mechanisms for each event type, thereby improving retrieval efficiency without proportionally increasing complexity.
Solution Approach 2:
The system introduces intermediary data structures (such as partition identifiers, event wrappers, and reference pointers) that mediate between the raw stream data and the query operations. These intermediaries organize and simplify access to the underlying complex partitioned data, allowing efficient retrieval without requiring the query mechanism to directly navigate complex data relationships. The intermediary layer abstracts the complexity while maintaining high retrieval efficiency.
Data Source
AI summary
A non-transitory computer-readable medium having computer-readable instructions stored thereon which, when executed by a computer, cause the computer to perform a method of processing data comprising the steps of: receiving data associated with event instances; and for each of a plurality of iteration methods: partitioning the incoming event instances into logical data partitions; assigning an identifier to each event instance such that events classified in the same logical data partition receive the same identifier and a given event instance is always assigned the same identifier; and inserting each event instance into a doubly-linked list associated with the identifier in an appropriate location.