Persistent Data Structure for Order Book Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Electronic trading systems generate vast amounts of transient order book data, which is often considered temporary and not stored due to high storage requirements, leading to loss of valuable historical data and impractical storage needs.
Innovation Solution
Implementing a queryable persistent data structure that accumulates and stores order book data over a configurable time span, using structures like RRB-trees, HAMTs, and Patricia trees to efficiently manage and reduce storage needs, allowing for historical data retention and queryability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If order book data is stored in conventional techniques, then historical data is preserved, but storage space requirements become impractically large (120 GB)
Solution Approach 1:
The order book data is segmented into individual events (order additions, modifications, cancellations, matches) rather than storing complete order book snapshots. Each event represents a discrete change, allowing the system to store only the necessary information to reconstruct historical states, reducing storage from 120 GB to 1 GB.
Solution Approach 2:
The patent transforms the storage approach from temporal snapshots (storing complete order book states at different times) to an event-based dimension (storing individual change events with timestamps). This dimensional shift allows efficient reconstruction of historical data without storing redundant information across time points.
2Loss of information
If complete order book snapshots are stored at each change, then data completeness is maintained, but storage requirements increase to 120 GB
Solution Approach 1:
The patent extracts only the essential change information from complete order book snapshots. Instead of storing entire order book states, it extracts and stores only the delta information (what changed, when, and how), which is then sufficient to reconstruct complete historical states when needed.
Solution Approach 2:
The system discards redundant information by not storing complete snapshots, keeping only the minimal event data necessary. When historical data is needed, the complete order book state is recovered by applying the stored events sequentially to the initial state, achieving data completeness without the storage cost of snapshots.
3Volume of stationary object
If order book data is not stored to reduce storage needs, then storage space is minimized, but valuable historical data is lost
Solution Approach 1:
The patent creates a compressed representation (copy) of the historical data in event form rather than storing full snapshots. This event-based copy contains all necessary information to reconstruct the complete historical order book states, preserving data value while minimizing storage space from 120 GB to 1 GB.
4Ease of operation
If conventional storage techniques are used, then data retrieval is straightforward, but storage complexity increases to manage 120 GB
Solution Approach 1:
The patent performs preliminary organization of data into structured events with standardized formats, timestamps, and types during data ingestion. This preliminary structuring enables efficient querying and retrieval operations later, as the data is already organized in a query-friendly format rather than requiring complex processing of raw snapshots.
Data Source
AI summary
An electronic message is read, and a delta is generated based on a comparison of the electronic message to an existing order book. A new order book is generated based on the delta. An event is generated based on the existing order book, the delta, and the new order book. A sequence of events, including the event, is accumulated in a queryable persistent data structure over a time span. The queryable persistent data structure thus efficiently stores representations of order books.