In-Memory Message Transformation for Database Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transforming and storing messages in large databases consumes significant time and computing resources due to the use of high-capacity memory devices with slower read or write times, such as hard disk drives, which are much slower than faster memory devices like semiconductor memory.
Innovation Solution
The system processes and transforms messages in an in-memory database with faster read/write times before transferring them to an on-disk database with slower read/write times, reducing the need for extensive computing resources and storage capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If messages are transformed in an on-disk database using high-capacity memory devices, then storage capacity is sufficient, but read/write times are slow and computing resources are consumed excessively
Solution Approach 1:
The system divides the database into two segments: an in-memory database (IMDB) for active message transformation and an on-disk database (ODDB) for archival storage. This segmentation allows fast in-memory operations for processing while maintaining adequate storage capacity in the ODDB, resolving the contradiction between storage capacity and access speed.
Solution Approach 2:
The in-memory database acts as an intermediary between the interface and the on-disk database. Messages are first loaded into the IMDB for transformation operations, then transferred to the ODDB only after processing. This intermediary approach eliminates the need to perform transformations directly on the slow on-disk database, reducing read/write time while preserving storage capacity.
2Quantity of substance
If messages are transformed in an on-disk database, then storage capacity is maintained, but computing resources are consumed excessively
Solution Approach 1:
The system segments processing functions from storage functions. The IMDB handles computationally intensive transformation operations using fast in-memory access, while the ODDB provides storage capacity. This segmentation reduces the computing resources required for transformations by avoiding repeated reads from and writes to the slow on-disk database.
Solution Approach 2:
Messages are loaded into the in-memory database and transformed before being transferred to the on-disk database. This preliminary action in memory completes all necessary transformations before persistent storage, eliminating the need for repeated computing operations on the on-disk database and reducing overall computing resource consumption.
3Manufacturing precision
If all associated messages are loaded into in-memory database before transformation, then transformation accuracy is improved, but memory usage increases
Solution Approach 1:
The system loads only the necessary associated messages into the in-memory database that are required for accurate transformation, rather than loading all possible messages. This partial action approach maintains transformation accuracy by having all necessary data available in memory while minimizing memory usage by loading only what is essential for the transformation process.
Data Source
AI summary
A system for storing messages in an on-disk database (ODDB), comprises an interface operable to receive a first message comprising first data from an execution venue, wherein the first message is one of a group of associated messages related to a financial transaction, a processor communicatively coupled to the interface, the processor is operable to load the first message into an in-memory database (IMDB), identify a second message in the group of associated messages, transform the second message with the first data from the first message, and determine whether all messages in the associated group of messages have been loaded into the IMDB, and the interface is further operable to transfer the first message to the ODDB.