Big-fast Data Connector for In-memory Database and Warehouse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for handling 'Big Data' and 'Fast Data' are inadequate for real-time or near real-time analysis due to limitations in query performance, data accessibility, and the inability to combine historical and real-time data effectively.
Innovation Solution
A fast ingest module is introduced to facilitate micro-batch updates between in-memory database systems and data warehouses, utilizing a distributed architecture with a listener and queue system to enable real-time data processing and replication, thereby overcoming the limitations of traditional OLTP and OLAP systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in separate OLTP and OLAP systems, then each system can be optimized for its specific function, but real-time analysis of combined historical and real-time data becomes difficult
Solution Approach 1:
The patent introduces a data connector as an intermediary component that bridges the OLTP and OLAP systems. This connector enables real-time data exchange and integration between the two previously separate systems, allowing historical and real-time data to be combined for analysis without compromising the optimized architecture of either system
Solution Approach 2:
The patent merges the functionality of separate OLTP and OLAP systems by implementing a unified data architecture where real-time transactional data and historical analytical data can be accessed together. The data connector facilitates this merging by enabling bidirectional data flow and integration between the systems
2Quantity of substance
If OLAP systems process large batches of data, then they can handle big data volumes, but processing delays occur
Solution Approach 1:
The patent segments the large batch data processing into smaller, more manageable units. By dividing the data flow into smaller batches or streams that can be processed incrementally, the system maintains the ability to handle large data volumes while reducing the processing delays associated with monolithic batch operations
Solution Approach 2:
The patent implements dynamic data processing where the batch size and processing frequency can be adjusted based on system conditions and requirements. This dynamic approach allows the OLAP system to adapt between handling large volumes and maintaining processing speed, rather than being locked into fixed batch schedules
3Speed
If OLTP systems operate in real-time, then they provide fast data processing, but they cannot handle huge volumes of data needed for deep analytics
Solution Approach 1:
The patent makes the OLTP system multi-functional by enabling it to not only handle real-time transactions but also to serve as a source for analytical processing. The data connector allows the OLTP system to fulfill both its real-time processing role and to contribute to large-scale analytics, effectively adding analytical capabilities to the transactional system
4Quantity of substance
If data is transmitted in large batches between systems, then data volume requirements are met, but query performance and responsiveness deteriorate
Solution Approach 1:
The patent applies partial action by transmitting only the necessary portions of data between systems rather than moving entire datasets. The data connector enables selective data transfer based on query requirements, updating only the specific data portions that need to be accessed, thereby maintaining query performance while still providing access to large data volumes when needed
Data Source
AI summary
Embodiments of the present invention include systems and methods for insuring better query consistency between at least two different databases, where one faster database has more up-to-date information than another slower database, and wherein updates are typically applied to the faster database first and then to the slower database. In embodiments, the systems and methods also insure that a query to the slower database is not performed until a set of one or more updates from the faster database have been applied to that slower database.