Hybrid Data Provider Segmentation Delta Merge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in managing and integrating transactional and reporting data models, leading to delays in data updates and inconsistencies across different data stores, which affects query performance and data accuracy.
Innovation Solution
A hybrid data provision system that includes a reporting data model, a transactional data model, and an analytic engine, utilizing additive delta records and data flow paths to aggregate and update data, ensuring quick access to current data while maintaining the benefits of large-scale data analysis and transactional integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a reporting data model uses relatively large data storage structures to facilitate quick access and analysis, then data analysis speed is improved, but data update speed deteriorates and transactional integrity becomes difficult to maintain
Solution Approach 1:
The system segments the data model into two distinct parts: a reporting data model with large aggregated structures optimized for query performance, and a transactional data model with fine-grained structures optimized for updates. This segmentation allows each part to be optimized for its specific purpose without compromising the other.
Solution Approach 2:
The patent introduces an intermediary mechanism (delta storage and merge logic) that bridges the reporting and transactional data models. This intermediary captures changes in the transactional model and applies them to the reporting model, ensuring data integrity while maintaining the performance benefits of large-scale reporting structures.
2Productivity
If a transactional data model uses relatively small data storage structures to facilitate quick modifications, then data update speed is improved, but data analysis performance deteriorates
Solution Approach 1:
The system separates the data model into transactional and reporting components, allowing the transactional portion to use small, modular structures optimized for individual record modifications, while the reporting portion uses large aggregated structures optimized for analytical queries.
Solution Approach 2:
The system performs preliminary actions by maintaining a delta storage that captures changes before they are fully integrated into the reporting model. This allows transactional updates to proceed quickly without being blocked by the need to immediately update large reporting structures.
3Quantity of substance
If data is stored in a single large structure to minimize storage overhead, then storage efficiency is improved, but query performance for specific transactions deteriorates
Solution Approach 1:
The patent segments the storage into multiple levels: the transactional data model stores individual records in small, accessible units, while the reporting data model stores aggregated data in large structures. This segmentation enables efficient storage utilization while allowing fast access to both individual transactions and aggregated analytics.
Solution Approach 2:
Different parts of the data storage system are given different qualities optimized for their specific functions. The transactional portion uses structures optimized for individual record access and modification, while the reporting portion uses structures optimized for bulk data analysis, achieving both storage efficiency and query performance.
4Measurement precision
If the reporting data model is updated frequently to maintain data currentness, then data accuracy is improved, but system performance and update overhead deteriorate
Solution Approach 1:
Instead of continuously updating the reporting data model with every transactional change, the system uses periodic action by batching updates through the delta storage mechanism. Changes are accumulated and applied to the reporting model at optimized intervals, maintaining data currentness while reducing the performance overhead of frequent updates.
Solution Approach 2:
The delta storage acts as an intermediary buffer between transactional operations and reporting model updates. This intermediary allows the system to maintain data accuracy by capturing all changes while deferring and optimizing the actual update operations, thereby preserving system performance.
Data Source
AI summary
Systems and techniques for the hybrid provision of data are described. In one aspect, a system includes a first collection of information stored in accordance with a reporting data model, a second collection of information setting forth a collection of one or more additive delta records, and an analytic engine to perform queries on the first collection of information and the second collection of information to yield an aggregated result set that is based on the contents of both the first collection of information and the second collection of information. The additive delta records each include a numeric value that characterizes the magnitude of a change to a numeric measure in the first collection and identification information that identifies the numeric measure in the first collection. The identified numeric measure is no longer current.