Data Aggregation for Database Read Write Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for accessing data structures in databases often compromise either write or read performance, failing to provide satisfactory response times as the number of users and requests increase.
Innovation Solution
Implementing a data aggregation technique where write data structures are separated from read data structures, with pre-aggregation of data occurring immediately or at scheduled intervals to create summarized 'data buckets' for efficient read operations, allowing for flexible time and dimension-based aggregation and caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional read and write techniques are used, then data access is simple and direct, but response time becomes unsatisfactory as the number of users and requests increases
Solution Approach 1:
The patent segments data storage into two separate structures: a write data structure that efficiently handles incoming write requests by appending data, and a read data structure that stores pre-aggregated data for fast reading. This segmentation allows write and read operations to occur independently without interfering with each other, resolving the performance degradation that occurs under concurrent load.
Solution Approach 2:
The system performs preliminary aggregation of data from the write data structure and stores it in the read data structure in advance. This pre-aggregation ensures that when read requests arrive, the data is already organized and ready for rapid retrieval, eliminating the need to process raw data during read operations and thus improving response time under load.
2Speed
If data is pre-aggregated for fast reading, then read performance improves, but write performance may be compromised due to additional aggregation overhead
Solution Approach 1:
The patent extracts the data aggregation function from the write path and places it in a separate aggregation process. Write requests are simply appended to the write data structure without triggering aggregation, maintaining high write throughput. The aggregation process operates independently, extracting aggregated data from the write structure and storing it in the read structure, thus preventing write operations from being slowed down by aggregation overhead.
Solution Approach 2:
The patent introduces an intermediary aggregation process that acts as a mediator between the write data structure and the read data structure. This intermediary handles all the complex aggregation logic, allowing write operations to proceed directly to the write structure without interruption, while read operations retrieve pre-processed data from the read structure. The intermediary isolates the write path from aggregation overhead.
3Measurement precision
If data is stored in detailed form for accurate querying, then query accuracy is maintained, but storage space requirements increase
Solution Approach 1:
The patent changes the parameter of data representation from detailed individual records to aggregated summaries. The read data structure stores data in aggregated form (e.g., sums, counts, averages) rather than storing every individual transaction detail. This parameter change reduces storage space requirements significantly while maintaining the ability to answer common queries accurately through the aggregated data.
Data Source
AI summary
Methods, systems, and computer program products for accessing data in a data structure are described. Data from a write data structure is read and aggregated into a plurality of data buckets, each data bucket corresponding to a time window. Each data bucket is stored in a read data structure.