Multi-Level Data Downsampling for Lower Memory and Storage Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current downsampling methods lead to high memory costs due to data being stored in memory before being written to disk, necessitating efficient strategies to reduce memory usage while maintaining data trend reconstruction.
Innovation Solution
A data downsampling method that applies user-defined policies at multiple levels after data is written to the database, allowing for multi-level downsampling and storage of downsampled data, thereby reducing memory storage space and improving processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in memory before being written to disk, then data accessibility and query speed are improved, but memory costs increase significantly
Solution Approach 1:
The patent applies preliminary downsampling action to data before it is written to disk storage. By performing downsampling in advance during the data ingestion phase, the system reduces the volume of data that needs to be kept in memory for future queries, thereby lowering memory costs while maintaining query performance for historical data trends
2Volume of stationary object
If downsampling is performed before data is written to disk, then disk storage space is reduced, but memory costs increase due to data being held in memory during the process
Solution Approach 1:
The patent performs downsampling as a preliminary action during data ingestion before data is committed to disk storage. This approach simultaneously achieves disk space reduction and memory cost reduction, as the downsampling process operates on data streams directly without requiring large amounts of data to be held in memory first
Solution Approach 2:
The system implements self-service downsampling where the database automatically applies downsampling policies to incoming data during the write process. This eliminates the need for separate batch processing jobs that would require additional memory resources, as the downsampling is performed inline as part of the normal data ingestion workflow
3Quantity of substance
If multi-level downsampling policies are implemented, then data storage efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments the downsampling process into multiple hierarchical levels, where each level applies a different downsampling factor to data at different time ranges. This segmentation allows the system to manage complexity by breaking down the multi-level downsampling into discrete, manageable policy layers that can be configured and executed independently
Solution Approach 2:
The system implements dynamic downsampling policies where the downsampling factor automatically adjusts based on the age of the data and the configured retention periods. This dynamic approach allows multi-level downsampling to adapt to different time ranges without requiring manual intervention, thereby managing system complexity through automated rule-based logic
Data Source
Figure 1
Figure 2~3
Figure 4~6
AI summary
This application relates to the database field, and provides a data downsampling method. The method includes: receiving downsampling policies at a plurality of levels that are defined by a user, where a downsampling policy at each level indicates to downsample data in the plurality of time shards based on a defined sampling time interval after raw data is stored in a plurality of time shards and a defined time interval has elapsed; determining a first downsampling policy in the downsampling policies at the plurality of levels based on a first time interval that has elapsed after first raw data is stored in a first time shard, where the longer first time interval indicates a longer sampling time interval of the first downsampling policy; obtaining first data currently stored in the first time shard; and downsampling the first data according to the first downsampling policy, to obtain second data. The user may set a downsampling policy based on a requirement of the user, to improve downsampling applicability.