On-Board Time-Series Storage with Partitioned Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
On-board databases face challenges with large amounts of time sequence data, consuming significant computing resources and storage capacity, leading to performance issues and potential device breakdowns due to limited resources and bandwidth constraints when processing and storing data.
Innovation Solution
A method for on-board data storage that involves determining nodes and partitions based on time sequence data, writing data into memory buckets, merging data, compressing it using timestamp and index value differences, and storing summary data in summary trees to optimize storage and querying efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all raw data is transmitted back to the cloud for analysis and calculation, then data processing capability is improved, but bandwidth consumption increases and real-time transmission requirements cannot be met
Solution Approach 1:
The patent segments data into different types (raw data, processed data, summary data) and processes them locally on the on-board device. This segmentation allows the system to handle different data categories independently, reducing the need to transmit all raw data to the cloud while maintaining real-time processing capabilities for time-sensitive operations.
Solution Approach 2:
The patent performs preliminary data processing and analysis locally on the on-board device before potential cloud transmission. By pre-processing data locally, the system reduces the volume of data that needs to be transmitted to the cloud, thereby conserving bandwidth while still enabling comprehensive data analysis through a combination of local and cloud-based processing.
2Quantity of substance
If a large amount of raw data is written into the on-board hard disk with high frequency, then data storage capacity is improved, but the hard disk capacity is fully occupied leading to device breakdown
Solution Approach 1:
The patent implements a data lifecycle management strategy where raw data is processed, transformed into processed data, and then discarded or archived selectively. This approach prevents the hard disk from being permanently filled with raw data while maintaining the ability to recover and utilize processed data, thus balancing storage capacity with device reliability.
Solution Approach 2:
The patent changes the state and format of data through processing operations, transforming raw data into processed data with different characteristics and storage requirements. This parameter change allows the system to reduce storage demands while preserving essential information, preventing hard disk capacity exhaustion.
3Measurement precision
If original data is directly queried from the on-board database for analysis and calculation, then data accuracy is improved, but computing resources and time are excessively consumed
Solution Approach 1:
The patent performs preliminary data processing and creates processed data locally on the on-board device before potential cloud transmission. By pre-processing data locally, the system reduces the volume of data that needs to be transmitted to the cloud, thereby conserving bandwidth while still enabling comprehensive data analysis through a combination of local and cloud-based processing.
Solution Approach 2:
The on-board device performs self-service data processing and analysis operations, handling time-sensitive and critical data locally without requiring constant cloud communication. This self-service capability reduces computing resource consumption for data transmission and processing while maintaining data accuracy for local operations.
Data Source
AI summary
This invention is directed to an on-board data storage method and system. The on-board data storage method includes obtaining, in a current time period, various pieces of time sequence data to be processed written into the on-board database; for each piece of the time sequence data to be processed, determining a node to be stored in the on-board database and a partition to be stored in the node according to the piece of the time sequence data to be processed, and writing the piece of the time sequence data to be processed into a corresponding time sequence in the determined partition to be stored; and writing each piece of the time sequence data written into each partition to be stored into each corresponding data bucket in memory.