Time Series Compression Preserving Abnormal Points and Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large volume of time series data generated during semiconductor fabrication poses challenges in storage, and existing compression techniques face issues in achieving a high compression ratio while maintaining data quality, especially when using lossy compression methods.
Innovation Solution
The method involves reading original time series data, measuring units, determining a threshold for data difference using clustering, performing Longest Distance downsampling with Flexible Interval to preserve abnormal points, storing start and end points, and performing drop-out on downsampling results, while storing reference points if the data length is below a certain threshold, and restoring data through linear interpolation and noise addition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If lossy compression is used to compress time series data, then compression ratio is improved, but data quality deteriorates
Solution Approach 1:
The time series data is divided into multiple intervals, and different processing strategies are applied to each interval. Critical intervals containing abnormal points are preserved with higher fidelity, while normal intervals are compressed more aggressively, resolving the contradiction between compression ratio and data quality.
Solution Approach 2:
Different quality levels are applied to different parts of the data based on their importance. Abnormal points and critical regions maintain higher precision while other regions use lower precision, achieving both high compression ratio and preserved data quality where it matters most.
2Manufacturing precision
If abnormal points are preserved during downsampling, then data quality is improved, but compression ratio deteriorates
Solution Approach 1:
The method performs preliminary identification of abnormal points before downsampling. By detecting and marking abnormal points in advance, the downsampling process can selectively preserve only these critical points while compressing the rest of the data, maintaining data quality without sacrificing compression ratio.
Solution Approach 2:
Instead of preserving all original data points, the method creates a selective copy of only the abnormal and critical points during downsampling. This selective copying approach maintains essential data quality while achieving high compression ratios by discarding redundant normal points.
3Manufacturing precision
If start point and end point are preserved during compression, then data quality is improved, but compression ratio deteriorates
Solution Approach 1:
The start point and end point are extracted and preserved separately from the main compression process. These critical boundary points are stored with full precision while the intermediate data undergoes aggressive compression, achieving both high compression ratio and preservation of essential data quality at boundaries.
Data Source
AI summary
A method for compressing time series data includes: reading original data including time series data; measuring a unit of the original data; determining a threshold for determining a range allowing for a difference between compressed data and the original data; performing longest distance downsampling to preserve an abnormal point of the original data; storing a start point and an end point of the original data; performing drop-out on the longest distance downsampling result, wherein the drop-out is not performed for the start point and the end point; and if the number of values of the original data is smaller than α times the original data length, storing at least one of the values of the original data as a reference point, wherein α is a real number having a value between 0 and 1.