File-Based Time Series Management for High-Frequency Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage system performance graphing tools face difficulties with high-frequency timestamped performance inputs, leading to excessive file numbers and storage issues, as well as sub-optimal data retrieval and graphical rendering, especially when inputs occur every 5 seconds or less over a long period.
Innovation Solution
The method involves recording raw time series data in row-formatted files for fast storage and subsequently converting it into column-formatted files for efficient retrieval and minimal storage space consumption, enabling effective management of time series data for frequent sampling periods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If each timestamped performance input is stored in a separate file, then data retrieval is simple, but the number of files becomes excessively high with high-frequency inputs
Solution Approach 1:
Multiple individual timestamped performance inputs are merged into a single consolidated time series file, reducing the excessive number of separate files while maintaining data accessibility through structured formatting
2Productivity
If a row-formatted file is used for fast storage, then storage speed is improved, but storage space consumption becomes excessive
Solution Approach 1:
The data storage format is changed from row-oriented to column-oriented compression, transforming how data is organized and stored to achieve both fast storage performance and minimal storage space consumption through efficient compression algorithms
3Speed
If a column-formatted file is used for fast retrieval, then data retrieval speed is improved, but writing new entries becomes impractical
Solution Approach 1:
Data is pre-processed and organized into a column-formatted file structure during the storage phase, enabling fast retrieval operations later without requiring costly file close and re-open operations when adding new entries
4Measurement precision
If timestamped performance inputs are recorded frequently every 5 seconds or less, then data completeness is improved, but the size of the database becomes excessive
Solution Approach 1:
The data format parameters are changed to use column-oriented compression, which maintains complete timestamped performance data at high recording frequencies while dramatically reducing the overall database size through efficient compression techniques
Data Source
AI summary
A technique performs file-based time series management. The technique involves initiating a recordation operation that records a source-provided time series in rows of a row-formatted file. The technique further involves, after the recordation operation is initiated, encountering a conversion event. The technique further involves, in response to encountering the conversion event, performing a conversion operation that converts the source-provided time series recorded in the rows of the row-formatted file into a file-provided time series and saving the file-provided time series in columns of a column-formatted file.