Semi-Sorted Data Store for Time-Series Performance Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for storing and retrieving large volumes of server performance data are inefficient, particularly when dealing with periodic updates and historical data access, as they either prioritize fast write speeds at the expense of slow read speeds or vice versa, and require complex purging and indexing processes, making it difficult for organizations to manage and analyze data in a timely manner.
Innovation Solution
A semi-sorted data store approach is implemented, where data is stored sorted by object and date, with new data appended daily and an index by object maintained, allowing for rapid updates and read access, eliminating the need for continuous sorting and purging, and optimizing data retrieval for both current and historical data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in unsorted form with simple appending, then write speed is fast, but read speed becomes slow and requires periodic purging
Solution Approach 1:
The patent applies preliminary action by pre-sorting data into separate sorted files for each object before they become large. This initial sorting structure is established when data is first received, allowing future read operations to be efficient without requiring continuous re-sorting. The sorted files are prepared in advance with proper indexing, so when read operations occur, data can be quickly located and retrieved without scanning entire unsorted datasets.
2Speed
If data is sorted by object and index maintained, then read access is fast, but data objects are scattered across storage locations requiring disk scans
Solution Approach 1:
The patent applies segmentation by dividing the storage structure into separate sorted files for each object (server, service, etc.). Each file contains data for a specific object sorted by date, rather than having all data in one scattered location. This segmentation allows the index to point to specific file locations, reducing the need for broad disk scans while maintaining organized access structures. The complexity is distributed across multiple small sorted files rather than one large unorganized structure.
3Speed
If sorting is performed on all new and old data each day, then read speeds are fast, but update time becomes very slow
Solution Approach 1:
The patent applies partial action by sorting only the new incoming data into appropriate sorted files, rather than re-sorting all existing data each day. The system maintains existing sorted files unchanged and only performs sorting operations on the current day's new data, appending it to the relevant sorted files. This partial sorting approach maintains fast read speeds on existing data while dramatically reducing update time compared to full re-sorting.
4Ease of manufacture
If data is kept physically separate by day, then purging is simplified, but read speed decreases and code complexity increases
Solution Approach 1:
The patent applies merging by combining data from multiple sources into unified sorted files organized by object. Instead of keeping each day's data in separate physical files, the system merges new data with existing data for the same object into the same sorted file, maintaining chronological order. This merging approach simplifies the storage structure compared to daily separation while preserving read efficiency through the sorted organization and index structure.
Data Source
AI summary
In various embodiments, a method for maintaining and analyzing a rolling history of performance data for a computer network is described. In one embodiment, one year's data is stored, sorted by object and date. New data is appended to the end of the file each day and an index by object is stored. Using this method, data for twelve full months can be provided in object order, and data for one partial month that was added daily in date order can also be provided.