Anomalous Operational Data Detection via Storage Volume Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern network computing and data storage systems face challenges in efficiently storing and processing operational data from diverse devices, such as sensors, which requires adaptive storage solutions that differentiate between primary and derivative data significance.
Innovation Solution
A data storage system that allocates multiple storage volumes in a sequence for operational data, allowing for interpolation and anomaly detection, using redundancy encoding techniques and jitter analysis to ensure durability and redundancy, while providing progressively refined data to users based on request granularity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If operational data from diverse devices is stored in a distributed network storage system, then data availability and accessibility are improved, but data storage complexity and management difficulty increase
Solution Approach 1:
The patent segments operational data into primary data and derivative data categories, with different storage strategies for each. Primary data is stored in a distributed file system across multiple nodes, while derivative data is stored in a columnar storage format optimized for analytical queries. This segmentation allows the system to provide high data availability through distributed storage while managing complexity through differentiated storage policies for different data types.
Solution Approach 2:
The patent implements a universal data lake architecture that can store and process both primary operational data and derivative analytical data using a single distributed storage system. This multi-functional approach eliminates the need for separate storage systems for different data types, reducing overall storage management complexity while maintaining high availability through the unified distributed architecture.
2Loss of information
If extensive operational data is stored for later analysis, then data significance for derivative contexts is improved, but storage resource consumption increases
Solution Approach 1:
The patent applies local quality by storing primary operational data in its original format at the source with high redundancy for immediate access, while storing derivative analytical data in a compressed columnar format optimized for analysis. This allows the system to preserve data significance for both immediate operations and later analysis while optimizing storage resource consumption based on the specific needs of each data type.
Solution Approach 2:
The patent implements partial storage by selectively retaining only the most significant operational data for long-term archival, while using compression and aggregation techniques to reduce the storage footprint of less critical data. This approach ensures that data significant for derivative contexts is preserved while minimizing overall storage resource consumption through selective retention and compression.
3Reliability
If data is stored with high redundancy for durability, then data reliability is improved, but storage efficiency decreases
Solution Approach 1:
The patent segments data storage into primary data with high redundancy factors (e.g., 3x replication) for critical operational data that requires high durability, and derivative data with lower redundancy factors for analytical data where some loss is acceptable. This segmentation allows the system to achieve high data durability for critical data while maintaining storage efficiency for less critical data, resolving the contradiction between reliability and storage efficiency.
Data Source
AI summary
A system stores data, such as sensor data or other operational data, on a plurality of storage volumes in a sequence so as to allow for interpolations or other approximations of the data using a subset of the storage volumes in response to a request for information regarding that data. For example, a plurality of devices connect to the system to provide operational data, which is then stored in a specified sequence on a specified set of volumes. In response to a request for operational information regarding some or all of the devices, the system reads at least one of the volumes, and approximates the values of the data over a specified period of time. In some embodiments, the data may be buffered prior to storage, and a jitter analyzer determines whether the incoming data is anomalous relative to a baseline, which may be determined using related data sets.