Network Data Aggregation and Change Detection for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale network applications face challenges in efficiently processing and storing vast amounts of data, which leads to high storage and processing costs, increased I/O operations, and potential degradation of user interface experience.
Innovation Solution
A multistep optimized solution involving data aggregation, change detection, and middle layer caching is implemented to reduce data volume, enhance storage efficiency, and improve data retrieval speeds. This solution includes aggregating data over time to convert non-deterministic data into deterministic data, comparing aggregated data to stored data for publication only when different, and using a middle layer cache to delegate data retrieval requests and reuse cached responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If raw data from all network devices is stored in full detail, then complete historical data is available for analysis, but storage costs and data processing volume increase significantly
Solution Approach 1:
The patent extracts only the essential information from raw network device data by implementing change detection mechanisms. Instead of storing all raw data, the system extracts and stores only state changes and relevant metadata, significantly reducing data volume while preserving analytical value.
Solution Approach 2:
The patent applies different data storage strategies to different network devices based on their importance and data characteristics. Critical devices maintain detailed historical records while less critical devices use aggregated or sampled data, optimizing the balance between data completeness and storage efficiency.
2Productivity
If detailed data is stored for all network devices, then comprehensive analysis is possible, but I/O operations and processing costs increase
Solution Approach 1:
The patent performs preliminary data aggregation and change detection at the data collection stage rather than during analysis. By pre-processing and filtering data to retain only state changes, the system reduces the amount of data that requires subsequent processing, storage, and retrieval operations.
Solution Approach 2:
The patent changes the data representation parameters from raw detailed records to aggregated state change representations. This parameter transformation reduces data complexity and processing requirements while maintaining the essential information needed for network analysis.
3Reliability
If historical data is stored in full detail, then accurate troubleshooting is possible, but user interface responsiveness degrades
Solution Approach 1:
The patent segments historical data into discrete state change events and time intervals. This segmentation allows the system to retrieve and present only the necessary portions of historical data during troubleshooting, improving UI responsiveness while maintaining accurate diagnostic capability through preserved state transition information.
Data Source
AI summary
In certain embodiments, a computer-implemented method includes aggregating data messages related to a device that are transmitted over a network within a defined aggregation period into a single data message; comparing the single data message to previously-aggregated data from a preceding aggregation period stored in a data store to determine whether the single data message differs from the previously-aggregated data; storing, in response to determining that the single data message differs from the previously-aggregated data, the single data message to the data store; and regenerating a state timeline of the device using the stored data in the data store.