Learning-Based Storage Reduction in Overlay Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing networks, such as content delivery networks, face challenges in efficiently managing and storing large volumes of high-dimensional data, leading to increased costs and delays in real-time reporting due to memory overflow and data backlogs.
Innovation Solution
Implementing a learning-based storage reduction technique using Principal Component Analysis (PCA) to reduce the dimensionality of data, allowing for more efficient storage and retrieval, where the data is processed into a lower-dimensional form and stored with its eigen properties, enabling cost-effective storage and efficient recovery of the original data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high-dimensional data is stored in its original form, then data completeness and accuracy are maintained, but storage costs and processing overhead increase significantly
Solution Approach 1:
The patent extracts and stores only the essential eigen properties (eigenvectors and eigenvalues) of the data rather than the complete high-dimensional dataset. This extraction approach maintains the core informational value while significantly reducing storage requirements, directly resolving the contradiction between data completeness and storage cost.
Solution Approach 2:
The patent transforms the data from its original high-dimensional form into a lower-dimensional representation by changing the parameter space. By storing data in terms of eigen properties rather than raw dimensions, the system achieves efficient storage while preserving reconstructability, thus resolving the storage cost versus data completeness contradiction.
2Quantity of substance
If data dimensionality is reduced for efficient storage, then storage costs decrease, but data reconstruction accuracy may be compromised
Solution Approach 1:
The patent implements a feedback mechanism where the system stores eigen properties that enable accurate reconstruction of the original data. By using eigenvectors and eigenvalues as the basis for reconstruction, the system ensures that dimensionality reduction does not compromise accuracy, as the eigen properties contain the essential information needed to recover the original dataset.
Solution Approach 2:
The patent changes the parameter representation from raw data dimensions to eigen properties (eigenvectors and eigenvalues). This parameter transformation allows for efficient storage while maintaining reconstruction accuracy, because the eigen properties capture the essential variance and structure of the original data in a compressed form.
3Productivity
If large volumes of data are processed and stored, then data availability for real-time reporting is maintained, but memory overflow and processing delays occur
Solution Approach 1:
The patent extracts only the essential eigen properties from the large datasets, significantly reducing the volume of data that needs to be stored and processed. This extraction enables real-time reporting capabilities by reducing memory overflow risks and processing delays, while still maintaining data availability for analytics and billing functions.
Solution Approach 2:
The patent transforms the data parameter space from high-dimensional raw data to lower-dimensional eigen properties, reducing the computational burden for real-time processing. This parameter change enables faster processing and reduces delays in real-time reporting, while the stored eigen properties remain available for reconstruction when needed.
Data Source
AI summary
An overlay network is augmented to provide more efficient data storage by processing a dataset of high dimension into an equivalent dataset of lower dimension, wherein the data reduction reduces the amount of actual physical data but not necessarily its informational value. Data to be processed (dimensionally-reduced) is received by an ingestion layer and supplied to a learning-based storage reduction application that implements the data reduction technique. The application applies a data reduction algorithm and stores the resulting dimensionally-reduced data sets in the native data storage or third party cloud. To recover the original higher-dimensional data, an associated reverse algorithm is implemented. In general, the application coverts an N dimensional data set to a K dimensional data set, where K<<N. The N dimensional dataset has a high dimension, and the K dimensional dataset has a low dimension. From a data storage perspective, it is cheaper and more efficient to store the low dimensional dataset.