Neural Data Storage Mapping for High-Dimensional Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage methods for high-dimensional data require significant storage space, even after using auto encoders for compression, as they still need to store large amounts of compressed low-dimensional data and network model parameters.
Innovation Solution
A method involving N-dimensional permutation of a parameter vector to generate multiple second parameter vectors, which are used to train a neural network model, allowing for the storage of a single low-dimensional first parameter vector that represents multiple high-dimensional data points, thereby reducing storage space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If auto encoders are used to compress high-dimensional data to low-dimensional data, then storage space is reduced to a certain extent, but large amounts of compressed low-dimensional data and network model parameters still need to be stored
Solution Approach 1:
The patent merges N data points into a single low-dimensional parameter vector through N-dimensional permutation and neural network mapping. Instead of storing N separate compressed vectors, the system stores one parameter vector that represents all N data points, thereby reducing storage quantity while maintaining data integrity through the learned mapping relationships.
Solution Approach 2:
The patent transforms the storage problem from storing N separate d-dimensional vectors to storing one Nd-dimensional parameter vector. By changing the dimensional organization from (N, d) to (Nd, 1), the system achieves more efficient storage representation while the neural network model enables reconstruction of original data when needed.
2Volume of stationary object
If a single low-dimensional parameter vector is used to represent multiple high-dimensional data points, then storage space is significantly reduced, but data integrity and accuracy may be compromised
Solution Approach 1:
The patent employs feedback mechanisms through the neural network model that learns bidirectional mappings between the parameter vector and high-dimensional data. The model is trained to ensure that when data is reconstructed from the parameter vector, the original information is preserved within acceptable error margins, thus maintaining data integrity despite compression.
Solution Approach 2:
The patent changes the parameters of the data representation by transforming fixed-dimensional data into a learned parameter space. Through N-dimensional permutation and neural network transformation, the system optimizes the parameter encoding to retain essential information while reducing dimensionality, balancing storage efficiency with data fidelity.
Data Source
AI summary
Embodiments of the present application provide a data storage method, data acquisition method and device thereof. The method includes allocating an N-dimensional first parameter vector for N pieces of to-be-stored data; performing N-dimensional permutation on the first parameter vector, to obtain N second parameter vectors each having N dimensions; constructing a neural network model that maps the current second parameter vectors to expected data samples of the N pieces of to-be-stored data; adjusting model parameters of the neural network model and/or the first parameter vector until expected data samples of the N pieces of to-be-stored data regress to the N pieces of to-be-stored data, the expected data samples being obtained from the current second parameter vectors based on the trained neural network model; storing the current first parameter vector. The embodiments of the present application make the storage of the first parameter vector equivalent to storing N pieces of to-be-stored data, which reduces high-dimensional data to low-dimensional data for storage, thus greatly reducing the storage space.