Intelligent Data Compression via Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data intake systems face challenges in efficiently managing large files, which consume significant memory resources and require substantial computing resources for storage and access, leading to increased costs and resource utilization.
Innovation Solution
Implementing a system that uses machine learning models to identify and store only new and important data points and features of files, rather than the entire files, thereby reducing storage requirements while preserving essential information for downstream use.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If entire files are stored in the database, then data completeness is maintained, but storage resources and memory consumption increase significantly
Solution Approach 1:
The system extracts only the essential features and data points from entire files and stores these extracted elements in the database. Machine learning models identify and extract meaningful features (e.g., from images, text, or structured data) while discarding redundant information, thereby maintaining data completeness for analysis purposes while significantly reducing storage requirements.
Solution Approach 2:
Instead of storing complete original files, the system creates simplified copies or representations (feature vectors, metadata, extracted data points) that capture the essential information. These copies serve as surrogates that consume far less storage space while preserving the key characteristics needed for downstream tasks.
2Productivity
If entire files are processed and stored, then all data is available for analysis, but computing resources and processing time increase
Solution Approach 1:
The system extracts only the relevant features and data points needed for analysis from complete files, using machine learning models to identify meaningful patterns. This extraction process reduces the volume of data requiring computation and storage while maintaining the essential information needed for productive analysis.
Solution Approach 2:
Rather than processing and storing complete files (excessive action), the system processes only the essential features and data points (partial action) that are sufficient for downstream analysis tasks. This partial processing approach reduces computing resource consumption while maintaining adequate data availability.
3Quantity of substance
If machine learning models identify and store only new data points and features, then storage efficiency improves, but system complexity increases
Solution Approach 1:
The system employs machine learning models that automatically identify, extract, and store only new and relevant data points and features without requiring manual intervention. The models self-adapt to the data being processed and autonomously determine what information is worth storing, thereby improving storage efficiency while the automated nature reduces the need for complex manual management procedures.
Data Source
AI summary
Systems, computer program products, and methods are described herein for intelligent data compression, in accordance with an embodiment of the invention. The present invention may be configured to receive a plurality of files for storage in a database and perform a series of steps iteratively, for each file of the plurality of files, and until each file of the plurality of files is represented in the database. The series of steps may include identifying one or more data points in the respective file, where each identified data point was previously unidentified in the database and adding the identified one or more data points to the database. The series of steps may also include identifying one or more features of the respective file for storage in the database and storing the identified one or more features in the database as a surrogate for the respective file.