Data Management System for AI Model Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques for AI learning and inference do not effectively reduce storage capacity for data verification purposes, as they store data continuously, leading to increased storage requirements for long-term problem analysis and investigation.
Innovation Solution
A data management system that includes a processor and a compressor/decompressor for compressing and decompressing data no longer used for learning and inference, allowing for reduced storage capacity by adapting compression methods based on data usage phases, specifically using neural networks for AI and human verification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is continuously stored without compression for AI learning and verification, then data quality and accessibility for human verification are maintained, but storage capacity requirements increase significantly
Solution Approach 1:
The system dynamically adjusts compression strategies based on data usage status. Data recently used for AI learning is stored in uncompressed format for immediate accessibility, while data that has aged beyond a threshold is automatically compressed. This dynamic transition between storage states resolves the contradiction by optimizing both data quality and storage efficiency at different time points.
Solution Approach 2:
The system performs preliminary compression on data before it is needed for human verification, rather than compressing on-demand. By identifying data that is no longer required for AI learning and pre-compressing it, the system reduces storage requirements while ensuring that verification-quality data remains accessible when needed.
2Quantity of substance
If compressed data is used for human verification, then storage capacity is reduced, but data processing time and decompression overhead increase
Solution Approach 1:
The storage system is segmented into multiple tiers: hot storage for recently used AI training data (uncompressed, fast access), warm storage for data needing occasional verification (compressed with moderate compression ratio), and cold storage for archival data (heavily compressed). This segmentation allows the system to reduce overall storage capacity while minimizing decompression time for frequently accessed verification data.
Solution Approach 2:
The system changes compression parameters based on data characteristics and access patterns. For data requiring frequent human verification, lower compression ratios are applied to reduce decompression time. For archival data with infrequent access, higher compression ratios are used to maximize storage efficiency. This parameter adaptation resolves the time-capacity tradeoff.
Data Source
AI summary
A data management system including a processor connected to a storage device that stores data usable for learning and inference in a machine learning model, the data management system including a compressor/decompressor for human that compresses and decompresses data to be usable for verification performed by human, in which the processor specifies data that is no longer used for learning and inference in the machine learning model among the data stored in the storage device, and compresses the specified data using the compressor/decompressor for human.