ML Data Flag Management for Storage Cost Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning data management systems do not effectively determine the necessity of input and output data for subsequent machine learning processes, leading to increased running costs due to accumulated unnecessary data throughout the model's life cycle.
Innovation Solution
A data management system and method that assigns flags to input and output data based on their involvement in machine learning processes, allowing for determination of data necessity, thereby enabling efficient deletion of unnecessary data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is accumulated throughout the machine learning life cycle to maintain model accuracy, then the model can be retrained with more data, but the running cost of the system increases
Solution Approach 1:
The patent changes the parameter of data retention by introducing flags that dynamically mark data based on its utility to future machine learning processes. This allows the system to transition from a static accumulation approach to a dynamic selection approach, where only data marked with appropriate flags is retained, thereby reducing storage costs while maintaining model accuracy through selective data preservation
Solution Approach 2:
The patent implements a mechanism to discard unnecessary data by removing it from storage after it has served its purpose in the machine learning life cycle. The flag management system identifies data that is no longer needed for retraining or validation, enabling systematic discarding of such data to reduce running costs while preserving data that may be recovered and reused in future training iterations
2Adaptability or versatility
If all input and output data are stored for potential future use, then data availability for retraining is maximized, but storage requirements and management complexity increase
Solution Approach 1:
The patent segments the data management function by introducing a dedicated flag management information table and flag assignment mechanism. This separates the concern of data utility assessment from data storage and retrieval operations, reducing management complexity by creating a distinct layer that tracks data relevance without requiring complex interdependencies in the storage system
Solution Approach 2:
The patent introduces flags as an intermediary between the data and the management system. These flags serve as a simple metadata layer that mediates the relationship between stored data and future retraining needs, eliminating the need for complex analysis algorithms to determine data utility while maintaining high data availability through straightforward flag-based queries
Data Source
AI summary
In a data management system of a machine learning model, flag management information (a flag importance management table) manages and defines respective flags corresponding to, of a plurality of processes included in the life cycle, one or more predetermined processes. An operation unit assigns flags defined in the flag management information to input data and output data of the model in accordance with involvement in the predetermined processes when the model is operated. A data management unit determines, with respect to each of the input data and the output data, the necessity of storage of data on the basis of a flag assigned to the data by the operation unit.