Dataset Annotation Mapping for Traceable Data Harmonization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in harmonizing heterogeneous data for efficient training of artificial intelligence models, particularly due to the cumbersome nature of manual annotation and the loss of traceability in data alteration.
Innovation Solution
A data management system with a storing unit comprising a dataset memory, metadata memory, and annotation memory, along with a processing unit that automatically maps dataset annotations onto predetermined annotation values, maintaining traceability and enabling flexible harmonization of datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual annotation is used to harmonize heterogeneous data, then data harmonization can be achieved, but the process becomes very cumbersome and time-consuming
Solution Approach 1:
The system enables self-service automated annotation by allowing the computer learning system to automatically generate annotations for training data using pre-defined annotation schemes, eliminating the need for manual human annotation while maintaining data harmonization quality
Solution Approach 2:
The patent replaces the mechanical manual annotation process with an automated computational system that uses algorithms and pre-defined annotation schemes to generate annotations automatically, substituting human labor with machine-based processing
2Manufacturing precision
If manual annotation is used to harmonize data, then learning criteria can be established, but traceability of original data is lost due to data alteration
Solution Approach 1:
The system segments the annotation process by separating the original data storage from the annotation generation, storing original data unchanged in a data repository while generating annotations separately through automated processing of selected portions, thereby preserving traceability
Solution Approach 2:
The system creates copies of portions of original data for annotation purposes while leaving the original data unchanged, allowing annotations to be generated from copies while the originals remain intact and traceable in the repository
3Quantity of substance
If large amounts of heterogeneous data are stored for training, then model training capability is enhanced, but data access and browsing efficiency decreases
Solution Approach 1:
The system implements a universal annotation scheme that can be applied across multiple data types and sources, creating a common framework that enables efficient access and browsing of heterogeneous data while maintaining the ability to store large volumes of training data
Solution Approach 2:
The system changes the parameter of data organization by introducing structured annotations with specific properties and values that enable efficient indexing, filtering, and retrieval of training data based on learning criteria without limiting data volume
Data Source
AI summary
The invention relates to a data management system including a storing unit and a processing unit. The storing unit includes a dataset memory that stores at least one dataset; a metadata memory that stores, for each dataset stored in the dataset memory, respective metadata; and an annotation memory that stores at least one predetermined annotation value. The processing unit, upon storing of an additional dataset in the dataset memory, determines whether the additional dataset includes dataset annotations descriptive of the data comprised in said additional dataset; performs a mapping of each dataset annotation onto the at least one predetermined annotation value; and writes, in the metadata memory, in relation to the additional dataset, metadata representative of a result of the mapping.