Metadata-Augmented Dataset Preparation for Production Emulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in accurately emulating production environments during API testing and machine learning training due to the use of raw data that may lose realistic trends, leading to undetected errors and inefficiencies.
Innovation Solution
A method involving the processing of raw datasets to generate index metadata, associating these metadata with the datasets, and merging qualified datasets based on user-defined inputs to create a production dataset, utilizing natural language processing and machine learning models like word2vec and N-gram predictive models to enhance data quality and relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If raw datasets are used directly for API testing and machine learning training, then data processing speed is improved, but data quality and realism are worsened
Solution Approach 1:
The system performs preliminary actions by generating index metadata and augmenting raw datasets before they are used for API testing or machine learning training. This advance preparation ensures data quality and realism are improved while maintaining processing efficiency during actual use.
2Reliability
If metadata is added to raw datasets to improve data quality, then data relevance is improved, but data processing complexity is worsened
Solution Approach 1:
The system introduces index metadata as an intermediary layer between raw datasets and their usage. This metadata acts as a mediator that enhances data relevance and quality without requiring complex processing of the underlying raw data, thus improving reliability while managing complexity.
3Quantity of substance
If multiple raw datasets are merged to increase data volume, then training data quantity is improved, but data accuracy is worsened
Solution Approach 1:
The system performs preliminary examination and qualification of datasets before merging them. By pre-processing and validating datasets using index metadata, the system ensures that only accurate and relevant data are combined, thus increasing training data quantity while maintaining or improving data accuracy.
4Ease of operation
If index metadata is generated and associated with raw datasets, then data organization is improved, but processing time is worsened
Solution Approach 1:
The system generates and associates index metadata with raw datasets in advance, during a preliminary processing phase. This upfront organization effort improves data accessibility and ease of operation during subsequent operations, while the initial time investment is offset by efficiency gains later.
Data Source
AI summary
Methods, computer program products, and systems are presented. The method computer program products, and systems can include, for instance: processing a plurality of raw datasets for generating index metadata and associating one or more index defining the index metadata to respective ones of the plurality raw datasets, wherein the respective ones of the plurality of raw datasets define respective metadata augmented datasets by the associating of the one or more index thereto; examining augmented datasets of the metadata augmented datasets in dependence on metadata of the index metadata and in dependence on user defined input data; and merging first and second ones of the augmented datasets in dependence on the examining augmented datasets, wherein the merging first and second ones of the augmented datasets in dependence on the examining augmented datasets is performed in support of preparing a production dataset.