Dataset Format Conversion with Link-Based Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data scientists face significant challenges in organizing and preparing large volumes of IoT data for machine learning models due to the lack of a standard mechanism for data organization and cataloging, leading to extensive time spent on data preparation and the creation of multiple data copies during format conversion.
Innovation Solution
A method and system that automatically convert datasets into formats compatible with machine learning frameworks without creating additional copies, by copying data from a capacity storage tier to a performance storage tier and maintaining links to the original data, ensuring the data is in the required format for machine learning or deep learning algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is copied from capacity storage tier to performance storage tier for format conversion, then data preparation time is reduced, but storage space is consumed due to creating additional data copies
Solution Approach 1:
The patent creates a copy of the dataset in the second format in the performance storage tier, while maintaining links to the original data elements in the first format. This allows the system to have both formats available simultaneously without storing complete redundant copies of all data, thus reducing storage space consumption while still enabling fast access for machine learning operations.
Solution Approach 2:
The patent implements a nested structure where the converted dataset in the second format is organized with links pointing to the original data elements in the first format. This nested arrangement allows the system to maintain a compact representation where converted data structures contain references to original data, rather than complete independent copies, thereby optimizing storage efficiency.
2Adaptability or versatility
If multiple copies of data are created during format conversion, then data availability in required formats is improved, but data organization complexity increases
Solution Approach 1:
The patent creates a universal data organization structure where the performance storage tier contains converted datasets in the second format that can serve multiple machine learning frameworks and applications. The link-based structure allows this single converted dataset to fulfill multiple format requirements simultaneously, reducing the need for separate conversion processes for different use cases.
Solution Approach 2:
The patent introduces a data format conversion and linking mechanism that acts as an intermediary between the capacity storage tier (first format) and performance storage tier (second format). This intermediary layer automatically manages the conversion and link creation processes, abstracting away the complexity of format transformations and data organization from users while ensuring data availability in required formats.
3Speed
If complete data copies are created in the performance storage tier, then data access speed is improved, but storage capacity is reduced
Solution Approach 1:
The patent creates selective copies in the performance storage tier - specifically, it converts and stores datasets in the second format with links to original data elements, rather than creating complete independent copies of all data. This selective copying approach enables fast access to converted format data while consuming minimal additional storage capacity, as the link structures require significantly less space than full data duplicates.
Data Source
AI summary
A dataset formatted in a first format is generated, wherein the dataset formatted in the first format includes at least a plurality of data elements that comprise data. A selection is made of machine learning framework, wherein the machine learning framework operates on datasets formatted in a second format that is different from the first format. The dataset is converted from the first format to the second format and links are generated from the converted dataset formatted in the second format to the plurality of data elements of the dataset formatted in the first format to avoid creating additional copies of the plurality of data elements.