Distributed Dataset Annotation Without Redundant Data Copying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems require redundant copying of datasets during annotation, leading to increased processing complexity and time, and the removal of original data context, limiting further annotations.
Innovation Solution
A storage layer for distributed systems that allows for the creation and processing of annotation data layers without duplicating the original data, using row identifiers to align annotation datasets with raw datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If datasets are copied to add annotations, then annotation data can be added to the dataset, but the processing complexity and time increase
Solution Approach 1:
The patent merges the original dataset and annotation dataset into a single unified dataset structure. Instead of maintaining separate copies of data, the system combines them with annotation columns added to the original data structure, eliminating redundant data storage and reducing processing complexity while maintaining full annotation capability
Solution Approach 2:
The unified dataset structure serves multiple functions simultaneously: it stores both original data and annotation data, supports multiple types of annotations, and enables both data retrieval and annotation retrieval through a single processing path, thereby reducing overall system complexity
2Adaptability or versatility
If datasets are copied to add annotations, then annotation data can be added, but the processing time increases
Solution Approach 1:
By combining original and annotation data into one dataset, the system eliminates the need to read and process separate files during analysis operations, reducing I/O time and computational overhead while preserving full annotation functionality
3Loss of time
If original data is removed during annotation to reduce processing time, then processing time decreases, but the context of original data is lost
Solution Approach 1:
The unified dataset retains all original data columns alongside annotation columns, ensuring that no contextual information is lost while maintaining efficient single-dataset processing that does not require periodic pruning of original data
4Adaptability or versatility
If data is copied multiple times for multiple annotations, then multiple annotations can be added, but the computational cost increases
Solution Approach 1:
The unified dataset structure is designed to accommodate multiple annotation types and layers within a single data structure. New annotations can be added as additional columns or nested structures without requiring full data duplication, thereby reducing computational cost while supporting multiple annotation capabilities
Data Source
AI summary
Disclosed embodiments are methods, apparatuses, and computer-readable media for annotating distributed data without redundant data copying. In one embodiment, a method is disclosed comprising reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows; generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows; assigning row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows based on the underlying storage of the raw dataset and annotation dataset; and writing the annotation dataset to a distributed storage medium.