Distributed Dataset Annotation Using Row-Aligned Data Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems require redundant copying of datasets during annotation, leading to increased processing complexity and time, and the removal of original data context, limiting further annotations.
Innovation Solution
A storage layer for distributed systems that allows for the creation and processing of annotation data layers without duplicating the original data, using row identifiers to align annotation datasets with raw datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If redundant copying of datasets is performed during annotation, then annotation data can be added to original datasets, but processing complexity and time requirements increase
Solution Approach 1:
The patent segments the dataset into original data and annotation data stored in separate locations. Instead of copying entire datasets, the system creates references or pointers to the original data and stores only the annotation information, thereby reducing processing complexity while maintaining annotation capabilities
Solution Approach 2:
The patent introduces a new dimension of data organization by creating a multi-layered data structure where original data and annotation data exist in different layers. This allows the system to access and process annotation data independently without requiring redundant copying of the original dataset
2Adaptability or versatility
If entire datasets are copied and processed multiple times for sequential annotations, then multiple annotations can be added, but computational costs increase
Solution Approach 1:
The patent performs preliminary actions by establishing reference links to the original dataset before annotations are added. This allows subsequent annotations to reference the same original data without requiring repeated copying, thereby reducing computational costs for multiple sequential annotations
Solution Approach 2:
Instead of copying the entire original dataset for each annotation, the patent creates lightweight references or pointers to the original data. This selective copying approach maintains the ability to add multiple annotations while significantly reducing the computational burden of data duplication
3Loss of time
If original data is removed during annotation pruning, then processing time is reduced, but context information is lost
Solution Approach 1:
The patent introduces an intermediary mechanism in the form of reference links or pointers that connect annotation data to the original dataset. This allows the system to work with pruned annotation data for faster processing while maintaining access to the original context information when needed, thus resolving the trade-off between processing time and information retention
Data Source
AI summary
Disclosed embodiments are methods, apparatuses, and computer-readable media for annotating distributed data without redundant data copying. In one embodiment, a method is disclosed comprising reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows; generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows; assigning row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows based on the underlying storage of the raw dataset and annotation dataset; and writing the annotation dataset to a distributed storage medium.