Content-Based Data Segmentation for Incremental Model Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data segmentation methods generate new segments with unique identifiers every time data is refreshed, requiring regeneration of machine learning artifacts, leading to inefficient and redundant processing.
Innovation Solution
Generate content-based segment identifiers that are unique to the underlying data content, allowing comparison across segmentation runs to identify unchanged segments and incrementally train the machine learning model only on modified segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional segmentation methods generate new UUIDs for all segments during each segmentation run, then segment identification is simple, but machine learning artifacts must be regenerated for all segments leading to redundant processing
Solution Approach 1:
The patent segments the identification space into two parts: content-based identification (using segment keys and values) and position-based identification (using UUIDs). This allows the system to identify unchanged segments through content comparison while maintaining unique identification through UUIDs, thereby avoiding redundant regeneration of machine learning artifacts for unchanged segments.
Solution Approach 2:
The patent changes the parameter used for segment identification from purely position-based (UUID) to a combination of content-based (segment keys and values) and position-based (UUID) parameters. This enables the system to detect unchanged segments through content parameter comparison, allowing selective updates rather than full regeneration of machine learning artifacts.
2Loss of time
If incremental training is implemented to train only on modified segments, then processing time is reduced, but the system must compare segment identifiers across multiple segmentation runs increasing complexity
Solution Approach 1:
The patent creates a content-based representation (copy) of segment identification using segment keys and values. This content-based copy serves as a fingerprint that can be efficiently compared across segmentation runs to identify unchanged segments, simplifying the comparison process while enabling incremental training.
3Loss of energy
If content-based segment identifiers are generated to enable incremental training, then redundant computations are reduced, but memory usage increases due to storing and comparing content attributes
Solution Approach 1:
The patent segments the identification process into content-based identification (using segment keys and values) and position-based identification (using UUIDs). This segmentation allows the system to use compact content-based identifiers for detecting changes while relying on UUIDs for unique identification, optimizing the balance between computational efficiency and memory usage.
Data Source
AI summary
A system and method are provided relating to segmentation of data sets. A processor may be configured to generate, for each of a plurality of data segments in first and second segmentation runs, a content-based segment identifier. The processor may be configured to identify, by the processor, a set of modified data segments between a first segmentation run and a second segmentation run by comparing the content-based segment identifiers for the first plurality of data segments with the content-based segment identifiers for the second plurality of data segments. The processor may be configured to incrementally train the machine learning model for only the set of modified data segments.