Shift-Varying Predicate Function for Data Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data segmentation techniques, particularly those using shift-invariant predicate functions, face inefficiencies due to the lack of segment boundary identification in certain data inputs and skewed distribution of segment sizes, leading to inefficiencies in communications and storage applications.
Innovation Solution
The implementation of shift-varying segmentation, where the predicate function evaluates input data within a sliding window and outputs a segment boundary based on both the input data and the window's position relative to previous boundaries, ensuring consistent segmentation across different data locations and optimizing segment size distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If shift-invariant predicate function is used for data segmentation, then consistent segmentation is achieved regardless of data location, but segment boundaries are not generated for certain continuous byte patterns and segment sizes become skewed
Solution Approach 1:
The patent transforms the static shift-invariant predicate function into a dynamic shift-varying predicate function that adapts its behavior based on the current position in the data stream. The predicate function now takes into account the distance from the previous segment boundary, allowing it to dynamically adjust its segmentation decisions. This resolves the contradiction by enabling the system to handle continuous byte patterns (improving productivity) while maintaining consistent segmentation through the positional awareness (preserving stability).
Solution Approach 2:
The patent changes the parameters of the predicate function from being purely data-dependent to being both data-dependent and position-dependent. By introducing the position parameter (distance from previous boundary), the function's output varies based on both the input data and its location in the stream. This parameter change enables the system to generate segment boundaries for previously problematic continuous patterns while maintaining overall segmentation consistency through the positional context.
2Reliability
If upper bound on segment length is imposed to prevent infinite segments, then segment boundaries are forced at regular intervals, but segment size distribution becomes skewed and inefficient
Solution Approach 1:
The patent replaces the static upper bound constraint with a dynamic segment length control mechanism. Instead of forcing boundaries at fixed intervals regardless of data content, the shift-varying predicate function dynamically determines boundary positions based on both data patterns and positional context. This allows the system to maintain reliable boundary generation (preventing infinite segments) while achieving more natural and efficient segment size distributions that adapt to the actual data characteristics.
Solution Approach 2:
The patent changes the constraint parameter from a fixed upper bound to a dynamic target segment length that varies based on positional information. The predicate function uses the distance from the previous boundary as a parameter to modulate its behavior, allowing segment lengths to naturally vary around a target value rather than being hard-capped. This produces more efficient segment size distributions while maintaining reliability in boundary generation.
3Stability of the object's composition
If shift-invariant predicate function is used, then same data is segmented identically regardless of position, but redundant data identification is reduced and communication efficiency decreases
Solution Approach 1:
The patent introduces dynamic positional awareness into the segmentation process while maintaining reproducibility. The shift-varying predicate function uses the position-dependent parameter (distance from previous boundary) to make segmentation decisions that are reproducible for the same data stream but adaptable to different positions. This enables better identification of redundant data patterns that occur at different locations, improving communication efficiency while maintaining consistent behavior for identical inputs.
Solution Approach 2:
The patent adds a positional parameter to the predicate function that enables it to distinguish between identical data patterns occurring at different positions in the stream. This parameter change allows the function to identify redundant data more effectively by considering both the data content and its positional context, thereby reducing information loss while maintaining segmentation reproducibility through the deterministic nature of the enhanced function.
Data Source
AI summary
Shift-varying segmentation uses a shift-varying predicate function to evaluate input data within a sliding window to determine if the current sliding window position should be a segment boundary. The shift-varying predicate function is a function of both the input data within the sliding window and the position of the sliding window relative to a previous segment boundary or the beginning of the input data. The shift-varying predicate function includes a containment property and may compute a hash value from the input data in the sliding window. The hash value is compared to a threshold value that is a function of the sliding window position. As the sliding window position advances away from a previous segment boundary, the probability of a segment boundary at the window position increases. Shift-varying segmentation can be used in compression, transaction acceleration, and data storage applications.