Media Data Object Segmentation for Distributed Storage Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing distributed approach for processing large data objects is inefficient due to the division of data objects into equal or nearly equal portions, which ignores structural features and introduces dependencies between data portions, leading to increased network traffic and inefficiencies.
Innovation Solution
A technique for managing data objects in a storage cluster involves splitting a data object into multiple portions at boundaries within the data object, transforming these portions into segments that provide individually processable units, and distributing these segments among multiple computing nodes for storage and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data objects are divided into equal or nearly equal portions for distributed storage, then storage distribution is simplified, but structural features are ignored and dependencies are introduced between portions
Solution Approach 1:
The patent divides data objects into segments based on their internal structure and access patterns rather than equal portions. Each segment is identified by analyzing the data object's structure, allowing segments to be stored independently while preserving their semantic meaning and eliminating cross-segment dependencies for common operations.
Solution Approach 2:
The patent applies different segmentation strategies to different parts of the data object based on their specific characteristics. Hot data segments that are frequently accessed are separated from cold data segments, and each type is stored and managed with appropriate optimizations for its access pattern.
2Volume of stationary object
If data portions are stored on different computers in a distributed manner, then storage capacity is increased, but network traffic increases due to data gathering and reassembly
Solution Approach 1:
By segmenting data objects into independent, self-contained units with clear boundaries, the patent enables selective retrieval of only the necessary segments for any given operation, avoiding the need to gather and reassemble entire data objects or large portions of them across the network.
3Ease of manufacture
If equal-sized portions are created from data objects, then distribution is simplified, but processing efficiency decreases due to introduced dependencies
Solution Approach 1:
The patent segments data objects into logically independent units based on structural boundaries and access patterns, allowing processing operations to be performed on individual segments without requiring access to other segments, thereby eliminating dependencies and improving parallel processing efficiency.
4Productivity
If segments are made individually processable and distributed across computing nodes, then parallel processing is enabled, but system complexity increases
Solution Approach 1:
The patent creates clearly defined segment boundaries based on data structure and access patterns, which simplifies the management of distributed segments compared to arbitrary divisions. Each segment is self-contained with explicit dependencies documented, making the overall system easier to manage despite the distributed architecture.
Data Source
AI summary
A technique for managing data objects in a storage cluster includes splitting a media data object into multiple portions at boundaries within the media data object. The technique further includes transforming the portions of the media data object into segments that provide individually processable units and distributing the segments among multiple computing nodes of the storage cluster for storage therein.