Object Storage File Consolidation to Reduce Management Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Object storage systems face challenges with file size limits, leading to issues in file system policies and data migration due to the need to upload large objects in multiple segments, which complicates data management within the system.
Innovation Solution
A method to identify and consolidate multiple segment files within an object storage system by determining all data blocks associated with these files and mapping them to a single new file, allowing for streamlined management and operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large objects are uploaded using multipart upload technique to overcome size limits, then the size limit constraint is satisfied, but the system complexity increases due to multiple segment files
Solution Approach 1:
The patent consolidates multiple segment files into a single unified object by mapping all data blocks from the segment files to one new file. This merging process eliminates the complexity of managing multiple separate files while preserving the ability to store large objects that exceeded the original size limits.
2Quantity of substance
If multiple segment files are used to store large objects, then size limits are overcome, but file system policy enforcement becomes difficult
Solution Approach 1:
By consolidating segment files into a single file representation, the system restores the ability to apply file system policies uniformly to the entire object. The single new file created from mapped data blocks becomes the target for policy enforcement, eliminating the complexity of applying policies to multiple separate segments.
3Quantity of substance
If data is stored as multiple segment files, then large objects can be stored, but data migration becomes complicated
Solution Approach 1:
The consolidation process creates a single new file that represents the entire object, simplifying data migration operations. Instead of migrating multiple segment files separately, the system can migrate the single consolidated file, reducing the complexity of tracking and managing multiple data pieces during migration processes.
Data Source
AI summary
A computer-implemented method according to one embodiment includes identifying a plurality of segment files within an object storage system, determining all data blocks associated with the plurality of segment files within the object storage system, and mapping all the data blocks associated with the plurality of segment files to a single new file within the object storage system.