Distributed File Objects for Scalable Concurrent Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file systems in data centers are limited in scalability due to processor bottlenecks from managing storage operations, preventing them from handling widespread frequent data collection and parallel access effectively.
Innovation Solution
Implement a distributed file system using 'smart objects' that distribute operations to responsible entities, allowing for highly parallel access by associating different functions with object types and reducing the need for locks, thus offloading processor cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If general purpose processors manage file system operations, then file system management is achieved, but processor capacity is consumed limiting scalability
Solution Approach 1:
The patent segments file system management operations into distributed operations across multiple storage systems. Each storage system independently manages its own operations through local operators, eliminating the bottleneck of centralized processor management and enabling parallel processing of file system operations across the network.
Solution Approach 2:
The patent introduces smart objects as intermediaries between processors and storage operations. These smart objects encapsulate file system operations and manage themselves without requiring processor intervention, effectively mediating between the need for file system management and the limitation of processor capacity.
2Ease of operation
If processors manage all file system operations, then complete control is achieved, but scaling is limited by processor capacity
Solution Approach 1:
The patent divides file system control into distributed segments where each storage system operates autonomously. This segmentation allows the system to scale by adding more storage systems without proportionally increasing processor management capacity, as each segment manages its own operations independently.
Solution Approach 2:
The patent implements self-service through smart objects that automatically manage their own operations without external processor control. This self-service capability enables storage systems to scale independently, as each unit can autonomously handle file system operations, locking, and coordination without requiring proportional increases in centralized processor capacity.
3Reliability
If locks are used to ensure data consistency, then data integrity is maintained, but access parallelism is reduced
Solution Approach 1:
The patent applies local quality by implementing fine-grained locking at the object level rather than using broad locks. Each smart object manages its own locks independently, allowing concurrent access to different objects without interference. This localized approach maintains data consistency for each object while enabling parallel access across multiple objects simultaneously.
Solution Approach 2:
The patent segments locking mechanisms into independent lock managers associated with each smart object. This segmentation allows locks to be applied selectively to specific objects rather than globally, enabling multiple concurrent operations on different objects while maintaining consistency within each object. The distributed lock management eliminates the single-point bottleneck of centralized locking.
Data Source
AI summary
Techniques are disclosed for providing scalable and concurrent file systems. A backend storage system comprising an interface and a processing unit may be configured to perform the techniques. The interface may present the file system storing objects representative of data. The processing unit may receive, from a frontend host system coupled to the backend storage system, a plurality of client operations to perform with respect to the objects identified by the client operations, and identify an object type associated with each of the identified objects. The processing unit may select, based on the object types, one or more backend operators that implement the plurality of client operations, and apply the backend operators to the identified objects.