Lockless Object I/O Controller for High-Density Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed object storage systems face performance bottlenecks due to kernel mode constructs, file system interfaces, and data locking mechanisms, which are not ideal for high-density storage media like NAND flash, leading to inefficiencies in data management and retrieval.
Innovation Solution
Implementing a lockless data plane with direct IO operations, splitting object processing into sub-tasks pinned to specific cores using an efficient zero copy message passing protocol, and employing consistent hashing to direct operations to appropriate storage resources, eliminating the need for data locking and context switching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional kernel mode constructs, file system interfaces and locking mechanisms are used to manage object storage operations, then data consistency and reliability are maintained, but system performance and I/O speed are significantly bottlenecked
Solution Approach 1:
The patent segments object storage operations into independent sub-tasks that can be processed in parallel by multiple worker threads. Each sub-task operates on a specific portion of data with its own lock-free data structures, allowing concurrent access without traditional locking mechanisms. This segmentation enables high-performance I/O operations while maintaining data consistency through structured parallel processing.
Solution Approach 2:
The patent introduces an intermediary layer between the application and storage resources that handles data consistency without requiring traditional file system interfaces or kernel mode constructs. This intermediary uses lock-free data structures and coordinated access patterns to ensure reliability while enabling direct I/O operations that bypass traditional performance bottlenecks.
2Productivity
If high-density storage media like NAND flash are used to improve storage capacity and performance, then storage density and speed are enhanced, but traditional locking mechanisms become unacceptable bottlenecks
Solution Approach 1:
The patent extracts and removes traditional locking mechanisms from the storage I/O path, replacing them with lock-free data structures. This extraction eliminates the overhead that becomes unacceptable when using high-density storage media like NAND flash, while still maintaining data consistency through alternative synchronization methods that do not impede performance.
Solution Approach 2:
The patent substitutes mechanical locking mechanisms with software-based lock-free data structures and coordinated access patterns. This substitution eliminates the performance overhead of traditional locking while maintaining data consistency, enabling high-density storage media to operate at their full potential without the bottleneck of locking overhead.
3Productivity
If object processing is handled by a single thread or core to simplify data management, then implementation is simpler, but parallel processing capabilities and I/O throughput are underutilized
Solution Approach 1:
The patent segments object processing into independent sub-tasks that can be distributed across multiple worker threads and processor cores. Each sub-task is self-contained and can be executed independently, allowing the system to utilize parallel processing capabilities while maintaining manageable complexity through structured task division and lock-free data structures.
4Ease of operation
If context switching is used to handle different I/O operations to improve resource utilization, then resource allocation is more efficient, but processing latency and overhead increase
Solution Approach 1:
The patent enables continuous I/O operations by eliminating context switching through lock-free data structures and pre-allocation of worker threads. Worker threads continuously process I/O operations without being blocked or requiring context switches, maintaining steady-state operation and eliminating the latency and overhead associated with context switching while efficiently utilizing system resources.
Data Source
AI summary
An object node apparatus, system, and method are described. An apparatus can include a lockless-mode controller configured to communicatively couple to a plurality of storage resources and to a plurality of processor cores each preassigned to process a specific type of sub-task at a different preassigned storage resource that is configured to receive object input/output (I/O) only from the preassigned core, the lockless-mode controller being further configured to receive a plurality of object I/O messages from one or more clients, each to perform an object I/O task, divide each object I/O task into a plurality of sub-tasks, identify a specific sub-task type for each sub-task, and send each sub-task for each specific sub-task type to a processor core preassigned to process the specific sub-task type, wherein the sub-tasks include storage operations related to storing sub-object data in, or retrieving sub-object data from, the preassigned storage resource for each processor core.