Near-Memory Allocation Unit State Control for Logging Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing logging storage architectures face challenges in efficiently managing allocation units, particularly in determining when and how to allocate the next unit, ensuring data integrity, and handling client device failures or disconnections during data writing.
Innovation Solution
A near memory processing (NMP) module that automatically manages allocation units by changing their states (ACTIVE, UNINITIALISED, SEALED) based on client commands, allowing seamless transitions and resource management without special client-side code, and a client device maintaining a state map to handle client failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single client is used to send data to the next allocation unit, then the system architecture is simple, but the reliability deteriorates because the client may never return or a new client may appear at any time
Solution Approach 1:
The patent introduces an intermediary mechanism (the storage system's allocation unit management logic) between the client and the allocation units. This intermediary handles the complexity of client management, allocation unit selection, and state tracking, allowing any client to write to any allocation unit without requiring a dedicated client-to-allocation-unit mapping. This resolves the contradiction by adding intermediary logic that improves reliability while managing complexity centrally.
Solution Approach 2:
The storage system performs self-service by automatically tracking which allocation units have been sealed and which are still active, without requiring the client to maintain this state information. The system autonomously determines the next available allocation unit and manages the transitions between states, improving reliability by eliminating client-side state management requirements.
2Reliability
If the system waits for all IOs to the current allocation unit to be completed or failed before writing to the next allocation unit, then data integrity is ensured, but the productivity deteriorates due to waiting time
Solution Approach 1:
The patent implements preliminary action by allowing the system to proactively seal allocation units and transition them to inactive states based on predefined criteria (such as reaching a threshold or being full), rather than waiting for all IOs to complete. This enables the system to prepare the next allocation unit in advance, improving productivity while maintaining data integrity through the sealed state mechanism that ensures all data up to a certain point is valid.
Solution Approach 2:
The system dynamically transitions allocation units between states (active, sealed, inactive) based on real-time conditions such as IO completion status, fill level, and system policies. This dynamic state management allows the system to optimize between data integrity and productivity by flexibly determining when to seal units and when to allow overlapping operations, rather than following a rigid wait-for-completion protocol.
3Adaptability or versatility
If the final size of allocation units is determined dynamically as append payload size is not fixed, then the adaptability is improved, but the device complexity increases due to the need to record and track sizes
Solution Approach 1:
The storage system performs self-service by automatically tracking the size and state of each allocation unit internally without requiring external clients to provide or verify this information. The system autonomously manages the dynamic sizing, recording, and state transitions of allocation units, which improves adaptability while containing complexity within the storage system's management layer rather than distributing it across clients and protocols.
Data Source
AI summary
Provided is a near memory processing (NMP) module for a centralized logging storage array, which includes one or more servers configured to host one or more allocation units assigned to a logical volume, wherein the NMP module is associated with a respective allocation unit. The NMP module is configured to return a state of the respective allocation unit in response to one or more commands received from one or more client devices. The state of the allocation unit includes a mode of the allocation unit. The mode includes one of active, uninitialized, or sealed. When the mode of the allocation unit is uninitialized, the NMP module is configured to change the mode of the respective allocation unit to active in response to receiving an append command from a client device.