Lightweight Event Server for Data Management Overhead Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Hierarchical Storage Management (HSM) systems using Data Management API (DMAPI) face high system overhead, leading to performance impacts and single-point failures, requiring multiple redundant DMAPI servers for fault tolerance, which increases costs and restricts data transfer between clusters due to limitations in managing remote filesystems.
Innovation Solution
Implementing a Light Weight Events (LWE) server to replace or supplement DMAPI operations, enabling efficient data migration and recall using lightweight events that intercept and process requests without significant performance decrease, allowing for parallel processing and true scale-out file recall, and supporting multiple endpoints per filesystem.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If DMAPI is used to migrate/recall data between storage devices, then data management functionality is provided, but system overhead increases and performance decreases
Solution Approach 1:
The patent creates a lightweight copy of the DMAPI functionality implemented within the filesystem kernel module. Instead of using the full DMAPI system, the filesystem implements essential data migration and recall capabilities directly, capturing only the necessary control messages to perform HSM operations. This copying approach provides the required data management functionality while avoiding the high overhead of the complete DMAPI system.
Solution Approach 2:
The patent extracts only the essential HSM functionality from the DMAPI system and implements it directly within the filesystem. By taking out just the critical data migration and recall operations and embedding them in the filesystem kernel module, the system achieves the necessary data management capabilities without the performance penalty of the full DMAPI infrastructure.
2Reliability
If multiple redundant DMAPI servers are provided for fault tolerance, then system reliability improves, but system cost and complexity increase
Solution Approach 1:
The patent merges the HSM functionality directly into the filesystem kernel module, combining what were previously separate components (filesystem and HSM management) into a unified implementation. This integration eliminates the need for separate DMAPI servers and their associated redundancy infrastructure, achieving fault tolerance through the embedded architecture rather than through multiple external servers.
Solution Approach 2:
The filesystem becomes self-sufficient by implementing HSM capabilities directly within its kernel module. Instead of relying on external DMAPI servers for data migration and recall operations, the filesystem serves its own HSM needs through the embedded lightweight implementation, eliminating the requirement for redundant external server infrastructure.
3Adaptability or versatility
If DMAPI system is used for remote filesystem management, then data transfer between clusters is enabled, but system overhead and processing time increase
Solution Approach 1:
The patent prepares the filesystem in advance by embedding the HSM functionality directly within the kernel module before runtime operations are needed. This preliminary integration ensures that data migration and recall operations can execute immediately using pre-positioned capabilities, eliminating the startup and communication overhead associated with remote DMAPI systems when processing time is critical.
4Adaptability or versatility
If DMAPI is used for local storage device migration, then data can be moved between devices, but the overhead is higher than native filesystem operations
Solution Approach 1:
The patent creates a universal implementation of HSM functionality that works for both local and remote storage operations through the embedded filesystem module. By making the HSM capabilities part of the filesystem itself rather than a separate system, the same lightweight mechanism handles diverse data migration scenarios without incurring the overhead of external DMAPI processing for each operation.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
A data management system coupled to a filesystem (16) and at least one data storage target (17), the system comprising:a processor configured to execute instructions;a memory storing instructions which, when executed by the processor, cause the processor to:receive a plurality of operation instructions from the filesystem(16); generate a plurality of light weight events, each light weight event corresponding to an operation instruction of the plurality of operation instructions; process each light weight event by enacting the operation instruction which corresponds to the light weight event to perform an operation on data stored by the at least one data storage target (17); and provide at least one response notification to the filesystem (16), the at least one response notification indicating a processing status for one or more of the light weight events such that the data management system can monitor each operation as it is performed on data stored by the at least one data storage target (17).