Storage Governance via Message Bus Interception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage arrays face increased workload and latency due to the need for multiple accesses for data governance functions like de-duplication, classification, and auditing, which can disrupt primary storage operations and lead to inefficiencies in managing large volumes of data.
Innovation Solution
Implementing a message bus system that intercepts IO operations, enables metadata and event distribution through an out-of-band channel, and uses a governance registry/repository to provide a coherent link for governance functions, minimizing the impact on primary storage by allowing external storage systems to handle these tasks independently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple accesses are performed for data governance functions (de-duplication, classification, auditing), then governance completeness is improved, but storage array latency and workload increase
Solution Approach 1:
The patent extracts data governance functions (de-duplication, classification, auditing) from the primary storage array path. A sidecar process intercepts I/O operations and extracts metadata, sending it to external governance systems via message bus. This separation removes the governance workload from the critical storage path, eliminating latency impact while maintaining complete governance coverage.
Solution Approach 2:
The patent introduces a message bus as an intermediary between the storage array and governance systems. Metadata and events are published to the message bus, which delivers them to external consumers (governance systems). This intermediary enables asynchronous communication, allowing the storage array to complete I/O operations without waiting for governance processing, thus reducing latency while ensuring complete governance execution.
2Reliability
If multiple accesses are performed for data governance functions, then governance accuracy is improved, but storage array workload increases
Solution Approach 1:
The patent extracts the computationally intensive governance workloads (de-duplication analysis, classification algorithms, auditing processing) from the storage array to external sidecar processes and governance systems. The storage array only performs I/O operations and metadata extraction, while external systems handle the heavy processing. This division maintains high governance accuracy through dedicated governance resources while preserving storage array productivity.
Solution Approach 2:
The patent segments the storage system into distinct functional components: the storage array handles core I/O operations, sidecar processes handle metadata extraction and event generation, and external governance systems handle de-duplication, classification, and auditing. This segmentation allows each component to specialize and optimize its function, maintaining high governance accuracy through dedicated processing while keeping storage array workload minimal.
3Productivity
If external storage systems handle governance tasks independently, then storage array performance is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal message bus interface that handles multiple governance functions (de-duplication, classification, auditing) through a single standardized communication channel. External governance systems can independently subscribe to relevant events and processes via the same message bus infrastructure. This universal interface simplifies the overall system architecture compared to implementing separate dedicated interfaces for each governance function, thereby reducing system complexity while maintaining improved storage array performance.
4Productivity
If I/O operations are intercepted and copied to message bus, then governance efficiency is improved, but primary storage path overhead increases
Solution Approach 1:
The patent applies partial action by intercepting only the metadata and event information necessary for governance functions, rather than copying entire data sets. The sidecar process extracts minimal required information (file names, paths, timestamps, operation types) and publishes only this metadata to the message bus. This selective partial copying maintains high governance efficiency while minimizing the overhead on the primary storage path, as the copied data volume is negligible compared to actual I/O operations.
Data Source
AI summary
A computer program product, system, and computer implemented method comprising intercepting, by an interceptor, IO send on an IO path to a storage array; sending a copy of the IO and metadata to a message bus, and enabling a consumer to register with the message bus to consume events sent to the message bus.