Virtualized File Analytics With Snapshot-Based Event Ordering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for managing enterprise data storage are complex and cumbersome, leading to incomplete analysis of file system usage characteristics and inadequate detection of anomalies due to incomplete catalogs and out-of-order event data processing.
Innovation Solution
A file analytics system that retrieves and organizes metadata and event data from file systems, using snapshot comparisons and distributed protocols to ensure chronological order, filters out ancillary events, and generates real-time reports to detect anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing file server systems are used, then data storage capability is provided, but system complexity increases and interaction becomes cumbersome
Solution Approach 1:
The system divides file server functionality into separate virtual machines (file server VMs) that can operate independently. Each VM manages specific file systems, allowing the complex storage infrastructure to be segmented into manageable, modular units that can be deployed and maintained more easily.
Solution Approach 2:
A centralized management system acts as an intermediary between administrators and the complex file server infrastructure. This management system provides simplified interfaces for deploying file systems, monitoring activity, and detecting anomalies, shielding users from the underlying complexity while enabling effective administration.
2Quantity of substance
If existing file server systems are used, then data storage is provided, but complete analysis of file system usage characteristics cannot be achieved
Solution Approach 1:
The system continuously monitors file server events, metadata changes, and user activities in real-time, maintaining an ongoing analysis of usage patterns. This continuous monitoring ensures that no information is lost and that usage characteristics can be analyzed comprehensively over time to identify trends and anomalies.
Solution Approach 2:
The management system receives feedback from multiple sources including event logs, metadata snapshots, and user activity records. This feedback is processed to generate comprehensive analysis reports that provide complete insights into file system usage characteristics, enabling better decision-making and anomaly detection.
3Quantity of substance
If existing file server systems are used, then data storage is provided, but anomaly detection is inadequate due to incomplete catalogs
Solution Approach 1:
The system performs preliminary actions by creating comprehensive catalogs of files, folders, and metadata before anomalies occur. By maintaining complete and up-to-date inventories of the file system structure and content, the system is better prepared to detect anomalies accurately when they arise, rather than reacting to incomplete information.
Solution Approach 2:
The management system continuously receives feedback from file server events and metadata changes, updating its catalogs in real-time. This feedback mechanism ensures that the catalogs remain complete and current, providing the foundation for reliable anomaly detection by comparing actual behavior against the complete expected state.
4Productivity
If event data is processed without ensuring chronological order, then processing speed increases, but analysis accuracy decreases
Solution Approach 1:
The system performs preliminary sorting and ordering of event data into chronological sequences before analysis begins. By pre-organizing events in the correct temporal order, the system enables accurate analysis while maintaining efficient processing, as the ordering operation is performed once during data ingestion rather than repeatedly during analysis.
Solution Approach 2:
The event processing system segments event data into distinct chronological streams or batches, processing each segment in order while maintaining overall efficiency. This segmentation allows the system to ensure chronological accuracy without requiring complete re-processing of all events, thus balancing speed and precision.
Data Source
AI summary
Examples of file analytics systems are described that may obtain metadata data and events data from a virtualized file server. The metadata may be obtained by scanning one or more snapshots of the virtualized file server. The metadata and event data may be used to report various metrics relating to the virtualized file server.