Metadata-Driven Data Collection Engine for Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data collection methods in distributed computing systems are inefficient and cumbersome due to hard-coded software agents and drivers, lacking scalability and requiring updates for new data or metrics.
Innovation Solution
Implementing a metadata-driven data collection engine that uses metadata to specify a data collection sequence for querying protocol-based data collectors, allowing flexible and dynamic data collection without the need for software updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hard-coded software agents and drivers are used for data collection, then data collection can be performed for existing components, but the system becomes inefficient, cumbersome, and error-prone when new data or metrics are needed
Solution Approach 1:
The patent segments the data collection mechanism into separate components: metadata definitions, data collection templates, and execution engines. This allows the system to collect new data by adding metadata entries rather than modifying core code, resolving the contradiction between adaptability and complexity
Solution Approach 2:
The patent creates a universal data collection framework that can handle multiple component types and data metrics through a single standardized interface. The metadata-driven approach allows the same execution engine to collect diverse data types, eliminating the need for separate hard-coded agents for each data type
2Productivity
If hard-coded data collection software is used, then data collection functions can be implemented, but the system requires software updates and code changes for every new data or metric requirement
Solution Approach 1:
The patent implements a dynamic data collection system where metadata can be added, modified, or removed without system reconfiguration. The execution engine dynamically loads and executes data collection templates based on metadata definitions, allowing the system to adapt to new requirements without code changes
Solution Approach 2:
The patent uses template-based data collection where standardized collection patterns are defined once and reused through metadata references. New data collection requirements are satisfied by creating metadata entries that reference existing templates, eliminating the need to rewrite collection logic
3Adaptability or versatility
If metadata-driven data collection with sequencing is implemented, then data collection becomes flexible and scalable, but the system complexity increases due to sequence management
Solution Approach 1:
The patent introduces a metadata layer as an intermediary between the data collection requirements and the execution engine. This metadata layer abstracts the complexity of sequence management, allowing users to configure data collection through simple metadata definitions while the execution engine handles the complex sequencing logic
Data Source
AI summary
Methods, systems, and computer readable media for performing metadata-driven data collection are disclosed. In some examples, a method includes receiving a request for system status data for components of a distributed computing system while the distributed computing system is in operation. The request includes metadata specifying a data collection sequence for collecting component-level system status data. The components include compute components, network components, and storage components. The method includes obtaining, using the metadata, the component-level system status data by querying protocol-based data collectors in an order, one after the other, as specified by the data collection sequence specified by the metadata. The method includes assembling the component-level system status data into assembled status data and storing the assembled status data in memory and/or a repository.