Smart Metric Tracking Object for Distributed Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face challenges in efficiently tracking and reporting real-time metrics across a large number of computing nodes, leading to enormous amounts of data that overwhelm existing storage and retrieval systems, particularly due to the complexity of key-value hashes and indexing schemes.
Innovation Solution
A metric tracking system utilizing a smart tracking object that provides programmatic access to data, abstracting away storage complexities, with detector nodes collecting data, a processing engine decomposing it for storage, and a reporter component generating human or machine-readable reports, while using a metric tracking object to encapsulate the storage system and facilitate efficient data retrieval and manipulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional key-value hashes and indexing schemes are used to store distributed system metrics, then data can be stored and retrieved, but the system becomes overwhelmed by the complexity and volume of data, leading to poor performance and reliability
Solution Approach 1:
The patent segments the storage system by separating raw data storage from processed data storage. Detector nodes collect raw metrics data, which is then processed by a processing engine that transforms it into processed data. This segmentation allows the system to handle large volumes of data by dividing the storage function into multiple components, reducing the complexity of any single storage system while maintaining the ability to store and retrieve data efficiently.
2Reliability
If real-time tracking of metrics across large numbers of computing nodes is implemented, then comprehensive monitoring is achieved, but data processing throughput decreases and system response time increases
Solution Approach 1:
The patent applies preliminary action by having detector nodes continuously collect and buffer raw metrics data before it needs to be processed. The processing engine then processes this pre-collected data in batches, transforming it into processed data that can be stored and retrieved efficiently. This preliminary collection and buffering of data allows the system to maintain comprehensive monitoring while improving processing throughput by avoiding real-time processing of each individual data point.
3Ease of operation
If a smart tracking object is introduced to abstract storage complexities, then ease of operation improves, but device complexity increases due to additional software layers
Solution Approach 1:
The patent introduces a smart tracking object as an intermediary between the data storage system and the applications that need to access the data. This smart tracking object encapsulates the complexity of the underlying storage system, providing a simplified interface for data access. While it does add a software layer, it significantly improves ease of operation by abstracting away the complexities of data storage, retrieval, and processing, allowing applications to interact with the system through a consistent and simple API.
Data Source
AI summary
A method and system for tracking data in a distributed system. A software object calculates, using first data, a location in a storage pool which includes calculating a name of a table at the location in the storage pool. The table includes data requested by a user. Calculating the name of the table includes concatenating a label that identifies a type of data, a group identifier that identifies a group of software objects that includes the software object, and time data that identifies a period of time specified by the user. The software object includes the first data. The first data includes the label, the group identifier, and the time data. The software object connects to the storage pool to retrieve the requested data from the calculated location in the storage pool. The software object sends, to the user, the retrieved requested data.