Two Phase Data Storage Using Flattened Index and Inventory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional NoSQL distributed database management systems face challenges in efficiently storing and retrieving large amounts of data from datacenters, particularly in handling the addition of new components, which requires significant redesign and consumes excessive computing resources.
Innovation Solution
A method involving synchronous storage of metric data in a flattened format in an index and asynchronous storage in an unflattened format in an inventory, using a Resource Description Framework (RDF) model, with slot identification generated by hashing, to facilitate efficient storage and retrieval while allowing for flexible onboarding of new components without extensive redesign.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If NoSQL distributed database management systems are used to store metric data, then data storage capacity increases, but database redesign is required every time new components are added, increasing human resource costs and system complexity
Solution Approach 1:
The database is segmented into multiple independent components: a schema-agnostic storage layer that handles raw metric data without requiring structural changes, and a query processing layer that handles data access patterns. This segmentation allows new components to be added to the storage layer without triggering database-wide redesigns, as each segment operates independently
Solution Approach 2:
The database system implements a universal data model that can accommodate multiple data types and component structures through a common storage format. The schema-agnostic design allows the same storage infrastructure to handle diverse metric data from various component types without requiring component-specific database schemas or redesigns
2Quantity of substance
If conventional database systems store large amounts of metric data, then data retention capacity increases, but computing resources such as cache memory are excessively consumed
Solution Approach 1:
The patent extracts frequently accessed metric data from persistent storage into an in-memory cache structure that organizes data by component hierarchy. This extraction reduces the amount of data that needs to be retrieved from disk storage, thereby reducing computing resource consumption while maintaining high data retention capacity in the underlying storage system
Solution Approach 2:
The system implements local quality optimization by caching only the metric data that is frequently accessed or recently modified, while leaving less frequently accessed data in persistent storage. This selective caching approach minimizes computing resource consumption by focusing memory resources on the most critical data subsets
3Reliability
If metric data is stored in a normalized relational format, then data integrity is maintained, but storage efficiency decreases and retrieval time increases for large datasets
Solution Approach 1:
The system performs preliminary action by pre-computing and storing aggregated metric data at multiple hierarchical levels (component, rack, aisle, datacenter). When queries are executed, the system can retrieve pre-aggregated results instead of computing them in real-time from raw data, significantly reducing retrieval time while maintaining data integrity through verified aggregation processes
Data Source
AI summary
Method and systems for data storage is provided. Metric data corresponding to a component of a datacenter is received, the metric data associated with a metric instance that identifies the component of the datacenter. It is determined that the metric instance is not stored at an index server. The metric instance is stored at the index server, the metric instance being synchronously stored in a flattened format. A slot identification is generated based on at least a portion of the metric instance, and the metric instance is stored at an inventory server in accordance with the slot identification, the metric instance being asynchronously stored in an unflattened format.