Cloud Metrics Binary Storage With Indexed Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of metric data generated by cloud system monitoring overwhelms storage resources and processing systems, leading to performance issues.
Innovation Solution
Transform metric data into compact binary files using field ID and type ID encoding, merge into blocks, and generate indexes for efficient retrieval without scanning the entire dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If metric data is stored in raw format, then data integrity is maintained, but storage space consumption increases significantly
Solution Approach 1:
The patent changes the parameter representation by transforming metric data from human-readable text formats into compact binary representations using field IDs and type IDs. This parameter transformation achieves 50%-60% compression while preserving data integrity through the mapping tables that maintain semantic equivalence between original and compressed formats.
Solution Approach 2:
The patent creates a compressed copy of the metric data by replacing actual field names and values with their corresponding IDs in the mapping tables. This copying approach reduces storage space while maintaining the ability to reconstruct the original data when needed, thus balancing storage efficiency with information preservation.
2Loss of time
If the entire metric data dataset is scanned for retrieval, then complete data access is achieved, but processing time and system performance deteriorate
Solution Approach 1:
The patent performs preliminary action by creating indexes during the data storage phase. These indexes map common field values to their corresponding byte array positions in the binary file, enabling fast retrieval without scanning the entire dataset. The indexing operation is performed once during storage, and subsequent retrievals benefit from this pre-computed structure.
Solution Approach 2:
The patent extracts the retrieval function from the entire data scanning process by using indexes to directly locate and retrieve only the required metric records. This extraction separates the retrieval operation from full dataset scanning, significantly reducing processing time and improving system performance for query operations.
3Quantity of substance
If metric data is transformed into compact binary format, then storage efficiency improves, but data processing complexity increases
Solution Approach 1:
The patent introduces mapping tables as intermediary structures that bridge the original metric data format and the compressed binary format. These mapping tables store the correspondence between field names, types, and their respective IDs, simplifying the processing complexity by providing a reference guide for both compression and decompression operations without requiring complex transformation algorithms.
Data Source
AI summary
Methods, systems, and computer-readable storage media for receiving metric data of a cloud system periodically; transforming the metric data of each type into a byte array using mapping tables, wherein the byte array is an encoded format of the metric data, where each field of the metric data is encoded as a field ID and a field type ID that are short integer variables; merging and storing the byte arrays of multiple metric data into a binary file, wherein the binary file comprises multiple blocks with each block comprising multiple byte arrays; generating indexes for common fields of different metric data in the binary file; receiving a retrieval request requesting metric records including a common field of a particular value; determining storage locations of one or more metric records satisfying the retrieval request; and obtaining the one or more metric records from the binary file using the corresponding storage locations.