Key-Value Storage Schema Integration Reduces Data Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud-based data monitoring systems using generic key-value based storage approaches are inefficient due to the need for duplicating keys for every interval sample, leading to redundant data storage.
Innovation Solution
The implementation of efficient storage techniques that integrate schema into key-value data storage, where metrics data messages are converted into a compact format based on identified schema definitions, removing redundant data and eliminating repeated key names.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generic key-value based storage approach is used, then flexibility and scalability are improved, but storage efficiency deteriorates due to key duplication
Solution Approach 1:
The patent creates a single copy of the schema definition stored externally, and references it multiple times through schema version identifiers in the key-value pairs. Instead of duplicating the entire schema for each data point, the system copies only the necessary version reference, eliminating redundant storage while maintaining the ability to retrieve and validate against the original schema definition.
Solution Approach 2:
The patent introduces a schema definition repository as an intermediary layer between the key-value storage system and the data validation process. This intermediary stores the master schema definitions and provides them to the monitoring system through version references, allowing the key-value system to remain simple and flexible while gaining storage efficiency through centralized schema management.
2Adaptability or versatility
If generic key-value based storage approach is used, then scalability is improved, but storage efficiency deteriorates due to redundant data
Solution Approach 1:
The system stores schema definitions as single external references rather than embedding them with each key-value pair. The schema version identifier acts as a lightweight copy that points to the full schema definition, enabling the system to scale while minimizing redundant data storage across multiple data points and time intervals.
Solution Approach 2:
The schema definition repository serves multiple functions: it stores the master schema definitions, provides version management, enables data validation, and supports retrieval operations. This universal component allows the system to maintain scalability while improving storage efficiency through centralized management of schema information.
3Loss of substance
If schema integration is implemented, then storage efficiency is improved, but data processing complexity increases
Solution Approach 1:
The system performs schema validation and format conversion in advance, before data is stored in the key-value system. By pre-processing the data to ensure it conforms to the schema definition, the system reduces storage complexity while maintaining the benefits of schema integration. The schema version identifier is attached during this preliminary phase, simplifying subsequent retrieval and validation operations.
Solution Approach 2:
The schema definition repository acts as an intermediary that handles the complexity of schema management, validation rules, and format conversions. This external component absorbs the processing complexity, allowing the core key-value storage system to remain simple while still benefiting from schema-based optimization and validation.
Data Source
AI summary
Methods, apparatus, and processor-readable storage media for efficient storage of key-value data with schema integration are provided herein. An example computer-implemented method includes obtaining a metrics data message associated with a product, wherein the metrics data message has a first format and comprises a schema version and a type of the product; identifying one of a plurality of schema definitions for the metrics data message based at least in part on the schema version and the type of the product; converting the metrics data message into a second format based on the identified schema definition, wherein the second format removes at least some redundant data from the metrics data message; and storing the converted metrics data message in a metrics database.