Metadata Storage Segmentation for Access Speed and Cost Trade-off
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In synchronization environments, managing metadata storage efficiently is challenging due to the need to balance access frequency, as storing all metadata indefinitely leads to higher costs and lower throughput, especially with large-scale deployments, where most metadata is stale and rarely accessed.
Innovation Solution
Classifying metadata as 'hot' or 'cold' based on access frequency, storing frequently accessed metadata in low-latency storage and less frequently accessed metadata in higher-latency storage, using a service to manage storage transitions and provide a common abstraction interface for clients.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all metadata is stored indefinitely in low-latency storage, then data availability and access speed are improved, but storage costs and resource consumption increase significantly
Solution Approach 1:
The patent segments metadata into two categories: hot metadata (frequently accessed) and cold metadata (infrequently accessed). Hot metadata is stored in low-latency storage while cold metadata is stored in higher-latency storage. This segmentation resolves the contradiction by ensuring fast access for frequently needed metadata while reducing storage costs for rarely accessed metadata.
Solution Approach 2:
The patent applies local quality by assigning different storage characteristics to different portions of metadata based on their access patterns. Frequently accessed metadata receives high-quality low-latency storage, while infrequently accessed metadata is placed in cost-effective higher-latency storage. This resolves the contradiction by optimizing storage quality locally rather than uniformly across all metadata.
2Quantity of substance
If metadata is moved to higher-latency storage to reduce costs, then storage efficiency is improved, but data retrieval time increases
Solution Approach 1:
The patent segments metadata based on access frequency, placing hot metadata in low-latency storage and cold metadata in higher-latency storage. This segmentation ensures that data retrieval time is minimized for frequently accessed metadata while achieving storage efficiency for rarely accessed metadata, resolving the contradiction between storage efficiency and retrieval time.
Solution Approach 2:
The patent performs preliminary classification of metadata into hot and cold categories based on access patterns. This preliminary action enables the system to proactively place metadata in appropriate storage locations before retrieval is needed, ensuring that frequently accessed metadata is already in low-latency storage and minimizing retrieval time when needed.
3Reliability
If metadata is retained indefinitely to support client rejoining, then data completeness is improved, but storage resource consumption increases
Solution Approach 1:
The patent segments retained metadata into hot and cold categories, maintaining data completeness for rejoining clients while optimizing storage resource usage. Recently accessed or frequently needed metadata is retained in low-latency storage, while older, less frequently accessed metadata is moved to higher-latency storage or selectively removed, resolving the contradiction between data completeness and storage resource consumption.
Data Source
AI summary
One or more techniques and/or systems are disclosed for efficiently storing metadata of partitioned data (e.g., directed to a specific individual/user). A first subset of data is identified from the set of partitioned data. The first subset of data comprises data that has a high frequency of access, and may be identified, at least in part, by frequency of access metadata that is associated with the data in the set. Metadata, such as discovery information, permissions, version info, etc., that is associated with the data in the first subset can be stored in low latency (e.g., fast) storage, proving more efficient access. Metadata associated with data from the set of partitioned data that is not in the first subset (e.g., that is not accessed frequently) can be stored in higher latency (e.g., slower, less expensive) storage.