Data Object Storage Separating Metadata and Data Fields
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage methods, such as B+ trees and Hash tables, lead to inefficient memory usage due to large data objects occupying significant cache space, resulting in insufficient memory, frequent page replacements, and reduced system performance during big data analysis.
Innovation Solution
A method and device that separate frequently accessed data objects from less relevant data blocks by dividing data into a data field and metadata, storing the data field in one external storage device and metadata and location information in another, allowing only the required data to be loaded into the cache for fast retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data objects are stored using traditional structures (B+ tree, Hash table), then data retrieval is enabled, but cache space is excessively occupied leading to insufficient memory
Solution Approach 1:
The data object is segmented into two parts: a first part (data field) stored in external storage and a second part (metadata) stored in cache. This segmentation allows only the essential metadata to occupy cache space while the actual data resides externally, resolving the contradiction between enabling data retrieval and reducing cache occupancy.
Solution Approach 2:
The data field is extracted from the traditional data object structure and stored separately in external storage. Only the metadata portion remains in the cache, effectively removing the bulk data from the cache and eliminating the space occupancy problem while preserving retrieval capability through metadata references.
2Quantity of substance
If data objects occupy large cache space, then data storage capacity increases, but page replacement frequency increases and CPU time is wasted
Solution Approach 1:
By extracting the data field from the cache and storing it in external storage, the patent eliminates the need for frequent page replacements of large data objects. The cache only holds compact metadata, dramatically reducing page replacement frequency and associated CPU time waste.
Solution Approach 2:
The patent discards the practice of loading entire data objects into cache and recovers cache space by storing only metadata. This allows the cache to be reused more efficiently for different metadata entries, reducing the need for frequent page replacements.
3Speed
If entire data objects are loaded into cache, then data access is fast, but memory becomes insufficient and structure adjustment is frequent
Solution Approach 1:
The patent segments the data object into metadata (in cache) and data field (in external storage). This segmentation enables fast access to metadata for retrieval operations while reducing memory pressure, thereby decreasing the frequency of memory structure adjustments.
Solution Approach 2:
The patent creates a dynamic storage architecture where metadata can be flexibly loaded into cache while data fields remain in external storage. This dynamic approach allows the system to adapt to varying access patterns without requiring frequent restructuring of the entire memory hierarchy.
Data Source
AI summary
A method of storing a data object includes dividing data of a data object into a first part of data and a second part of data, in response to receiving a writing request for a data object, the first part of data including a data field of the data object, the second part of data including a meta data of the data object; writing the first part of data into a first data file stored in a first external storage device; and writing, into a second data file stored in a second external storage device, first location information, and the second part of data, the first location information including location information of the first part of data within the first external storage device.