Dual Data Store System for Variable Size Object Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud platform database systems face inefficiencies when storing large numbers of data blocks that vary widely in size, as existing data stores are optimized for either small or large files but not both, leading to reduced performance and difficulty in maintaining data synchronization across distributed systems.
Innovation Solution
Implementing a dual-data store system where smaller files are stored in HBase and larger files in HDFS, with metadata management to ensure synchronization, allowing the system to determine the appropriate data store based on file size and update metadata to track successful writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single data store is configured to store data blocks within a certain size range, then storage efficiency for that size range is improved, but performance deteriorates when storing data blocks that vary widely in size
Solution Approach 1:
The patent divides the data storage system into multiple data stores, each optimized for specific size ranges. The first data store handles data blocks within a certain size range, while the second data store handles data blocks of different size ranges. This segmentation allows each data store to maintain high efficiency for its designated size range while the overall system achieves versatility across all size ranges.
2Productivity
If data blocks are stored in a single data store optimized for a certain size range, then storage performance for that range is improved, but performance deteriorates when handling data blocks of varying sizes
Solution Approach 1:
The patent applies local quality by configuring each data store with specific characteristics optimized for its designated data block size range. The first data store is configured with parameters optimized for smaller data blocks, while the second data store is configured with parameters optimized for larger data blocks. This ensures that each local component (data store) has the quality needed for its specific function, thereby maintaining high storage performance across all data block sizes.
3Adaptability or versatility
If multiple data stores are used for different size ranges, then adaptability to various data block sizes is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary component that manages the interaction between multiple data stores. This intermediary handles the routing of data blocks to appropriate data stores based on size ranges, manages synchronization between data stores, and maintains the mapping between data blocks and their storage locations. By centralizing these complex management functions, the system achieves high adaptability while the intermediary abstracts and simplifies the complexity of managing multiple data stores.
Data Source
AI summary
A system may be configured to implement two or more different types of data stores for data storage. For example, the system may include a small object data store and a large object data store. The system may compare the size of a data file to a predetermined threshold size. If the data file is smaller than the predetermined threshold size, the system may write the data to the small object data store. If the data is larger than the predetermined threshold size, the system may write an intent message to the small object data store referencing the data in the large object data store, and may write the data to the large object data store. The system may update the intent message in the small object data store to indicate that the data was successfully written.