Time-Partitioned Data Storage With Bucket Allocation for Higher Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems require global indexing to distinguish between new and updated data, which is limited by traffic and affects throughput performance, leading to low data storage efficiency.
Innovation Solution
A data storage method that determines a target time partition based on data occurrence time and allocates buckets using primary keys, eliminating the need for global indexing, and adjusts bucket numbers based on festival attribute information or storage capacity to improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If global indexing is performed to distinguish new and updated data, then data accuracy is ensured, but throughput performance deteriorates due to traffic limitations
Solution Approach 1:
The patent segments the storage system into multiple time partitions (e.g., daily partitions) with each partition containing multiple buckets. This segmentation eliminates the need for global indexing by confining data organization to local partitions, thereby improving throughput performance while maintaining data accuracy through partition-level indexing.
Solution Approach 2:
The patent performs preliminary actions by pre-defining time partitions and buckets before data ingestion. Each partition is pre-configured with a fixed number of buckets, allowing data to be directly routed to appropriate buckets based on timestamp and partition information without requiring real-time global index operations, thus enhancing throughput.
2Device complexity
If the number of buckets is fixed, then system simplicity is maintained, but adaptability deteriorates when facing varying data storage demands
Solution Approach 1:
The patent introduces dynamic bucket management within each time partition. While each partition maintains a fixed number of buckets for simplicity, the system dynamically determines the appropriate partition based on data occurrence time and dynamically allocates data to specific buckets within that partition using hash functions on primary keys. This provides adaptability to varying data storage demands while preserving system simplicity.
Solution Approach 2:
The patent applies local quality by allowing different time partitions to have different numbers of buckets based on their specific data storage requirements. Each partition can be independently configured with an appropriate bucket count, enabling the system to adapt to local data patterns while maintaining overall system simplicity through modular design.
3Reliability
If too many buckets are created, then data overflow is prevented, but storage space efficiency deteriorates due to excessive remaining space
Solution Approach 1:
The patent segments the storage capacity across multiple time partitions, with each partition containing a manageable number of buckets. This segmentation prevents any single partition from requiring an excessive number of buckets, thereby preventing data overflow while maintaining storage space efficiency through balanced distribution of data across partitions.
Solution Approach 2:
The patent changes the parameter of bucket allocation by determining the number of buckets in each partition based on the data storage capacity of that partition. This adaptive parameter adjustment ensures that each partition has sufficient buckets to prevent overflow while avoiding the creation of excessive empty buckets, thus optimizing storage space efficiency.
Data Source
AI summary
The present disclosure provides a data storage method, an apparatus, an electronic device and a storage medium. The data storage method includes: acquiring at least one piece of feature data to be stored in a target storage system, wherein each piece of the feature data has a corresponding primary key respectively; for each piece of the feature data, determining, from a plurality of first time partitions corresponding to the target storage system, a target time partition to which a data occurrence time of the feature data belongs; and allocating, based on a total number of buckets corresponding to the target time partition and a primary key corresponding to the feature data, a corresponding bucket to the feature data, and storing the feature data in a storage space that is associated with the corresponding bucket of the feature data.