Hierarchical Cache Mail Indexing Reduces Disk IO
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mailbox search systems face inefficiencies and high storage costs due to the need for frequent updates of inverted index files as mail data scales, leading to increased hard disk IO resources and storage requirements.
Innovation Solution
Implementing a hierarchical caching system with multiple levels of caches, where inverted index records are initially stored in low-latency memory (Level 1 cache), then transferred to higher-latency storage (Level 2 and Level 3 caches) based on preset threshold values, reducing the frequency of hard disk operations and optimizing storage usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If inverted index files are updated frequently to ensure search completeness when new mail is added, then search result completeness is improved, but hard disk IO resources and storage costs increase significantly
Solution Approach 1:
The patent segments the inverted index system into multiple cache levels (L1 cache in memory, L2 and L3 caches on disk). This segmentation allows frequently accessed index data to be stored in fast memory while less frequently accessed data resides on disk, reducing hard disk IO operations while maintaining search completeness.
Solution Approach 2:
The patent introduces a hierarchical dimension to the storage system by creating multiple levels of caches with different access speeds and capacities. This multi-dimensional storage architecture enables the system to balance between search completeness and IO resource usage by strategically placing data at appropriate cache levels.
2Quantity of substance
If the scale of mail data expands to accommodate hundreds of millions of subscribers and billions of mail messages, then system capacity is improved, but storage costs and hard disk IO resources become prohibitively high
Solution Approach 1:
The patent applies local quality by making different parts of the storage system have different characteristics - L1 cache provides fast access for hot data, while L2 and L3 caches provide capacity for cold data. This allows the system to handle large-scale mail data efficiently by optimizing access patterns for different data subsets.
Solution Approach 2:
The patent changes the physical state and access parameters of stored data by implementing a hierarchical cache structure. Data transitions between different cache levels based on access frequency and importance, changing its effective access speed and storage location parameters to optimize both capacity and cost efficiency.
3Reliability
If keyword searches are performed on entire inverted index files to ensure complete search results, then search completeness is improved, but search speed decreases due to large file sizes
Solution Approach 1:
The patent implements preliminary action by pre-loading frequently accessed inverted index records into L1 cache memory before they are needed for search operations. This preliminary placement of hot data in fast memory ensures that common search operations can be completed quickly without scanning entire index files, while still maintaining complete search results.
Data Source
AI summary
Electronic mail message processing includes: obtaining a set of keywords associated with an electronic mail message; updating, based at least in part on the set of keywords, a set of inverted index records stored in a level 1 cache; determining whether size of the set of inverted index records stored in the level 1 cache exceeds a first preset threshold value; in the event that the first preset threshold value is exceeded, transferring the set of inverted index records in the level 1 cache to a level 2 cache; determining whether size of a level 2 cache file exceeds a second preset threshold value; in the event that the second preset threshold value is exceeded, transferring, according to a path file, inverted index records in the level 2 cache file to a level 3 cache storing a set of inverted index files.