NoSQL Email Indexing with Virtual Fields
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing email storage systems face inefficiencies in scalability and search performance due to redundant data storage and the computational cost of joining multiple indexes, especially when dealing with large data items like email bodies and attachments, which are copied multiple times and require extensive server resources for searching.
Innovation Solution
A scalable IMAP4 compliant email system using a NoSQL database and a distributed full-text search engine, where data is normalized by storing each email message in multiple tables, and a single index is created based on a metadata table with virtual fields derived from message content, allowing for efficient querying and caching to avoid redundant data conversion and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If email data is stored in individual files with separate metadata database, then email storage and retrieval can be performed, but server resources are consumed excessively during searching operations
Solution Approach 1:
The patent pre-converts and indexes email data during the storage phase, transforming email content into searchable formats and creating indexes before search operations are needed. This preliminary processing eliminates the need for resource-intensive real-time conversion during search operations, significantly reducing server resource consumption when searching emails.
2Ease of operation
If large data items like email bodies and attachments are copied multiple times for different recipients, then each recipient receives their copy, but additional storage space is consumed for each copy
Solution Approach 1:
The patent segments email data into distinct components: a master copy stored once, and separate index records for each recipient that reference the master copy. This segmentation allows the email content to be stored a single time while maintaining the ability to distribute to multiple recipients through references in the index, dramatically reducing storage space requirements.
Solution Approach 2:
Instead of creating physical copies of large email data items for each recipient, the patent creates lightweight index records that reference the master copy. These index records contain only the necessary metadata and references, allowing multiple recipients to access the same email content without duplicating the actual large data items.
3Adaptability or versatility
If indexes are created on multiple tables to enable searching by different fields, then search coverage is improved, but join operations are required which are computationally costly
Solution Approach 1:
The patent merges the indexing functionality into a single index structure that can handle searches across multiple fields without requiring join operations. The index is designed to store and retrieve email records based on any searchable field directly, eliminating the need for separate table indexes and the computationally expensive join operations that would be required to combine results from multiple indexes.
4Speed
If email data is converted and cached for each recipient, then search performance is improved, but redundant data conversion and storage occurs
Solution Approach 1:
The patent performs data conversion and indexing as a preliminary action when the email is initially stored, rather than converting data for each recipient individually. The converted and indexed data is cached in a reusable format that can be quickly accessed for any recipient, eliminating redundant conversion operations and reducing overall processing time.
Solution Approach 2:
The patent creates a universal index structure that serves all recipients simultaneously. The index is designed to be recipient-agnostic, allowing the same indexed data to be used for searching emails for any recipient without requiring separate conversions or caches for each user. This multi-functional index improves search performance while avoiding redundant storage.
Data Source
AI summary
Techniques are herein described for creating a scalable IMAP4 compliant email system using a NoSQL database and a distributed full text search engine. Data for each email message is stored in multiple tables to avoid storing redundant data unnecessarily. However, a full text search index is created based on a single table as if the index refers to a single table. In embodiments herein described, the single index is created on the fields of a message metadata table with virtual fields added to the table that are derived from the message content. During this process, data is pulled from a message table in “blob” format and broken down into corresponding fields and data items, so the data items may be converted and placed in the proper virtual fields for index creation. Each blob section that is converted is cached, so the same blob section does not need to be converted multiple times. After index creation, the index may be used to search for emails based on metadata and data within the body of the email.