Encrypted Search Index File I/O Layer

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional search indexing methods fail to scale effectively for large databases, as encrypting and decrypting entire search indexes becomes computationally infeasible, leading to performance issues and vulnerability to frequency-based attacks, while also limiting search capabilities to exact term processing and excluding wildcard searches.

Innovation Solution

Implementing an encrypted search index with a file input/output layer that selectively encrypts and decrypts only portions of the search index files, maintaining the internal structure and enabling exact term and wildcard searches, thereby protecting sensitive customer data and improving search performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the entirety of a large search index is encrypted and decrypted for search operations, then data security is improved, but search performance deteriorates significantly due to computational overhead

Engineering Contradiction:

Improvedata securityVSAvoidsearch performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The search index is divided into multiple segments or chunks, where only the necessary segments are decrypted for search operations rather than the entire index. This segmentation allows selective decryption of minimal data portions, maintaining security while improving performance.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different portions of the search index have different encryption states - hot/data-intensive portions are decrypted for fast access while cold/less-accessed portions remain encrypted. This local quality differentiation optimizes the balance between security and performance based on access patterns.

Inventive Principle:

Principle #3Local quality

2Reliability

If the entirety of a search index is encrypted, then data security is improved, but the system becomes vulnerable to frequency-based attacks

Engineering Contradiction:

Improvedata securityVSAvoidfrequency-based attacks

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

Instead of encrypting the entire index uniformly, the system inverts the approach by keeping frequently accessed portions decrypted and only encrypting less-accessed portions. This inversion makes frequency-based attacks ineffective as the encryption pattern no longer directly correlates with access frequency.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The encryption state of index portions is dynamic rather than static - portions are decrypted when needed and re-encrypted when not in use. This dynamic approach prevents attackers from using static frequency analysis on encrypted data.

Inventive Principle:

Principle #15Dynamics

3Adaptability or versatility

If the entirety of a search index is decrypted for search operations, then search functionality is improved, but computational overhead increases significantly

Engineering Contradiction:

Improvesearch functionalityVSAvoidcomputational overhead

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The system applies partial decryption - only decrypting the specific portions of the search index that are needed for the current search operation rather than decrypting the entire index. This partial action reduces computational overhead while maintaining full search functionality for the required data.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

Frequently accessed index portions are pre-decrypted and kept in memory for fast access, while less-frequently accessed portions remain encrypted on disk. This preliminary action reduces the computational overhead during routine search operations.

Inventive Principle:

Principle #10Preliminary action

4Reliability

If conventional encryption is applied to the entire search index, then data protection is improved, but wildcard searching capability is lost

Engineering Contradiction:

Improvedata protectionVSAvoidwildcard searching capability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The index is segmented into encrypted and unencrypted portions, where unencrypted portions maintain wildcard search capability while encrypted portions provide data protection. This segmentation allows the system to support both security requirements and advanced search functionality.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

An intermediary layer or index structure is introduced that allows wildcard searches to be performed on unencrypted metadata or index structures while the actual data remains encrypted. This intermediary enables versatile searching without compromising data protection.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10025951B2Systems and methods for implementing an encrypted search index

Publication Date: 2018.07.17 SALESFORCE INC

AI summary

An encrypted search index is disclosed. For instance, an exemplary system may include a search index stored on disk with customer information stored therein, the search index files having a term dictionary or a term index type file having internal structure which allows a portion of the individual search index file to be updated, encrypted, and/or decrypted without affecting the internal structure of the individual search index file; a file input/output (IO) layer to encrypt the customer information being written into the individual search index file and to decrypt the customer information being read from the individual search index file; and a query interface to execute the operation against the customer information stored in the memory in its decrypted form.