Deduplicated Data Search Using Segment Boundary Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems, particularly deduplicated systems, face inefficiencies in search operations due to the large size of indexes and limitations in processing character and binary byte searches, which are not supported by traditional indexing techniques.
Innovation Solution
A microservices architecture is employed to perform text searches on deduplicated data without constructing a large index, utilizing object storage, deduplication service, and search service to segment and fingerprint data, allowing parallel processing of segments and efficient search term matching across segment boundaries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a traditional index is built on client-written objects for searching, then search functionality is provided, but the index becomes enormously large and consumes excessive resources
Solution Approach 1:
The patent segments data into fixed-size chunks and builds indexes only on unique chunks after deduplication, rather than indexing entire client-written objects. This segmentation approach reduces index size by eliminating redundant entries for duplicate data while maintaining search capability on the actual stored data segments.
Solution Approach 2:
The patent extracts and indexes only the unique data segments from deduplicated storage, separating the indexing process from the original object structure. By taking out only the essential unique segments for indexing, the system avoids creating enormously large indexes while preserving search functionality on actual stored data.
2Adaptability or versatility
If traditional indexing techniques are used, then text searches can be performed, but character searches and binary byte searches are not supported
Solution Approach 1:
The patent creates a universal indexing structure that handles multiple search types (text, character, and binary byte searches) through the same mechanism. The fixed-size chunk indexing approach provides a unified solution that works for all search types, eliminating the need for separate indexing strategies for different search functionalities.
3Productivity
If segments are read multiple times due to multiple references, then all referencing objects can be searched, but search time and resource utilization increase
Solution Approach 1:
The patent creates a virtual copy of the index structure that maps multiple object references to unique data segments. Instead of physically reading segments multiple times, the system uses reference tracking and virtual indexing to allow simultaneous search operations on the same underlying data without actual repeated I/O operations.
Data Source
AI summary
A deduplicated storage system storing objects receives a search term. Storage includes metadata and segments into which the objects have been split and deduplicated. Metadata includes fingerprint sequences according to which the segments should be assembled. A partial match is found when a prefix of the term is found at an end of a segment or a suffix is found at a beginning of the segment. The partial matches are recorded and merged to form full matches. Each full match includes a first fingerprint of a first segment in which the prefix was found and a second fingerprint of a second segment in which the suffix was found. A check is made of whether a fingerprint and a next fingerprint from a first sequence of fingerprints associated with a first object have been recorded in the database. If so, the first object is reported as having the search term.