Controller-Worker Search Architecture for Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in searching large datasets, particularly in deduplicated systems, due to the need for extensive indexing and the inability to handle character or binary byte searches effectively, leading to increased time and resource consumption.
Innovation Solution
A microservices architecture is employed to perform text searches on deduplicated data without constructing a reverse index, utilizing segment-based fingerprinting and metadata to efficiently locate search terms across segmented data, allowing for parallel processing and reduced resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If an index is built on client-written objects in a deduplicated storage system, then search efficiency is improved, but the index size becomes enormously large and storage space is wasted
Solution Approach 1:
The patent segments the index into multiple smaller indexes, each corresponding to a specific data chunk or segment in the deduplicated storage system. Instead of creating one massive index covering all data, the system divides the index into manageable pieces that can be stored and processed separately, significantly reducing the volume of any single index while maintaining search efficiency.
Solution Approach 2:
The patent creates local indexes for specific data segments or chunks rather than a global index for all data. Each local index is optimized for its specific segment, allowing efficient searches within that segment while avoiding the need to maintain a large global index. This localizes the indexing function to match the local data structure.
2Adaptability or versatility
If traditional indexing is used to support character or binary byte searches, then search versatility is improved, but the system becomes unable to handle such searches effectively due to index limitations
Solution Approach 1:
The patent implements a dynamic indexing approach where the system can adapt the indexing strategy based on the search requirements. For character or binary byte searches, the system dynamically selects appropriate indexing methods or creates temporary indexes, rather than relying on a static indexing structure that may not support all search types effectively.
Solution Approach 2:
The patent changes the parameters of the indexing system to support different search types. By adjusting indexing parameters such as granularity, data structure format, and organization method, the system can effectively handle character searches, binary byte searches, and other search types without compromising reliability.
3Quantity of substance
If the storage system stores multiple petabytes of data, then data capacity is improved, but the time and resources required to conduct searches increase significantly
Solution Approach 1:
The patent divides large datasets into smaller segments or chunks, each with its own index. When a search is performed, the system only needs to query the relevant segment indexes rather than scanning through all petabytes of data. This segmentation approach maintains constant search time regardless of the total data capacity.
Solution Approach 2:
The patent introduces an additional dimension to the storage architecture by creating a hierarchical indexing structure. Data is organized not just by content but also by spatial or logical dimensions (such as data chunks, segments, or partitions), allowing searches to navigate through multiple dimensions to quickly locate relevant data without examining the entire dataset.
Data Source
AI summary
A controller is provided to manage searching a storage system. A search query including a search term is received. The controller generates a set of tasks associated with conducting a search of the storage system for any objects in the storage system having the search term. These tasks are pushed onto a message queue. A set of workers are allocated to retrieve and process the tasks. A worker processes a task in parallel with another worker processing another task.