Centralized Search Indexing for Shared Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file searching methods in shared storage systems are resource-intensive and fail to provide real-time updates, as they require each client device to crawl and index the shared file system, leading to inefficiencies and delays in capturing recently added or modified files.
Innovation Solution
A centralized search engine with an event monitoring service that maintains incremental indexes, allowing for efficient resource utilization by indexing the shared storage system once and updating indexes in real-time through incremental updates, eliminating the need for repeated crawling and ensuring up-to-date file information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a centralized search engine performs full crawl and incremental crawls to generate and maintain a central index, then the index can be maintained centrally, but the crawling operations are resource intensive and the index is not updated fast enough to capture recently added or modified files
Solution Approach 1:
The system performs a full crawl of the shared file system in advance to build an initial central index before users need to search. This preliminary indexing action ensures that the majority of files are already indexed and searchable, while the resource-intensive crawling operation is completed beforehand rather than continuously
Solution Approach 2:
The system implements incremental crawls that periodically update the central index with newly added or modified files. Instead of continuous crawling, the search engine performs periodic scans to detect changes and update the index, reducing resource consumption while maintaining relatively current search capabilities
2Ease of operation
If each client device builds a local index by crawling the shared storage file system, then each device can search independently, but the crawling and indexing operations become a burden on the file system, particularly when many users access the same shared storage
Solution Approach 1:
The system merges the indexing function into a centralized search engine that runs on the shared storage server rather than distributing indexing operations across multiple client devices. This consolidation eliminates redundant crawling of the same file system by multiple clients and reduces overall resource consumption on the shared storage
Solution Approach 2:
The centralized search engine serves as a universal indexing service for all client devices accessing the shared storage. Instead of each client performing its own indexing, the single centralized engine provides indexing services to all users, making the system more efficient while still enabling independent searching capabilities for each client
Data Source
AI summary
A method of searching shared files stored on a general purpose shared storage includes receiving a search term in a native search interface of an operating system on a computing device; and performing, on the computing device, a search based on the search term including accessing a search functionality of the general purpose shared storage, wherein the search functionality uses indexes of the shared files stored on the shared storage and updates the indexes of the shared files in response to update events generated from the shared files. In one embodiment, the method performs a federated search based on the search term.