Hadoop File Indexing via Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for transferring large file indexes to search engines, such as Solr, are inefficient due to high overhead and network unreliability, especially when dealing with large files like those of one petabyte size, leading to prolonged transfer times and increased communication costs.
Innovation Solution
The method involves segmenting a file into multiple smaller segments and indexing each segment using a distributed file system like HDFS, with multiple nodes generating and storing indices, which are then transferred using a more efficient protocol like SSH, reducing the size of each index and minimizing retransmission in case of network failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If HTTP protocol is used to transfer large file indexes to search engine, then data can be transferred over network, but transfer time is very large and overhead is high
Solution Approach 1:
The patent divides a large file into multiple segments and indexes each segment separately. This segmentation allows parallel processing and reduces the size of each index transfer, thereby reducing overall transfer time and network overhead while maintaining reliability through distributed handling of index segments.
2Reliability
If entire index is transferred over network, then search engine can access file content, but network failure requires complete retransmission
Solution Approach 1:
The patent segments the file and its index into multiple independent parts that can be transferred separately. If network failure occurs, only the failed segment needs retransmission rather than the entire index, improving both reliability and productivity by reducing retransmission overhead.
Solution Approach 2:
The patent performs preliminary indexing of file segments locally before transfer. This preliminary action creates ready-to-transfer index segments that can be independently transmitted, reducing the impact of network failures and improving overall transfer efficiency.
3Loss of information
If file is indexed as single unit, then complete index is available, but transfer overhead and time are very large
Solution Approach 1:
The patent divides the file into segments and creates indexes for each segment independently. These segmented indexes are then transferred in parallel, reducing total transfer time while maintaining complete indexing coverage through aggregation of segment indexes.
Solution Approach 2:
The patent merges multiple segment indexes into a complete index structure at the search engine. This merging process reconstructs the full indexing capability from distributed segment indexes, ensuring information completeness while benefiting from reduced transfer times.
4Ease of operation
If conventional indexing method is used, then file content can be searched, but communication costs and network overhead are high
Solution Approach 1:
The patent segments the indexing process to allow distributed processing and transfer of smaller index units. This reduces network overhead and communication costs while maintaining full search functionality through aggregation of segment indexes at the search engine.
Data Source
AI summary
A file indexing system for indexing a file to be stored onto a distributed file system includes a segmentation module to segment the file into a plurality of segments. The file indexing system further includes an index generation module to initiate indexing of the file through a plurality of nodes of a Hadoop cluster, where each of the plurality of nodes indexes one or more segments from amongst the plurality of segments to generate at least one index corresponding to the one or more segments. The file indexing system further includes an index transfer module to store the at least one index onto the distributed file system.