Document Indexing via Cluster Computing for Real-Time Data Freshness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional document indexing processes are time-consuming and often performed as batch jobs, leading to stale or outdated information being used in systems between batch processing cycles.
Innovation Solution
Implementing a cluster computing platform that distributes document indexing tasks across multiple processing devices, allowing for iterative processing of documents and reducing the time required to complete the indexing process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch processing is used for document indexing, then system resource usage is reduced, but indexing speed and data freshness deteriorate
Solution Approach 1:
The patent divides the document indexing task into multiple independent sub-tasks that can be processed in parallel. Multiple indexer components are created, each handling a portion of the documents. This segmentation enables concurrent processing across multiple threads or processes, significantly improving indexing speed while maintaining manageable system complexity through modular design.
Solution Approach 2:
The patent implements a dynamic indexing system where the number and allocation of indexer components can be adjusted based on system load and requirements. The indexing process transitions from static batch processing to dynamic parallel processing, allowing the system to adapt resource allocation in real-time, thereby improving productivity without permanently increasing system complexity.
2Loss of information
If batch processing is used for document indexing, then system complexity is reduced, but data freshness deteriorates
Solution Approach 1:
The patent implements continuous or near-continuous indexing by maintaining multiple indexer components that process documents as they become available, rather than waiting for batch processing cycles. This continuous action ensures that the index remains fresh and up-to-date, minimizing information loss while using a standardized parallel processing approach that doesn't dramatically increase system complexity.
3Productivity
If parallel processing is implemented, then indexing speed is improved, but coordination overhead increases
Solution Approach 1:
The patent introduces a queue component as an intermediary between document sources and multiple indexer components. This queue manages the distribution of documents to indexers, handles load balancing, and coordinates processing without requiring complex direct communication between indexers. The intermediary simplifies coordination overhead while enabling parallel processing to achieve high indexing throughput.
Data Source
AI summary
Methods, apparatus, and processor-readable storage media for document indexing with cluster computing are provided herein. An example computer-implemented method includes obtaining and monitoring a list of document identifiers corresponding to documents to be indexed as part of an indexing process across at least a portion of a computing cluster; initiating the indexing processing based on one or more inputs; iteratively requesting at least one of the document identifiers; obtaining, from across at least a portion of the computing cluster, a document corresponding to a document identifier provided in response to the request; and carrying out one or more functions related to the indexing process on the obtained document.