Distributed Data Indexing API for Scalable Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data indexing methods for search tools in cloud-based data processing platforms are not scalable, requiring manual updates for new data resources and placing a heavy processing burden on centralised systems, which can lead to delays and inefficiencies.
Innovation Solution
A method and system for data indexing that utilizes a predetermined API for data resources to indicate their readiness for indexing, generating an event queue for events associated with the data, and periodically querying these event queues to crawl and index data without the need for manual updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a centralised crawling and indexing system is used, then data can be indexed for search, but the system cannot be easily scaled and requires manual updates for new data resources
Solution Approach 1:
The patent segments the centralized indexing system into distributed components by enabling individual data resources to implement their own crawling and indexing logic through a standardized API interface. This allows each data resource to be independently managed and scaled without affecting the entire system, resolving the contradiction between scalability and complexity.
Solution Approach 2:
The patent implements self-service by allowing data resources to automatically register themselves with the search service through the API and perform their own crawling and indexing operations. This eliminates the need for manual updates and centralized management, enabling the system to scale automatically as new data resources are added.
2Productivity
If a centralised crawling system is used, then existing data resources can be indexed, but new data resources introduce delays requiring manual system updates
Solution Approach 1:
The patent applies preliminary action by having data resources pre-implement the standardized API interface, which includes pre-configured crawling and indexing capabilities. When new data resources are added, they are already prepared to be discovered and indexed immediately by the search service, eliminating the manual update delay.
Solution Approach 2:
The patent implements feedback mechanisms where data resources automatically report their availability and status to the search service through the API. This allows the search service to dynamically discover and index new data resources in real-time, eliminating the time delay associated with manual system updates.
3Reliability
If a centralised system processes all indexing operations, then comprehensive search coverage is achieved, but heavy processing burden slows down other services
Solution Approach 1:
The patent segments the processing burden by distributing crawling and indexing operations to individual data resources rather than concentrating them in a centralized system. Each data resource processes its own data locally, reducing the overall processing load on the search service while maintaining comprehensive search coverage across all resources.
Solution Approach 2:
The patent transitions from a single-dimensional centralized processing model to a multi-dimensional distributed model where indexing occurs at multiple levels (local data resource level and centralized search service level). This dimensional change allows processing to be parallelized across numerous data resources, reducing the burden on any single system while maintaining comprehensive coverage.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods and systems for indexing data for a search tool are disclosed, wherein the method is performed using one or more processors. The methods and systems may comprise an operation of providing an indexing application programming interface (API) for implementation in relation to one or more data resources of a data processing platform requiring data of the data resource to be indexed for searching using a search tool of the data processing platform. The interface may be configured, when implemented in relation to a data resource, to send data indicative of one or more definitions of data to be crawled from said data resource to an indexing system of the data processing platform, and to generate an event queue indicative of one or more events associated with said defined data that have occurred in relation to the data resource. A further operation may comprise identifying one or more data resources that have implemented the indexing API. A further operation may comprise querying the one or more identified data resources to determine one or more events indicated in their event queue. A further operation may comprise crawling the one or more identified data resources based on the determined one or more events to receive data for indexing. A further operation may comprise indexing the received data associated with said one or more events for access by a search tool.