Asynchronous Enterprise Data Crawling via Event Queue
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronous search crawls on production systems cause significant processing overhead and security concerns by requiring direct data access from enterprise applications, while asynchronous techniques may compromise security by storing data in regular files accessible to unauthorized users.
Innovation Solution
Implementing an asynchronous crawling method that uses an event message queue to de-queue and transform update notifications into an asynchronous feed, which is then provided to a search engine crawler, preserving security and minimizing runtime overhead by allowing access to only searchable data within the enterprise system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If synchronous crawling is used to access enterprise application data, then data can be retrieved directly from the application, but processing overhead on the production system increases significantly
Solution Approach 1:
The patent implements preliminary action by pre-processing enterprise application data into an asynchronous feed before the search crawler needs it. The data is extracted, transformed, and staged in advance in a location accessible to the crawler, eliminating the need for the production system to process crawl requests in real-time. This resolves the contradiction by preparing data beforehand, so efficient retrieval is possible without imposing processing overhead on the running application.
2Ease of operation
If synchronous crawling directly accesses enterprise application data, then data access is straightforward, but security concerns arise from exposing sensitive data
Solution Approach 1:
The patent introduces an intermediary component that sits between the enterprise application and the search crawler. This intermediary pre-processed data into an asynchronous feed format, allowing the crawler to access data without directly connecting to the production system. The intermediary transforms and stages data in a secure manner, maintaining security protocols while enabling straightforward data access for the crawler. This resolves the contradiction by mediating the interaction, so ease of operation is maintained without compromising security.
3Use of energy by moving object
If asynchronous techniques are used to pre-process data into regular files, then processing overhead is reduced, but security is compromised by storing data in accessible files
Solution Approach 1:
The patent applies local quality by creating a specialized, controlled-access storage location for the asynchronous feed data, rather than using regular accessible files. The feed is stored in a location with restricted access permissions, ensuring that while the data is pre-processed and available for efficient crawling, it remains protected from unauthorized access. This resolves the contradiction by making the storage location locally optimized for crawler access while maintaining security through restricted permissions, so processing overhead is reduced without creating security vulnerabilities.
Data Source
AI summary
Methods and systems for providing searchable data associated with enterprise applications are provided. An asynchronous feed may be generated from data stored in a database and searched by search engine crawlers. The feed may be populated with searchable data based on a searchable object definition that describes the location of searchable data within the database. The feed also may enforce access restrictions set by the enterprise applications to prevent unauthorized access to the searchable data.