Universal Crawler Interface for Structured Application Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for crawling and indexing structured application data are inefficient, requiring significant user effort and being dependent on specific search engine plug-ins, which are business object-specific and expose proprietary data, leading to security concerns and deployment difficulties.
Innovation Solution
A computer-based method and interface for crawling structured application data that allows for the creation of crawlable definitions through a user interface, enabling the extraction and indexing of complex business objects without requiring plug-ins, while maintaining data security by controlling exposure through defined metadata tags and secure endpoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a crawler plug-in is used to crawl structured application data, then the data can be extracted and indexed, but the implementation becomes complicated and time-consuming due to the need for individual plug-in creation for each business object
Solution Approach 1:
The patent creates a universal crawler interface that can crawl multiple types of structured application data across different business objects without requiring individual plug-ins. The interface uses a standardized approach with configurable parameters to handle diverse data sources, making the crawling capability applicable to enterprise applications, web applications, and other structured data sources uniformly.
Solution Approach 2:
The patent introduces an intermediary layer between the search engine crawler and the application data. This interface acts as a mediator that translates crawler requests into application-specific queries and formats results in a standardized manner, eliminating the need for direct plug-in implementations for each business object.
2Adaptability or versatility
If individual crawler plug-ins are created for each business object, then specific data extraction requirements are met, but the overall system complexity increases significantly
Solution Approach 1:
The patent implements a single universal crawler interface that can adapt to multiple business objects and data structures through configuration rather than through multiple specialized plug-ins. The interface supports customizable query templates, field mappings, and data transformation rules that can be adjusted to meet specific extraction requirements without increasing system complexity.
Solution Approach 2:
The patent uses parameter-based configuration to adapt the crawler interface to different business objects. Instead of creating structurally different plug-ins, the system modifies behavioral parameters such as query templates, data source connections, and result formatting to accommodate various extraction needs, thereby maintaining low system complexity while preserving high adaptability.
3Reliability
If the crawler is fixed to the business object structure at creation time, then initial crawling works, but the crawler becomes inoperable when the business object structure changes
Solution Approach 1:
The patent makes the crawler interface dynamic by allowing configuration changes after initial setup. The interface can be reconfigured to adapt to structural changes in business objects through updated query templates, modified field mappings, and adjusted data source connections, ensuring continued operability without requiring complete plug-in recreation.
Solution Approach 2:
The patent establishes a flexible framework in advance that anticipates future structural changes. By designing the interface with configurable parameters and templates from the beginning, the system prepares for adaptability upfront, allowing easy adjustment when business object structures evolve without compromising initial crawling reliability.
4Adaptability or versatility
If multiple search engines are used for different applications, then specific search requirements are met, but the development effort multiplies due to the need for separate plug-in implementations
Solution Approach 1:
The patent creates a universal crawler interface that can work with multiple search engines through a standardized connection layer. The interface abstracts search engine-specific details behind a common API, allowing the same interface configuration to serve multiple search engines without requiring separate plug-in implementations for each engine-application combination.
5Ease of operation
If JDBC API is used to access proprietary application data, then data retrieval is enabled, but data security and application architecture integrity are compromised
Solution Approach 1:
The patent introduces an intermediary layer between the crawler and the application data that prevents direct JDBC access. This interface layer retrieves data through application-provided mechanisms, formats it appropriately, and returns it to the crawler without exposing the underlying data structure or enabling direct database access, thereby maintaining security and architectural integrity while enabling data retrieval.
Data Source
AI summary
Systems and methods in accordance with various embodiments of the present invention provide for a computer based method for crawling application data from an application data store. The applications data store has business objects of an application stored thereon. The method may include identifying a first request for application data received from a search engine as a seed universal resource locator (URL). A crawlable definition for the identified business object is accessed, the crawlable definition including a query selecting one or more attributes of the business object. Moreover, the method can include sending the query to the application data store and receiving query results in response thereto. Additionally, the method can include forming a crawlable document which includes the retrieved results of the business object.