Crawler Emulator Extracts App Content for Search Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current applications generate content that is not searchable by conventional search engines, as this content is only accessible upon installation and interaction with the application, limiting user discovery of available information.
Innovation Solution
The described technologies enable crawling of computer-executable applications to extract and index static and dynamic content, allowing for searchable pages by emulating application execution in an emulator, using optimization techniques like pre-fetching and location granularity to reduce crawling time and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional search methods are used to access application content, then users must install and execute the application to view content, but this makes content discovery difficult and time-consuming
Solution Approach 1:
The system performs preliminary crawling and indexing of application content before users search for it. The crawler executes applications in an emulator environment to extract static content from binaries and dynamic content from runtime execution, pre-processing all content into a searchable index. This eliminates the need for users to install and manually explore applications to discover content.
Solution Approach 2:
The patent introduces an intermediary crawling system that acts as a mediator between applications and users. Instead of direct user-application interaction requiring installation, the crawler serves as an intermediary that extracts, processes, and indexes content from applications, making it accessible through standard search interfaces without requiring user installation or execution of the actual application.
2Loss of information
If full crawling of all applications is performed to make all content searchable, then complete content coverage is achieved, but crawling time and computational resources increase significantly
Solution Approach 1:
The crawling process is segmented into distinct phases: static content extraction from application binaries, dynamic content extraction through emulator execution, and indexing. Each phase processes different types of content using specialized techniques. Static content is extracted without full execution, while dynamic content is captured through selective emulator interactions, allowing parallel processing and optimized resource allocation.
Solution Approach 2:
The system performs partial crawling by focusing on extracting essential content rather than complete exploration of all application features. The crawler identifies and extracts key static strings, URLs, and data from binaries, and captures representative dynamic content through selective emulator interactions, achieving sufficient content coverage without exhaustive exploration of every possible application state and user interaction path.
3Loss of information
If applications are executed in an emulator to extract dynamic content, then comprehensive content retrieval is achieved, but execution time and computational overhead increase
Solution Approach 1:
Static content extraction is performed preliminarily by analyzing application binaries directly without emulator execution. The crawler parses executable files to extract hardcoded strings, URLs, and data structures, obtaining a significant portion of content before emulator invocation. This preliminary extraction reduces the burden on the emulator phase to only capture dynamic, runtime-generated content.
Solution Approach 2:
The emulator execution is structured as periodic, controlled interactions rather than continuous full execution. The crawler triggers specific user actions and navigation sequences at predetermined intervals, capturing dynamic content generated during these periodic execution cycles. This allows the system to obtain representative dynamic content through multiple short emulator sessions rather than one prolonged execution.
Data Source
AI summary
Technologies related to crawling computer-executable applications are described. A full crawl is executed over an application, where executing the full crawl includes causing the application to output a plurality of pages. The application retrieves content from the World Wide Web when generating the pages for output. Thereafter, a fast crawl is executed over the application, where executing the fast crawl takes less time when compared to the time needed to execute the full crawl.