Ajax Web Page Content Retrieval via JavaScript Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web crawlers fail to effectively retrieve dynamic content from Ajax web pages, leading to incomplete search results in search engines.
Innovation Solution
A method and system that involves obtaining web page codes, retrieving JavaScript information, analyzing it to identify functions with Ajax calls, and triggering these functions to obtain dynamic content, allowing for complete web page content retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional web crawlers are used to retrieve web page content, then static content can be collected, but dynamic content generated by Ajax calls cannot be retrieved
Solution Approach 1:
The crawler performs preliminary analysis of JavaScript code to identify Ajax call functions before executing the web page. This allows the system to prepare and trigger specific functions that generate dynamic content, ensuring complete content retrieval while maintaining a structured approach that doesn't excessively increase system complexity
Solution Approach 2:
The patent introduces an intermediary component that acts as a bridge between the conventional crawler and the dynamic content generation process. This intermediary analyzes JavaScript, identifies Ajax functions, and triggers them appropriately, enabling dynamic content retrieval without requiring a complete redesign of the crawler architecture
2Loss of information
If the entire web page is refreshed to update content, then all content including dynamic content can be retrieved, but network bandwidth is wasted and server workload increases
Solution Approach 1:
The patent extracts and isolates only the specific JavaScript functions that generate dynamic content through Ajax calls. Instead of retrieving or refreshing the entire web page, the crawler identifies and executes only these necessary functions, thereby obtaining dynamic content while avoiding the network bandwidth consumption and server workload associated with full page refreshes
Solution Approach 2:
The system performs preliminary analysis of the web page code to identify which specific functions generate dynamic content. This preparation step enables targeted execution of only the necessary functions, preventing unnecessary network traffic and server processing that would occur with complete page refreshes
Data Source
AI summary
The present disclosure describes a method and a system of retrieving Ajax web page content. The method of retrieving Ajax web page content includes: obtaining information of web page codes; retrieving JavaScript information in the information of the web page codes; analyzing the JavaScript information to determine function(s) that include(s) Ajax call(s) in the web page codes; and triggering the determined function(s) that include(s) the Ajax call(s) to obtain web page content. Through analyzing JavaScript information in web page codes, the above technical scheme obtains function(s) that include(s) Ajax call(s) in the web page codes and further triggers the function(s) to obtain web page content generated thereby, thus achieving retrieval of dynamic content in an Ajax web page. Using the above technical scheme, a search engine can collect more complete web page contents and hence provide a better search service to users.