Crawler Route Server for Transit Mode Web Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing webpage access technologies in transit mode face slower access speeds and inefficiencies due to cross-network access between transit servers and target website servers, particularly affecting low-end mobile devices with limited resources.
Innovation Solution
A crawler route server is introduced to receive webpage access requests, identify the target crawler server with the fastest speed for accessing a target website, and direct the client to use that server, optimizing network routes based on ISP matching and periodic speed reporting from multiple crawler servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a transit server uses proxy to access target website server, then the mobile browser client can access webpages, but the access speed is slow and efficiency is low due to cross-network access
Solution Approach 1:
The system pre-establishes multiple crawler servers in different network environments (different ISPs) before actual access occurs. When a client needs to access a website, the route server has already have the pre-established crawler servers ready to provide optimal routing, eliminating the need for real-time network discovery and reducing access latency
Solution Approach 2:
The patent introduces a route server as an intermediary between the mobile browser client and the target website server. This route server receives access requests from clients, determines the optimal crawler server based on network environment matching, and forwards requests through the most efficient path, thereby resolving the cross-network access efficiency problem
2Speed
If multiple crawler servers are deployed in different network environments, then access routes can be optimized, but the system complexity increases
Solution Approach 1:
Crawler servers automatically report their access speeds and network status to the route server without requiring manual configuration or intervention. The route server autonomously collects this data, processes it, and makes routing decisions based on the reported information, reducing the operational complexity of managing multiple servers
Solution Approach 2:
The system dynamically adjusts routing parameters based on real-time network conditions. The route server monitors access speeds and network status of different crawler servers, and automatically changes the routing path to use the optimal server for each access request, thereby optimizing connection speed without requiring complex manual intervention
Data Source
AI summary
A method for accessing webpages based on a transit mode is provided. A crawler route server receives a webpage access request sent from a client where the webpage access request contains identification of a target website to be accessed. The crawler route server obtains a target crawler server which has the fastest speed for accessing the target website corresponding to the identification of the target website to be accessed from at least one crawler server. Further, the method includes the crawler route server returns the identification of the target crawler server to the client, and the client accesses the target website through the target crawler server corresponding to the identification of the target crawler server.