Distributed Crawling of Interactive Client-Server Applications
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional validation techniques for dynamic Web 2.0 applications are inadequate in capturing the rich, stateful behavior of interactive client-server applications, as they are computationally intensive and can only explore a fraction of the true behavior-space due to practical limitations.
Innovation Solution
A distributed computing system with a master node and worker nodes that crawl interactive client-server applications in parallel, using a master state graph to integrate results and minimize communication overhead, enabling comprehensive exploration and validation of dynamic web applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional validation techniques are used to crawl interactive client-server applications, then the crawling process is computationally intensive and can only explore a fraction of the true behavior-space, but the system complexity and resource requirements increase
Solution Approach 1:
The patent divides the crawling task into multiple independent sub-tasks that can be executed in parallel. Worker nodes each handle specific portions of the application state space, allowing the system to explore more behavior-space without proportionally increasing central coordination complexity. The master node manages only the coordination and integration, not the actual crawling work.
2Productivity
If distributed parallel crawling is implemented to explore comprehensive behavior-space, then the crawling coverage improves, but the communication overhead and coordination complexity increase
Solution Approach 1:
The patent merges results from multiple worker nodes into a single master state graph. This consolidation approach allows comprehensive exploration of behavior-space while minimizing communication overhead by combining results rather than requiring continuous coordination during the crawling process. The master node integrates partial results efficiently without requiring extensive back-and-forth communication.
3Productivity
If multiple worker nodes crawl in parallel to increase productivity, then the crawling speed and coverage improve, but the integration and coordination difficulty increases
Solution Approach 1:
The master node serves as an intermediary that coordinates between multiple worker nodes. It receives crawling results from workers, integrates them into the master state graph, and manages the overall crawling process. This intermediary structure simplifies integration complexity by providing a centralized coordination point rather than requiring direct peer-to-peer coordination between all worker nodes.
Data Source
AI summary
An electronic device includes a memory and a processor coupled to the memory. The memory contains a master state graph. The master state graph includes information regarding the operation of interactive client-server application. The processor is configured to send a first job to a first worker node, send a second job to a second worker node, receive results of crawling the interactive client-server application, and integrate results of crawling the interactive client-server application into the master state graph. The first job includes crawling instructions for crawling a first portion of an interactive client-server application. The second job includes crawling instructions for crawling a second portion of the interactive client-server application. The first worker node and second worker node crawl the interactive client-server application in parallel.