Stateless Distributed Parallel Crawling of Interactive Applications

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional methods for validating dynamic Web 2.0 applications are inadequate in capturing their rich, stateful behavior, and existing crawlers are computationally intensive, limiting the exploration of web applications' true behavior-space.

Innovation Solution

A distributed computing system with a master node and worker nodes that crawl interactive client-server applications, utilizing a master state graph to identify reconvergent traces and add results without removing duplicates, enabling efficient parallel crawling of dynamic web applications.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional validation techniques based on manual testing are used, then the process is simple to implement, but they are completely inadequate at capturing or exploring the rich, stateful behavior of web applications

Engineering Contradiction:

Improvebehavior capture capabilityVSAvoidvalidation system complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent replaces manual testing mechanisms with an automated crawler system that uses programmatic state exploration and graph-based tracking to capture web application behavior, thereby achieving comprehensive behavior capture without manual intervention

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent transforms the validation approach by changing from manual observation parameters to automated state graph parameters, where each node represents an application state and edges represent transitions, enabling systematic exploration of behavior space

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If custom AJAX web application crawlers are used to comprehensively explore behavior, then behavior capture capability is improved, but the crawling becomes very computationally intensive

Engineering Contradiction:

Improvebehavior exploration completenessVSAvoidcomputational intensity

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent merges the state graph maintenance functionality into the crawler itself, eliminating the need for separate synchronization processes between master and worker nodes, thereby reducing overall computational overhead while maintaining comprehensive behavior exploration

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The crawler performs self-service by automatically maintaining its own state graph and detecting reconvergent states through local computation, eliminating the need for external coordination and reducing computational intensity

Inventive Principle:

Principle #25Self-service

3Loss of substance

If duplicate states or transitions are removed during crawling, then data redundancy is reduced, but the crawling process becomes more computationally intensive

Engineering Contradiction:

Improvedata redundancyVSAvoidprocessing overhead

Core Design Contradiction:

Loss of substanceVSUse of energy by moving object

Solution Approach 1:

The patent inverts the traditional approach by not removing duplicates but rather by detecting and utilizing reconvergent states as validation opportunities, where the same state reached through different paths confirms correct application behavior

Inventive Principle:

Principle #13The other way round (Inversion)

4Productivity

If parallel crawling is implemented across multiple worker nodes, then productivity is improved, but coordination overhead and complexity increase

Engineering Contradiction:

Improvecrawling speedVSAvoidsystem coordination complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the crawling task across multiple worker nodes, each independently maintaining its own state graph and exploring different portions of the application behavior space, thereby achieving parallel productivity without complex coordination

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent converts the potential harm of duplicate exploration by multiple workers into a benefit by using reconvergent state detection as a validation mechanism, where independent workers reaching the same state through different paths confirms correct application behavior

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Data Source

PatentUS8880588B2Technique for stateless distributed parallel crawling of interactive client-server applications

Publication Date: 2014.11.04 FUJITSU LTD

AI summary

A distributed computing system includes worker nodes and a master node including a processor coupled to a memory. Each worker node crawls a portion of an interactive client-server application. The memory includes a master state graph, including the results of crawling. The master node is configured to examine the master state graph to determine a number of reconverging traces, receive a result from a job from a worker node if the number of reconverging traces is below a threshold, and add the result to the master state graph without attempting to remove duplicate states or transitions. A trace includes states and transitions representing valid. A reconvergent trace includes a trace including a reconvergent state, which is a state that can be reached through two or more distinct traces. The result containing states and transitions is associated with crawling a first portion of the interactive client-server application.