Web Crawler Session State Detection for Logout Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies for identifying logout pages in web applications are inefficient, relying on expert knowledge and regular expressions, which fail when dealing with non-English languages or varying logout page terminology, leading to unpredictable session invalidation and crawl performance issues.
Innovation Solution
A computer-implemented process that excludes marked logout requests, crawls the web application, determines the session state, logs in when out of session, selects unprocessed requests, and marks requests meeting logout criteria, thereby accurately identifying logout requests and improving crawl performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If regular expressions are used to identify logout pages, then the identification process is simple, but it fails when dealing with non-English languages or varying logout page terminology
Solution Approach 1:
The system performs self-learning by automatically analyzing web application responses to identify logout pages without requiring pre-configured regular expressions or expert knowledge. The crawler autonomously determines which pages are logout pages by analyzing response patterns, thereby adapting to any language or terminology used in the target application.
2Measurement precision
If expert knowledge is used to identify logout pages, then identification accuracy is improved, but the complexity of the system increases
Solution Approach 1:
The system eliminates the need for expert knowledge by implementing self-learning capabilities. The crawler automatically analyzes web application responses, identifies logout pages through pattern recognition, and builds its own knowledge base during the crawling process, thereby achieving high accuracy without increasing system complexity.
Solution Approach 2:
The system performs preliminary analysis of web application responses during the crawling phase to identify logout pages before actual security testing begins. This preliminary action allows the system to learn and adapt to the specific application's logout mechanisms without requiring preconfiguration or expert intervention.
3Productivity
If logout requests are not accurately identified, then crawl performance is maintained, but false positives and false negatives increase
Solution Approach 1:
The system implements feedback mechanisms where the crawler monitors application responses during crawling, analyzes session state changes, and uses this information to identify logout pages. The identified logout pages are then used to improve the accuracy of subsequent security tests, reducing false positives and negatives while maintaining crawl performance.
Data Source
AI summary
An illustrative embodiment of a computer-implemented process for identifying a request invalidating a session excludes all marked logout requests of a Web application, crawls an identified next portion of the Web application and responsive to a determination, in one instance, that the state of the crawl is out of session, logs in to the Web application. The computer-implemented process further selects all crawl requests sent since a last time the crawl was in-session, excluding all marked logout requests and responsive to a determination that requests remain, crawls a selected next unprocessed request. Responsive to a determination, in the next instance, that state of the crawl is out of session and the selected request meets logout request criteria, the computer-implemented process marks the selected request as a logout request.