Network Crawler Detection via Browsing History Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network-based services face challenges in effectively identifying and providing personalized content and services to users without proper user identification, as well as in distinguishing between genuine users and web crawlers, leading to inaccessible services and inefficient resource allocation.
Innovation Solution
Implementing a tracking service that analyzes user actions by inserting identifiers and JavaScript code into network documents, which access the user's browsing history to classify users based on their behavior, providing targeted content, authenticating users, and detecting web crawlers by associating user characteristics with specific actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If user tracking and classification services are implemented to provide personalized content and authentication, then service personalization and user experience are improved, but device complexity and implementation difficulty increase
Solution Approach 1:
The patent introduces a tracking service as an intermediary component that sits between the network documents and the user classification system. This tracking service maintains classifiers and inserts tracking identifiers into documents, mediating the complex interactions between user behavior tracking and service personalization. By isolating the complexity into a dedicated intermediary service, the system achieves personalization capabilities while managing complexity through clear separation of concerns.
2Measurement precision
If browsing history analysis is implemented to classify users and detect crawlers, then user identification accuracy and crawler detection capability are improved, but information processing time and computational resources increase
Solution Approach 1:
The patent implements preliminary classification by pre-defining user characteristics and maintaining classifiers that map browsing behaviors to user types. The tracking service pre-processes and categorizes browsing history data as users navigate, performing classification actions in advance rather than analyzing complete histories later. This preliminary action reduces the computational burden and time required for real-time user identification and crawler detection.
3Loss of information
If JavaScript code is inserted into network documents for tracking user actions, then user behavior tracking capability is improved, but network document size and loading time increase
Solution Approach 1:
The patent extracts the tracking functionality from within the main network documents by using a separate tracking service that inserts tracking identifiers. Rather than embedding extensive JavaScript code directly in each document, the system extracts tracking concerns into a dedicated service that manages classifiers and inserts only essential tracking identifiers. This extraction reduces the quantity of tracking code in individual documents while preserving comprehensive behavior tracking capability.
Data Source
AI summary
Techniques for detecting a network crawler may be described. In particular, a request for information may be received from a computing system. Based on this request, a network-based document may be provided to the computing system. The network-based document may include a portion of the information, code, and an identifier of another network-based document. The code may be configured to, upon execution, determine whether the other network-based document was accessed prior to providing the network-based document to the computing system. An indication whether the other network-based document was accessed may be received from the computing system. The indication may be received based on an execution of the code at the computing system. Based on the indication, the request for the information may be determined as being associated with a network crawler hosted on the computing system.