Domain-Agnostic Bot Detection via Behavioral Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems are inefficient in distinguishing between good and bad robotic activity, often filtering out good bots while allowing malicious ones to bypass detection, leading to wasted resources and user frustration, and lack flexibility in detecting robotic activity across multiple domains.
Innovation Solution
The system uses machine-learning models to classify network sessions as human, good robotic, or bad robotic activity by training classifiers with labeled data from high-quality session information, generating a domain-agnostic model that can identify robotic activity across various domains, and exclude robotic activity from analytics reports.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If robot exclusion protocols (robots.txt) are used to limit robotic traffic, then good bots that self-identify are filtered out, but bad bots that ignore protocols are not filtered
Solution Approach 1:
The system changes the detection parameters from simple protocol compliance (robots.txt) to analyzing multiple behavioral parameters including session duration, navigation patterns, click behavior, and interaction quality. This allows differentiation between good bots that follow protocols and bad bots that ignore them, resolving the contradiction by using more nuanced detection criteria.
Solution Approach 2:
The patent introduces an intermediary classification system between the robots.txt protocol and traffic filtering. This intermediary layer analyzes bot behavior patterns and assigns classifications (good, bad, or unknown) before applying filtering decisions, allowing good bots to pass through while blocking bad bots that would otherwise evade simple protocol-based filtering.
2Object-affected harmful factors
If Turing tests (CAPTCHA) are implemented to limit robotic activity, then some robotic activity is blocked, but good bots cannot be distinguished from bad bots and user frustration increases
Solution Approach 1:
Instead of applying a uniform Turing test to all traffic, the system applies different detection strategies to different traffic segments. Good bots are identified through behavioral analysis without user interaction, while only suspicious or unclassifiable traffic undergoes Turing testing. This localized application reduces user frustration while maintaining effectiveness against malicious bots.
Solution Approach 2:
The patent segments traffic into different categories (human, good bot, bad bot, unknown) and applies different handling strategies to each segment. This segmentation allows most legitimate traffic to flow smoothly while directing only potentially malicious traffic through Turing tests, improving user experience while maintaining security.
3Object-affected harmful factors
If IP address blacklisting is used to restrict known robotic sources, then identified sources are blocked, but new sources cannot be detected and maintenance costs are high
Solution Approach 1:
The system performs preliminary behavioral analysis on all incoming traffic to classify bots before they can establish malicious activity patterns. By pre-classifying traffic based on navigation patterns, session characteristics, and interaction quality, the system proactively identifies both known and unknown bad bots, eliminating the need for continuous blacklist updates and improving detection of new threats.
Solution Approach 2:
The patent implements a universal detection system that works across all IP addresses and domains using behavioral analysis rather than domain-specific blacklists. This multi-functional approach detects both known and unknown bad bots through consistent behavioral patterns, making the system adaptable to new threats without requiring separate detection mechanisms for different sources.
4Measurement precision
If conventional filtering methods are applied uniformly across all traffic, then computing resources are wasted on analyzing good bot traffic, but detection accuracy decreases
Solution Approach 1:
The system segments traffic analysis into multiple stages with increasing computational intensity. Quick behavioral checks are applied to all traffic first, with only potentially malicious traffic proceeding to more computationally intensive analysis. This segmentation maintains high detection accuracy for robotic activity while minimizing computing resource waste on clearly legitimate human or good bot traffic.
Solution Approach 2:
Instead of applying full detection algorithms to all traffic, the system applies partial analysis to most traffic and reserves excessive (intensive) analysis only for suspicious cases. This approach achieves sufficient detection accuracy for the majority of traffic while conserving computing resources, applying intensive processing only when necessary to identify malicious bots.
Data Source
AI summary
Methods, systems, and non-transitory computer readable storage media are disclosed for detecting robotic activity while monitoring Internet traffic across a plurality of domains. For example, the disclosed system identifies network session data for each domain of a plurality of domains, the network session data including network sessions comprising features that indicate human activity. In one or more embodiments, the disclosed system generates a classifier to output a probability that a network session at a domain includes human activity. In one or more embodiments, the disclosed system also generates a classifier to output a probability that a network session includes good robotic activity. Additionally, the disclosed system generates a domain-agnostic machine-learning model by combining models from a plurality of domains with network sessions including human activity.