Web Crawler JavaScript De-obfuscation for Malicious Site Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional web crawler systems are ineffective in detecting malicious sites due to anti-analysis techniques such as JavaScript obfuscation, making it difficult to identify and block new malicious sites efficiently.
Innovation Solution
A web crawler system that performs anonymous browsing, automatic JavaScript de-obfuscation, and signature-less detection of malicious artifacts, utilizing heuristics to identify new JavaScript functions and domains in de-obfuscated content, and supports large-scale site classification for network security systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional web crawler techniques are used to browse websites, then the crawler can access and analyze site content, but the malicious site can detect the crawler through anti-analysis techniques (such as JavaScript obfuscation, victim's system reconnaissance, geo IP detection) and avoid detection
Solution Approach 1:
The patent introduces a headless browser as an intermediary component that executes JavaScript and renders web pages before analysis. This headless browser acts as a mediator between the crawler and the malicious site, executing the site's JavaScript code in a controlled environment while preventing the site from detecting the crawler's analytical intent. The headless browser masks the crawler's identity and provides a more realistic browsing environment that bypasses anti-detection mechanisms.
Solution Approach 2:
The patent performs preliminary actions by pre-configuring the headless browser with necessary credentials, cookies, and browsing history before analyzing the website. This preliminary setup creates a more authentic browsing environment that prevents malicious sites from detecting the crawler through system reconnaissance techniques, as the browser already appears to have legitimate user characteristics established beforehand.
2Measurement precision
If the crawler analyzes every website in detail to detect malicious content, then detection accuracy improves, but the time and resources required increase significantly
Solution Approach 1:
The patent segments the website analysis process into distinct phases: initial scanning phase that quickly evaluates basic characteristics, and detailed analysis phase that focuses only on suspicious sites. This segmentation allows the crawler to process large numbers of websites efficiently by applying comprehensive analysis only where necessary, rather than analyzing every site in full detail, thus reducing overall time and resource consumption while maintaining high detection precision for malicious sites.
Solution Approach 2:
The patent applies different levels of analysis quality to different websites based on their risk profile. High-quality detailed analysis is applied locally to suspicious or malicious sites, while standard analysis is applied to legitimate sites. This local quality approach ensures that detection precision is maximized for malicious sites without unnecessarily spending time on clearly benign websites, optimizing the balance between accuracy and efficiency.
3Adaptability or versatility
If the crawler uses anonymous browsing to avoid detection, then the anti-analysis techniques are bypassed, but the complexity of the crawler system increases
Solution Approach 1:
The patent makes the headless browser serve multiple functions: it acts as both the browsing engine for accessing websites and the anonymity mechanism for avoiding detection. The same headless browser configuration that enables JavaScript execution and page rendering also provides the anonymity by masking the crawler's identity. This multi-functionality reduces overall system complexity compared to having separate components for browsing and anonymity, as the headless browser inherently provides both capabilities.
Data Source
AI summary
Systems and methods include receiving a list of web sites; anonymously browsing to each web site in the list; receiving a response based on the browsing; and analyzing the response to classify each web site as malicious or not based on a plurality of techniques including JavaScript (JS) obfuscation detection based on de-obfuscation. The systems and methods can further include providing a blacklist of web sites classified as malicious. The systems and methods can further include determining the list of web sites periodically based on a plurality of factors. The JS obfuscation detection can be performed by de-obfuscating JS content and utilizing heuristics to determine if the de-obfuscated JS content is malicious, and the heuristics can include a presence of any of a new JS function and a domain in the de-obfuscated JS content.