URL Categorization via Suffix Tree and Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Firewalls face challenges in efficiently enforcing policies based on URL information due to the vast number of URLs and dynamic content, leading to difficulties in maintaining up-to-date categorization and reducing latency in URL lookups.
Innovation Solution
Implementing a policy enforcement appliance that uses a word-based suffix tree and distributed database infrastructure, including a remote URL classification server, to efficiently manage URL categorization and reduce the number of remote queries, leveraging bloom filters and caching to improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a distributed database infrastructure with remote URL classification server is used, then URL categorization accuracy is improved, but system complexity and query latency increase
Solution Approach 1:
The patent implements preliminary action by maintaining local caches of URL categorization data and using bloom filters to pre-screen URLs before querying the remote classification server. This allows frequently accessed URLs to be resolved locally without remote queries, significantly reducing latency while maintaining accurate categorization for URLs that require remote lookup.
Solution Approach 2:
The patent introduces an intermediary layer between the firewall and the remote URL classification server. This intermediary uses bloom filters and caching mechanisms to mediate queries, filtering out URLs that can be resolved from local cache and only forwarding novel URLs to the remote server, thereby reducing overall query latency while preserving categorization accuracy.
2Reliability
If comprehensive URL categorization data is maintained, then policy enforcement accuracy is improved, but resource consumption increases
Solution Approach 1:
The patent extracts only the essential categorization information needed for policy enforcement and stores it in a distributed database. Rather than maintaining complete URL databases locally at every firewall, the system extracts and stores only the necessary categorization data distributed across multiple nodes, reducing overall storage requirements while maintaining enforcement accuracy.
Solution Approach 2:
The patent uses copying by replicating essential categorization data across multiple distributed database nodes rather than storing comprehensive data at every location. This allows the system to maintain high availability and accuracy through replicated copies of critical data while avoiding the need for each node to store complete URL databases.
3Speed
If real-time URL categorization is implemented, then security response time is improved, but processing overhead increases
Solution Approach 1:
The system performs preliminary categorization actions by maintaining local caches and using bloom filters to quickly determine if URLs have been seen before. This preliminary filtering allows the majority of URLs to be processed with minimal overhead, while only novel URLs require full real-time categorization processing.
Solution Approach 2:
The patent applies partial action by implementing selective real-time categorization only for URLs that are not found in local caches or bloom filters. The system performs full categorization processing only when necessary, rather than processing every URL in real-time, thereby reducing overall processing overhead while maintaining security response time for critical cases.
Data Source
AI summary
A URL and a categorization associated with the URL are received. The received URL comprises a hostname portion and a path portion. The hostname portion of the received URL comprises a plurality of segments separated by one or more delimiters. A key associated with the received URL is determined. An operation is performed on a database using the determined key. Examples of such operations include inserting the categorization into the database, changing a value associated with the key in the database, removing a key-value pair from the database, and querying the database.