URL Categorization via Suffix Tree and Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing firewall systems face challenges in efficiently enforcing URL-based policies due to the vast number of URLs and dynamic content, leading to difficulties in maintaining up-to-date categorization information.
Innovation Solution
The implementation of a policy enforcement appliance that utilizes a word-based suffix tree and a distributed database configuration to efficiently store and retrieve URL categorization information, reducing the need for frequent remote queries and improving latency and cache efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional firewall systems store and query URL categorization information using conventional methods, then they can enforce URL-based policies, but they experience high latency and poor cache efficiency due to the vast number of URLs and dynamic content
Solution Approach 1:
The patent segments the URL categorization system into multiple components: a distributed database configuration that divides storage across multiple nodes, a caching layer that stores frequently accessed categorization data locally, and a query optimization layer that uses suffix trees for efficient pattern matching. This segmentation allows the system to handle the vast number of URLs without centralized bottlenecks, improving query speed and reducing latency.
Solution Approach 2:
The patent implements preliminary action by pre-computing and storing URL categorization results in a distributed database before they are needed for policy enforcement. The system proactively categorizes URLs and maintains an updated cache of categorization information, so when a firewall needs to enforce a URL-based policy, the categorization data is already available locally rather than requiring real-time computation, thereby reducing latency.
2Reliability
If firewall systems query remote servers for URL categorization information, then they can maintain up-to-date categorization, but they incur high network traffic and reduced performance
Solution Approach 1:
The patent introduces a distributed database as an intermediary between the firewall system and remote URL categorization sources. The database stores categorization information locally at multiple distributed nodes, acting as a mediator that eliminates the need for frequent remote queries. The system can maintain reliable, up-to-date categorization information through the distributed cache without the performance penalty of continuous network communication, as the intermediary holds the data locally for rapid access.
3Adaptability or versatility
If the system stores detailed categorization information for all URLs, then it can enforce granular policies, but it consumes excessive memory and processing resources
Solution Approach 1:
The patent applies local quality by allowing different nodes in the distributed database to store different subsets of URL categorization information based on their local needs and access patterns. Each node can optimize its storage to hold the most frequently accessed or relevant categorization data locally, rather than every node storing complete categorization information for all URLs. This enables granular policy enforcement at each location while conserving overall memory resources through localized, selective storage.
Data Source
AI summary
A URL categorization query is received. The URL categorization query includes at least one URL. The URL is used to determine a set of data distribution keys. A distributed key-value data store is queried using at least one data distribution key included in the determined set of data distribution keys. Categorization information is returned. The returned URL categorization information can be used to enforce policies.