URL Lattice Pruning for Web Data Pattern Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large volumes of Web activity data to extract useful information is challenging due to varying techniques and codes used by online entities to represent similar information, and existing methods lack effective techniques for identifying significant information patterns.
Innovation Solution
The system processes Web activity data by generating a URL lattice and corpus lattice structures to identify common patterns among URLs, applying decision procedures for pruning and generating output data structures that allow for the application of common rules to groups of URLs, enabling the extraction of useful information and characterization of URLs for data mining purposes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing methods are used on large volumes of Web activity data, then data processing can be performed, but the efficiency and effectiveness of extracting useful information is low due to varying techniques and codes used by online entities
Solution Approach 1:
The patent segments URLs into hierarchical components (protocol, host, path, query parameters) and organizes them into lattice structures. This segmentation transforms the complex, unstructured Web activity data into organized, processable units that can be efficiently analyzed for patterns and trends.
Solution Approach 2:
The patent introduces URL lattices and corpus lattices as intermediary data structures between raw Web activity data and the final information extraction. These lattices serve as mediators that standardize and organize diverse URL formats from different online entities, enabling efficient pattern recognition without directly processing the raw complexity.
2Measurement precision
If all URLs are processed in detail to ensure accurate information extraction, then measurement precision improves, but processing time and computational resources increase significantly
Solution Approach 1:
The patent merges multiple URLs into lattice structures based on their hierarchical components. By combining URLs that share common patterns into lattice nodes, the system processes groups of URLs simultaneously rather than individually, maintaining accuracy through the lattice structure while significantly reducing processing time.
Solution Approach 2:
The URL lattice structure serves multiple functions: it organizes URLs hierarchically, identifies patterns across diverse formats, enables efficient querying, and supports various analysis operations. This multi-functionality allows the same data structure to maintain precision while enabling rapid processing through its organized framework.
Data Source
AI summary
The present disclosure provides a computer-implemented method of processing a group of resource identifiers. The method includes generating a node candidate based on a resource identifier, wherein the node candidate includes a component corresponding to a portion of the resource identifier according to a dimension of specificity. The method also includes adding the lattice node candidate to a corpus lattice based on whether a matching corpus node corresponding to the node candidate is already included in the corpus lattice. The corpus lattice includes corpus nodes corresponding to a plurality of resource identifiers. The method also includes generating a pruned corpus lattice by a process that includes determining that a child node in the corpus lattice is compatible with a parent node of the child node and removing the child node from the corpus lattice. The method also includes generating an output file corresponding to the pruned URL lattice.