URL Similarity Calculation Excluding Temporary ID Segments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods fail to accurately evaluate the similarity between URLs due to the inclusion of system-specific IDs, leading to incorrect determinations of web page identicalness.
Innovation Solution
A similarity calculation apparatus that extracts and identifies temporary IDs from URLs using statistical information from operation logs, excluding these IDs from similarity calculations and assigning higher weight to permanent IDs, thereby accurately determining web page similarity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complete URL comparison is performed to determine web page identicalness, then the determination process is simple, but the accuracy is low due to system-specific IDs causing false differences
Solution Approach 1:
The URL is segmented into multiple components including scheme, authority, path, query, and fragment. The extraction unit specifically identifies and separates ID parts from these segments, allowing the system to evaluate similarity based on structural components rather than treating the entire URL as a single string. This segmentation enables accurate comparison by isolating the meaningful parts of the URL from temporary identifiers.
Solution Approach 2:
The extraction unit extracts ID parts from URL components based on predetermined extraction rules. The determination unit then identifies which extracted ID parts are temporary by analyzing statistical information from operation logs. By extracting and separately handling these temporary ID parts, the system can exclude them from similarity calculations, thereby improving accuracy without requiring complete URL matching.
2Reliability
If traditional string comparison methods are used for URLs, then the processing is fast and simple, but the similarity evaluation is inaccurate when system-specific IDs are present
Solution Approach 1:
The system performs preliminary analysis by collecting operation logs and pre-determining extraction rules for ID parts before actual similarity comparison. The determination unit pre-identifies which ID parts are temporary by analyzing statistical patterns in advance. This preliminary action allows the main comparison process to quickly exclude known temporary IDs without performing complex real-time analysis, thus maintaining speed while improving reliability.
Solution Approach 2:
The system uses feedback from operation logs to continuously improve ID extraction accuracy. By analyzing historical operation data, the determination unit refines its identification of temporary IDs, and this learned information feeds back into the extraction process. This feedback mechanism enables the system to become increasingly accurate in distinguishing temporary from permanent URL components over time, improving reliability without proportionally increasing processing time.
Data Source
AI summary
An extraction unit (15c) extracts a part representing an ID from each of two processing target URLs contained in an operation log. A determination unit (15d) determines whether or not the part representing the ID is a temporarily generated part, by using statistical information in operation logs for a predetermined period. A calculation unit (15e) calculates similarity between the two processing target URLs by excluding the part representing the ID in a case where the part representing the ID is a temporarily generated part.