Machine Learning Taxonomy for Web Data URLs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The manual creation of taxonomies for web data from diverse sources is labor-intensive, prone to human error, and lacks automation, hindering scalability and repeatability, particularly in the context of web data aggregation.
Innovation Solution
A machine-learning-aided approach that trains a model to predict classes for web-based activities using annotated features from URLs and metadata, employing a deep neural network with a language transformer for vectorization, and a graphical user interface for manual overrides and data storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual taxonomy creation is used, then flexibility and understanding of data structure are improved, but labor intensity and time consumption increase significantly
Solution Approach 1:
The system performs preliminary actions by automatically generating taxonomy mappings before manual review. The machine learning model pre-processes web data and creates initial taxonomy assignments, which are then presented to users for verification and refinement. This reduces the overall time required while maintaining quality through the preliminary automated classification.
Solution Approach 2:
The machine learning model acts as an intermediary between raw web data and the final taxonomy structure. It processes and transforms unstructured web data into structured taxonomy mappings, which are then refined by human operators. This intermediary step reduces the manual effort required while preserving the understanding and flexibility that human oversight provides.
2Measurement precision
If manual taxonomy creation is used, then accuracy and customization are improved, but human error and inconsistency increase
Solution Approach 1:
The system implements feedback loops where the machine learning model is trained on annotated data that includes corrections from human operators. The model continuously learns from feedback regarding classification accuracy and consistency issues, improving its performance over time. This feedback mechanism maintains high accuracy while reducing inconsistency through iterative refinement.
Solution Approach 2:
The machine learning model performs self-service by automatically generating taxonomy mappings and identifying patterns in web data. It autonomously processes data and creates initial classifications without human intervention, reducing human error and inconsistency. Human operators then review and refine these self-generated mappings, maintaining accuracy while scaling the process.
3Productivity
If automated taxonomy creation is implemented, then scalability and repeatability are improved, but complexity of implementation increases
Solution Approach 1:
The system segments the taxonomy creation process into distinct modules: data ingestion, feature extraction, machine learning classification, and manual review. This segmentation allows each component to be independently optimized and managed, reducing overall system complexity while enabling scalability. The modular architecture makes it easier to deploy and maintain the automated system.
Solution Approach 2:
The machine learning model serves as an intermediary layer that handles the complex automated classification tasks, shielding users from the underlying system complexity. It transforms complex web data into structured taxonomy mappings that can be easily reviewed and refined manually. This intermediary approach enables scalability while managing complexity through automated processing.
4Productivity
If automated taxonomy creation is implemented, then efficiency and cost reduction are improved, but loss of control and understanding decrease
Solution Approach 1:
The system performs preliminary automated classification to generate initial taxonomy mappings efficiently. These preliminary results are then presented to human operators for review and refinement, ensuring that understanding and control are maintained. This preliminary action approach improves efficiency while preventing loss of understanding through human oversight.
Solution Approach 2:
The system implements feedback mechanisms where human operators review and annotate automated taxonomy mappings. This feedback ensures that the automated system maintains understanding of the data structure and classification logic. The feedback loop prevents loss of control by allowing human verification while preserving the efficiency gains from automation.
Data Source
AI summary
Machine-learning-aided automatic taxonomy for web data. In an embodiment, a training dataset of annotated features is used to train a model to predict a class in a taxonomy of web-based activities. The features may be derived from a uniform resource locator (URL) of an online resource and associated metadata. During operation, the features may be extracted from the URL and metadata of each activity record in web data. The trained model may be applied to the extracted features for each activity record to predict a class within the taxonomy. The predicted taxonomic class may be stored in association with the URL that was extracted from the activity record to produce a taxonomized URL.