Automated Taxonomy Training Set Construction via Web Crawling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing taxonomic classification methods require human intervention to label documents, which is time-consuming and expensive, and cannot be easily updated to reflect rapid changes in content, such as new products or concepts.
Innovation Solution
A computer system that automatically generates a training set by crawling top-level sites, determining site-specific extraction templates, and associating crawl data with hierarchical categories, reducing the need for human labeling and enabling fast, inexpensive updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If human readers are employed to label documents manually, then the accuracy and reliability of classification is improved, but the time consumption and cost increase significantly
Solution Approach 1:
The patent uses web crawlers to automatically copy and collect data from websites, replacing the manual copying process. The crawler systematically navigates through websites and extracts relevant information, creating a digital copy of the data that can be processed without human intervention for each individual document
Solution Approach 2:
The patent replaces the mechanical human labeling process with an automated computer-based system. The crawler and extractor components perform the labeling function that would otherwise require human readers, using algorithmic processing instead of manual analysis to assign taxonomic categories to documents
2Reliability
If manual labeling processes are used, then the initial classification accuracy is improved, but the ability to update quickly to reflect new content is worsened
Solution Approach 1:
The patent implements continuous operation through the crawler, which continuously navigates and collects data from websites without interruption. This ongoing automated process ensures that the system is always updated with current content, enabling rapid adaptation to new products, services, or concepts that emerge on the web
Solution Approach 2:
The system performs self-updating through automated crawling and extraction. The crawler independently monitors web content changes and the extractor automatically processes new information, allowing the classification system to maintain accuracy without requiring manual re-labeling when new content appears
3Productivity
If automated crawling and extraction is implemented, then the speed and cost-effectiveness is improved, but the complexity of the system increases
Solution Approach 1:
The patent divides the complex labeling task into separate functional modules: a crawler component that handles data collection, an extractor component that handles data processing and labeling, and a taxonomic classification framework. This segmentation allows each component to be developed, tested, and maintained independently, managing overall system complexity
Solution Approach 2:
The patent creates a universal labeling framework that can be applied across different domains and taxonomies. The same crawler and extractor components can serve multiple purposes by adjusting the taxonomic classification schema, eliminating the need for separate specialized systems for different classification tasks
Data Source
AI summary
A training set generator may be configured to input a taxonomy including a hierarchy of categories and a plurality of top-level sites, and to output a training set of categorized data. The training set generator may include a crawler configured to crawl each of the top-level sites to determine at least one lower-level site associated therewith and to store the top-level sites and associated lower-level sites as crawl data. The training set generator also may include an extractor configured to determine, for each of the top-level sites, a corresponding site-specific extraction template associating at least one portion of the corresponding top-level site with at least one category of the hierarchy of categories, and further configured to apply each site-specific extraction template to corresponding crawl data to thereby associate the crawl data with the categories of the hierarchical categories and obtain categorized data of the training set.