Retail Crawling Engine Segmentation for Faster Data Extraction

The patent segments the webpage content into relevant and irrelevant elements using machine learning models. The crawler focuses only on extracting data from relevant elements (such as product information, prices, and descriptions) while ignoring irrelevant elements (such as navigation menus, advertisements, and footer content). This segmentation enables complete data extraction from target elements without requiring the crawler to process the entire webpage, thereby reducing crawling time while maintaining data extraction completeness.

The patent applies local quality by training machine learning models to identify and process only specific regions or elements of the webpage that contain relevant data. Instead of uniformly processing all webpage content, the system assigns different processing priorities to different parts of the page, focusing computational resources on areas with high data value. This approach achieves complete extraction of relevant data while minimizing time spent on irrelevant sections.

The patent performs preliminary action by pre-training machine learning models to recognize relevant webpage elements before the actual crawling process. The models are trained offline on large datasets of webpage structures and content patterns. During crawling, these pre-trained models quickly classify elements as relevant or irrelevant, enabling the system to achieve comprehensive data extraction from relevant elements without processing every element on the page, thus maintaining high processing speed.

The system employs self-service through automated machine learning model training and optimization. The models continuously learn from extracted data and improve their ability to identify relevant elements without manual intervention. This self-improving capability allows the system to maintain high processing speed while ensuring complete data extraction from relevant elements, as the models become increasingly efficient at distinguishing valuable content from noise.

The patent applies partial action by implementing intelligent stopping criteria that allow the crawler to achieve sufficient website coverage without processing every single webpage. The machine learning models identify high-value pages and prioritize their extraction, allowing the system to achieve comprehensive data coverage from key pages while reducing the total number of requests made to the website, thereby minimizing performance impact.

The system uses feedback mechanisms where the machine learning models continuously learn from the data extracted and the patterns observed during crawling. Based on this feedback, the crawler dynamically adjusts its behavior to focus on high-value pages and avoid redundant requests. This feedback-driven approach ensures comprehensive website coverage is achieved efficiently, reducing the total request count and minimizing negative impact on website performance.