Data Refining Engine for Real-Time Price Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search engines fail to prioritize price and product information effectively, leading to delayed and incomplete real-time analysis of price histories and relationships across various dimensions such as stores, merchants, brands, regions, and time, which frustrates customers seeking up-to-date product and price insights.
Innovation Solution
A system and method that utilizes a network of computing devices to crawl and index web pages for price and product data, employing unique identifiers (iPIDs and MPIDs) to track and analyze price attributes and product attributes in real-time, enabling close-to-real-time search and analysis of price histories and relationships through modules like Core Price Module, Insight Module, and Data Ingestion Module.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional search engines use traditional crawlers to index webpages, then general content can be searched, but price and product information cannot be prioritized or analyzed in real-time
Solution Approach 1:
The system applies different crawling and indexing strategies to different types of content on webpages. Specifically, price and product information are extracted and indexed with higher priority and different metadata structures compared to general webpage content, enabling specialized real-time analysis while maintaining general search capabilities
Solution Approach 2:
The patent segments webpage content into distinct categories including price information, product attributes, and general content. Each segment is processed independently with appropriate indexing strategies, allowing price data to be tracked and analyzed separately from other webpage elements
2Reliability
If batch-based data ingestion processes are used to process large numbers of webpages, then comprehensive data collection is achieved, but real-time search and analysis are delayed
Solution Approach 1:
The system performs preliminary extraction and indexing of price and product information during the crawling phase, preparing data in advance for rapid querying. Price data is extracted and stored in a structured format with unique identifiers before full webpage processing completes, enabling faster real-time access
Solution Approach 2:
The patent implements continuous data ingestion and indexing processes that operate alongside batch processing. Crawlers continuously discover and index new price information as it appears on webpages, maintaining an up-to-date corpus without requiring complete batch reprocessing
3Productivity
If traditional crawlers prioritize webpages with overall content changes, then traffic-relevant pages are indexed, but pages with price changes are not prioritized
Solution Approach 1:
The system uses visual or structural markers to highlight price information on webpages, such as identifying price elements through specific HTML tags, CSS classes, or data attributes. This allows crawlers to detect price changes regardless of overall webpage content stability, prioritizing these pages for indexing
Data Source
AI summary
Price and product attributes from webpages are imported, indexed, analyzed, and made available to be searched in close-to realtime, allowing search for price changes specific to products on individual webpages and for products across all webpages as well as to identify longitudinal correlations between price changes and product attributes. Users may search the data and set alerts.