Web Crawler Checkout Page Access via Shopping Cart Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web crawlers face difficulties in accessing e-commerce checkout pages as they typically require a product to be added to a shopping cart, limiting their ability to index and validate elements on these pages.
Innovation Solution
A system and method that identifies x-paths specific to e-commerce platforms, allowing a web crawler to automatically add a product to a digital shopping cart and access the checkout page, enabling validation of elements on the checkout page.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a web crawler uses traditional hyperlink following to navigate e-commerce sites, then it can access publicly visible pages, but it cannot access checkout pages which require a product to be in the shopping cart
Solution Approach 1:
The web crawler performs preliminary actions by automatically adding products to the shopping cart before attempting to access the checkout page. This preliminary setup of the shopping cart state enables the crawler to reach previously inaccessible checkout pages without requiring manual user interaction.
Solution Approach 2:
The system employs self-service by using automated bot accounts to perform shopping cart operations independently. The crawler manages its own navigation by automatically completing the prerequisite action of adding items to the cart, eliminating the need for human users to manually set up the shopping cart state.
2Reliability
If a web crawler manually adds products to shopping carts to access checkout pages, then it can validate checkout page elements, but the process becomes time-consuming and resource-intensive
Solution Approach 1:
The bot account system serves multiple functions: it acts as both a shopping cart manager (adding products) and a navigation tool (accessing checkout pages). This multi-functionality allows a single automated account to handle both the prerequisite shopping cart setup and the subsequent page validation, improving overall crawling productivity.
Solution Approach 2:
The bot account serves as an intermediary between the web crawler and the e-commerce platform. It mediates the interaction by automatically managing shopping cart state and enabling access to protected pages, thereby facilitating efficient validation without requiring direct human intervention or multiple separate systems.
3Reliability
If e-commerce sites require products to be in the shopping cart before accessing checkout pages, then user experience is protected, but web crawlers cannot index or validate these pages
Solution Approach 1:
The web crawler creates a virtual copy of the user shopping experience by using bot accounts to replicate the actions of adding products to carts and accessing checkout pages. This copying approach allows the crawler to access and validate checkout page content without affecting real user experiences, maintaining the integrity of the user interface while enabling comprehensive indexing.
Data Source
AI summary
A computer system identifies a platform that corresponds to a first resource, wherein the first resource includes one or more resource pages. The computer system determines one or more x-paths that corresponds to the identified platform. The computer system utilizes at least a first x-path of the one or more x-paths to add a product to a digital shopping cart associated with the first resource. In response to adding the product to the digital shopping cart associated with the first resource, the computer system utilizes at least a second x-path of the one or more x-paths to access a first resource page of the one or more resource pages, wherein the first resource page is a checkout page. The computer system determines whether one or more elements located within the first resource page are valid.