User Profile Cookie Sets for Web Crawler Data Collection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing systems face challenges in generating and maintaining accurate user profile cookie sets for simulating human website users, particularly due to expiration dates and varied user behavior, which complicates data collection and targeted advertising.

Innovation Solution

A system comprising cookie harvesting computing equipment, databases, and proxy servers that generate and maintain user profile cookie sets by selecting user profiles, obtaining site-specific segmentation data, and using web crawlers to load websites, allowing cookies to be placed and updated, thereby simulating human user behavior for data collection and analysis.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If web crawlers are used to gather data about publisher webpages, then data collection efficiency is improved, but obtaining accurate user profile cookie sets becomes difficult due to cookie expiration dates and varied user behavior

Engineering Contradiction:

Improvedata collection efficiencyVSAvoidaccuracy of user profile cookie sets

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent creates simulated user profiles that copy and replicate the characteristics of real human users, including their browsing patterns and cookie acceptance behaviors. These synthetic user profiles serve as digital twins that mimic real user interactions with websites, allowing web crawlers to collect data while maintaining realistic cookie sets without relying on actual user cookies that expire or vary unpredictably

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system generates and maintains its own user profile cookie sets autonomously through automated processes. The web crawlers interact with websites using these self-generated cookie sets, which are continuously updated and refreshed by the system itself, eliminating the need to rely on external sources for obtaining and maintaining valid user cookies

Inventive Principle:

Principle #25Self-service

2Adaptability or versatility

If cookies are used to simulate human users, then targeted advertising capability is improved, but maintaining valid cookie sets becomes challenging due to expiration dates

Engineering Contradiction:

Improvetargeted advertising capabilityVSAvoidvalidity period of cookie sets

Core Design Contradiction:

Adaptability or versatilityVSDuration of action of stationary object

Solution Approach 1:

The patent implements continuous generation and refreshing of user profile cookie sets through automated web crawling processes. Instead of relying on static, expiring cookies, the system continuously creates new simulated user profiles and updates their cookie sets, ensuring an uninterrupted supply of valid cookies for targeted advertising campaigns

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The system pre-generates and stores multiple user profile cookie sets in advance before they are needed for advertising campaigns. These pre-created cookie sets are maintained in a ready state, allowing immediate deployment for targeted advertising without waiting for cookies to expire or needing to collect new ones at the moment of use

Inventive Principle:

Principle #10Preliminary action

3Loss of information

If third party data gathering companies place cookies on user equipment, then user behavior data is collected, but simulating realistic human users becomes difficult due to varied user behavior patterns

Engineering Contradiction:

Improveuser behavior data collectionVSAvoidrealism of simulated users

Core Design Contradiction:

Loss of informationVSAdaptability or versatility

Solution Approach 1:

The patent creates user profiles with localized, site-specific characteristics tailored to different websites and contexts. Each simulated user profile has customized attributes and browsing patterns appropriate to the specific website being visited, making the simulation highly realistic for each local context rather than using generic, one-size-fits-all user models

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system employs dynamic user profiles that adapt and change their behavior patterns based on the website context, time, and other variables. These simulated users exhibit varied and evolving behaviors rather than static, predictable patterns, closely mimicking the natural variability of real human users across different browsing scenarios

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS10970743B2Systems and methods for generating and maintaining internet user profile data

Publication Date: 2021.04.06 PATHMATICS INC

AI summary

Systems and methods are provided for automatically generating and maintaining user profile cookie sets. The user profile cookie sets may be used by a web crawler when gathering data such as advertisement data associated with one or more websites. The cookie sets may be generated by choosing a user profile with a set of user traits, selecting a set of websites related to the user traits, and browsing the selected set of websites using a web crawler while allowing the website to place cookies in storage of the web crawler. The cookie sets may be maintained by selecting a website to browse, selecting a user profile associated with the selected website, loading a previously generated cookie set for the selected user profile into the storage of a web crawler, and loading the webpage while allowing the website to place, update, or replace cookies in the storage of the web crawler.