Web Data Joining Spreadsheet Using Input-Output Examples
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for joining web data with relational data in spreadsheets are cumbersome, requiring users to manually navigate websites, write complex scripts, and handle diverse data formats, posing challenges due to the lack of direct interfaces and varying data structures.
Innovation Solution
A system that decomposes the integration task into URL learning and data extraction, using domain-specific languages and hierarchical search to generate URLs and extract data efficiently from web pages, allowing users to provide examples for learning programs that automate the process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If users manually navigate websites and write complex scripts to extract data, then data integration can be performed, but the operation complexity and time consumption increase significantly
Solution Approach 1:
The system enables self-service data extraction by automatically generating and executing extraction logic based on user-provided examples. The system learns patterns from sample inputs and outputs, then autonomously performs data extraction from web pages without requiring users to manually write or navigate through complex extraction scripts.
Solution Approach 2:
The patent introduces an intermediary learning system that acts as a mediator between user intent and web data extraction. This learning intermediary translates user-provided examples into automated extraction logic, eliminating the need for users to directly interact with complex web scraping mechanisms while maintaining reliable data integration.
2Quantity of substance
If users write complex extraction scripts based on DOM structure, then data can be retrieved from web pages, but the script complexity and maintenance difficulty increase
Solution Approach 1:
The patent replaces manual script-writing mechanics with automated learning mechanics. Instead of users mechanically constructing extraction scripts based on DOM structure analysis, the system uses machine learning to automatically generate extraction logic from examples, substituting complex manual programming with automated pattern recognition.
Solution Approach 2:
The system learns extraction patterns by copying successful examples provided by users. Rather than requiring users to create complex extraction scripts from scratch, the system replicates and generalizes patterns from sample inputs and outputs, automatically adapting the learned patterns to extract data from similar web pages.
3Adaptability or versatility
If data is transformed to match different formats before joining, then spreadsheet and web data can be integrated, but the processing time and computational resources increase
Solution Approach 1:
The system performs preliminary learning of data format patterns during the training phase using user-provided examples. By pre-learning the transformation rules and format mappings from examples, the system prepares extraction logic that inherently produces data in the required format, eliminating the need for time-consuming post-extraction transformation operations.
Data Source
AI summary
Provided are methods and systems for joining semi-structured data from the web with relational data in a spreadsheet table using input-output examples. A first sub-task performed by the system learns a string transformation program to transform input rows of a table to URL strings that correspond to the webpages where the relevant data is present. A second sub-task learns a program in a rich web data extraction language to extract desired data from the webpage given the example extractions. Hierarchical search and input-driven ranking are used to efficiently learn the programs using few input-output examples. The learnt programs are then run on the remaining spreadsheet entries to join desired data from the corresponding web pages.