Automated Financial Data Aggregation Script Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current account aggregation systems require human intervention for generating scripts to extract financial data from financial institution websites, which is time-consuming and inefficient, and fail to automatically adjust to changes in website structures.
Innovation Solution
An automated system that generates site-specific scripts for extracting financial data by crawling the website, categorizing web pages and segments using statistical analysis and machine learning techniques, and updating scripts without human input to adapt to changes in website structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If manual script generation is used to extract financial data from websites, then script accuracy can be maintained, but the process becomes time-consuming and requires continuous human intervention
Solution Approach 1:
The system performs self-service by automatically generating and maintaining extraction scripts through web crawling and statistical analysis, eliminating the need for manual human intervention in script creation and updates
Solution Approach 2:
The system performs preliminary actions by pre-crawling websites and building statistical models in advance, so that when website structures change, the system already has the necessary data and models to quickly adapt without requiring manual script rewriting
2Productivity
If manual updates are performed for website structure changes, then script reliability is maintained, but productivity decreases due to continuous human intervention
Solution Approach 1:
The system implements feedback by continuously monitoring website structures through crawling and comparing them against existing statistical models, automatically detecting changes and triggering script regeneration to maintain reliability without manual intervention
Solution Approach 2:
The system applies dynamics by making the extraction scripts adaptive and flexible through statistical analysis, allowing them to automatically adjust to website structure changes while maintaining extraction reliability through probabilistic matching rather than rigid fixed patterns
3Extent of automation
If statistical analysis and machine learning are used for web page categorization, then automation is improved, but system complexity increases
Solution Approach 1:
The system segments the complex task of web page categorization into distinct components: web crawling, statistical analysis of page structures, machine learning classification, and script generation. This modular segmentation manages complexity by handling each function separately rather than as a monolithic system
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data aggregation. The methods, systems, and apparatus include determining whether a site-specific script for extracting financial data from a particular financial institution website is available; in response to determining that a site-specific script for extracting financial data from the particular financial institution website is not available, generating a site map of web pages and web page segments in the financial institution website, wherein the site map is generated based on at least in part on a statistical analysis of web pages and web page segments that are not in the financial institution website; generating, based on the site map of the financial institution website, a site-specific script for extracting financial data from the financial institution website; and extracting, for one or more users, financial data from the particular financial institution website using the generated site-specific script.