Self-Orchestrated Data Extraction System for High Throughput Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High throughput data acquisition and analysis from external sources is time-consuming and resource-intensive due to vast quantities and varieties of data, making it difficult for users to efficiently find and filter relevant information.
Innovation Solution
A self-orchestrated system for extracting, analyzing, and presenting entity data by extracting web pages to object-based storage, parsing content using regular expressions, and passing metadata and content to an advanced analysis function decider for analysis, followed by streaming data to a JSON file for flattening, enabling efficient data collection and filtering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is acquired from external sources in high throughput, then the quantity of data increases, but the time and resources required for analysis increase
Solution Approach 1:
The patent segments the data processing workflow into distinct modular components: web page extraction, metadata extraction, content pulling, parsing with RegEx, advanced analysis, and JSON flattening. Each module handles a specific aspect of data processing independently, enabling parallel processing and reducing overall analysis time while maintaining high data throughput
Solution Approach 2:
The system performs preliminary actions by extracting and storing web page metadata separately before full content analysis. The metadata is extracted to object-based storage with globally unique identifiers, and web page content is pulled to a content store in advance, allowing the parsing and analysis stages to work with pre-prepared data structures, thus reducing processing time
2Loss of information
If data from webpages is collected and presented, then information availability increases, but access complexity increases for users
Solution Approach 1:
The patent extracts only the necessary metadata and specific content from web pages using regular expressions and advanced analysis functions. Instead of presenting entire web pages or all available data, the system extracts and presents only the relevant entity data, simplifying user access while maintaining comprehensive information availability
Solution Approach 2:
The system introduces an intermediary processing layer between web page collection and user presentation. This layer includes the advanced analysis function decider and JSON flattening processes that transform complex web page structures into simplified, standardized data formats, making the data accessible without exposing users to the underlying complexity
3Productivity
If web page metadata and content are extracted and analyzed, then data processing capability improves, but system resource consumption increases
Solution Approach 1:
The patent applies partial action by using regular expressions to extract only specific patterns and entities from web page content rather than processing and analyzing every piece of data. The advanced analysis function decider selectively applies analysis based on the extracted metadata and content, avoiding unnecessary processing of irrelevant data, thus improving productivity while controlling resource consumption
Data Source
AI summary
A method for operating a self-orchestrated system for extraction, analysis, and presentation of entity data involves extracting a web page to object-based storage including web page content, web page metadata and a globally unique identifier. The method extracts the web page metadata from the object-based storage. The method inputs the web page metadata to a queue. The method pulls web page content from a content store. The method receives RegEx from a model parameter store. The method parses the web page content using RegEx and web page metadata. The method passes web page metadata and extracted content from the web page and positions of extracted content to an advanced analysis function decider (AAF Decider) for analysis. The method streams web page metadata and extracted content from the web page and positions of the extracted content to a JSON file batch for flattening.