Data Collection System for Aggregating Unstructured Information
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies fail to efficiently process and analyze the vast amounts of information available on the Internet, making it difficult for users to locate, access, filter, and analyze data in a timely and comprehensive manner.
Innovation Solution
A web-based system and method that aggregates, analyzes, and displays data from multiple sources in one virtual location, allowing users to collect, store, and visualize both historic and real-time data, with customizable graphical representations and data sharing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If current search engine technology is used to locate information, then users can find pertinent information at discrete intervals, but users cannot efficiently process and analyze the volumes of identified information, and must repeatedly perform searches for current data
Solution Approach 1:
The system performs preliminary actions by automatically collecting, aggregating, and organizing data from multiple sources in advance. Data collection agents continuously gather information from specified sources, and the system pre-processes this data through filtering, validation, and organization before users need it, eliminating the need for repeated searches and manual processing
Solution Approach 2:
The patent introduces an intermediary system between data sources and users. This intermediary automatically collects data from multiple sources, aggregates it in a centralized database, and presents it in organized formats. The intermediary handles the complexity of data collection and processing, allowing users to access processed information without directly interacting with multiple data sources
2Loss of information
If users manually search for information from multiple sources, then they can locate relevant information, but they face increased difficulty in efficiently aggregating, analyzing, storing, and processing the copious amounts of available relevant information
Solution Approach 1:
The system merges multiple data collection functions into a unified automated system. Data collection agents from various sources are consolidated into a single platform that aggregates information centrally. The system combines filtering, validation, storage, and analysis functions into an integrated workflow, reducing the complexity users would face when manually managing multiple information sources
Solution Approach 2:
The system performs self-service by automatically collecting, filtering, validating, and organizing data without user intervention. The data collection agents autonomously navigate sources, extract relevant information, and the system automatically processes this data through validation rules and organizational structures, freeing users from manual data aggregation tasks
3Ease of operation
If data is collected from multiple specified data sources in one virtual location, then users can access data centrally, but the system must manage multiple data sources and convert unstructured data into structured data
Solution Approach 1:
The system segments data management into distinct modular components: data collection agents that handle specific sources, data validation modules that process different data types, and storage systems that organize information by category. This segmentation allows the system to manage complexity internally while presenting a simple unified interface to users for accessing data from multiple sources
Data Source
AI summary
A system and method for collecting and processing data over a communications network. A data mining marshaller module associates each plugin to a particular data source and manages the plugin to periodically retrieve unstructured data from the data source based on a plurality of data items to be monitored on behalf of a plurality of users. The plugins convert unstructured data received from the data sources to structured data and the data marshaller module stores the structured data in a database. This enables the system and method to aggregate and display the structured data in multiple graphical representations according to the user's preference.