Adaptive Data Mining Query Expansion for Real-Time Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data mining technologies are limited in analyzing real-time data across multiple sources, particularly due to their inability to search non-indexed web pages and proprietary data silos, which restricts their effectiveness in providing comprehensive and timely insights.
Innovation Solution
A method and apparatus for generating and expanding queries based on topics of interest, executing them across various data sources, including both open-source and proprietary data, and monitoring selected sources for matches, enabling real-time data extraction and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If typical search engines are used to query data, then structured data sources can be searched, but real-time analysis of multiple data points and non-indexed web pages cannot be performed
Solution Approach 1:
The patent implements a universal data extraction system that can handle multiple data source types (indexed web pages, non-indexed web pages, proprietary data silos, social media, web feeds) through a single platform. The system uses configurable data extractors that can be deployed to different data sources and adapted to various extraction requirements, making the system versatile across structured and unstructured data environments while maintaining real-time processing capabilities.
Solution Approach 2:
The patent introduces data extractors as intermediary components that bridge the gap between different data sources and the analysis system. These extractors act as mediators that can be deployed to various data sources (including non-indexed sources) and retrieve data in real-time, translating diverse data source formats into a unified structure that the analysis system can process, thereby enabling real-time analysis across previously inaccessible data sources.
2Loss of information
If query expansion is performed to improve search coverage, then more comprehensive data can be extracted, but system complexity increases
Solution Approach 1:
The patent implements query expansion by performing preliminary actions - expanding search terms and generating related queries before the actual data extraction process. The system expands queries using techniques such as synonym generation, related term identification, and hierarchical topic expansion, creating a set of pre-expanded queries that are then executed against multiple data sources. This preliminary expansion ensures comprehensive information extraction while managing complexity by organizing the expansion process in a structured, pre-planned manner.
Data Source
AI summary
A method of analyzing data is presented. The method includes generating a query based on a topic of interest, expanding search terms of the query, executing the query on one or more data sources, monitoring a specific data source selected from the one or more data sources. The monitoring is performed to monitor for matches to the query.