Adaptive Data Mining Query Expansion for Unindexed Sources

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data mining technologies are limited in analyzing real-time data across multiple sources, particularly failing to extract relevant information from unindexed web pages and proprietary data silos, which restricts their ability to provide comprehensive and timely insights.

Innovation Solution

A method and apparatus for generating and expanding queries based on topics of interest, executing them across various data sources, including both open-source and proprietary data, and monitoring selected sources for matches, enabling real-time data extraction and analysis.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If typical search engines are used to query data, then structured data sources can be searched, but unindexed web pages and proprietary data silos cannot be accessed

Engineering Contradiction:

Improvedata source coverageVSAvoidunindexed data accessibility

Core Design Contradiction:

Adaptability or versatilityVSLoss of information

Solution Approach 1:

The system implements a universal data extraction platform that can access multiple types of data sources (indexed web pages, unindexed web pages, proprietary data silos, social media, web feeds) through a single integrated architecture, eliminating the limitation of search engines being restricted to only structured indexed data

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Productivity

If traditional data mining approaches are used, then batch processing can be performed, but real-time data analysis cannot be achieved

Engineering Contradiction:

Improvedata analysis speedVSAvoidreal-time response delay

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-configuring query templates, expanding search terms in advance, and establishing monitoring mechanisms for selected data sources before real-time analysis is needed, enabling immediate execution and monitoring of data extraction operations when required

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system maintains continuous monitoring of selected data sources for query matches, ensuring uninterrupted real-time data extraction and analysis capabilities rather than periodic or batch processing, thereby eliminating time delays in detecting relevant information

Inventive Principle:

Principle #20Continuity of useful action

3Productivity

If query search terms are kept simple, then execution speed is maintained, but comprehensive data extraction from multiple sources is limited

Engineering Contradiction:

Improvequery execution speedVSAvoidsearch term comprehensiveness

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The system segments the query processing into distinct components: base query generation, search term expansion, and execution. By separating these functions, the system can expand search terms comprehensively without compromising the execution speed of the core query across multiple data sources

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10114870B2Real-time and adaptive data mining

Publication Date: 2018.10.30 RULE 14 LLC

AI summary

A method of analyzing data is presented. The method includes generating a query based on a topic of interest, expanding search terms of the query, executing the query on one or more data sources, monitoring a specific data source selected from the one or more data sources. The monitoring is performed to monitor for matches to the query.