Deep Miner for Enterprise Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database search tools are limited in their ability to access data flexibly and often require expert-level knowledge, struggling to find relevant information across multiple databases of varying types and complexity, especially as database environments grow and become more heterogeneous.
Innovation Solution
The development of 'deep miners' that can directly identify relevant columns within data sources, allowing for seamless integration of structured and unstructured data sources, and executing queries across multiple databases without requiring explicit table selection, thereby reducing the computational burden and improving search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional database search tools are used, then database access is possible, but the search reach is limited and relevant data is often missed
Solution Approach 1:
The system performs preliminary actions by pre-building and maintaining a comprehensive search index that maps multiple data sources, columns, and data types before actual search queries are executed. This index is continuously updated as data changes, enabling the search system to immediately access relevant information across all data sources without requiring manual exploration or expert knowledge during the search process.
Solution Approach 2:
The invention introduces an intermediary search indexing system that acts as a mediator between users and the complex database environment. This intermediary layer translates user queries into optimized search operations across multiple data sources, handling the complexity of data mapping, type conversion, and source routing automatically, thereby extending search reach without requiring users to understand the underlying database structure.
2Measurement precision
If expert level knowledge of database organization is required, then accurate search results can be obtained, but the complexity of operation increases
Solution Approach 1:
The search system performs self-service by automatically discovering, indexing, and maintaining metadata about all available data sources, columns, and data types. The system autonomously handles query optimization, data type conversion, and source selection without requiring user expertise. Users simply provide their search criteria, and the system independently manages the complex operations needed to retrieve accurate results from heterogeneous data sources.
3Reliability
If multiple databases are searched individually, then each database can be accessed with appropriate expertise, but the time and resources required increase
Solution Approach 1:
The invention merges multiple separate database search operations into a single unified search operation. The search indexing system consolidates metadata from multiple data sources into a centralized structure that can be queried simultaneously. When a user submits a search query, the system executes a single search operation that automatically traverses all indexed data sources in parallel, returning combined results from all databases without requiring multiple sequential searches or separate expertise for each database.
4Quantity of substance
If database environments increase in size and complexity, then more data becomes available, but the ability to address particular needs decreases
Solution Approach 1:
The system segments the large and complex database environment into manageable units through hierarchical indexing. Data is organized and indexed at multiple levels (e.g., database level, table level, column level, row level), allowing the search system to efficiently navigate and query specific segments based on user needs. This segmentation enables the system to handle vast quantities of data by breaking it down into smaller, manageable units that can be searched independently and combined as needed.
Data Source
AI summary
Methods and apparatus are disclosed for deep mining of data sources. A deep miner provides extended reach into available structured databases and/or unstructured data sources. Direct evaluation of columns for relevance to a client query provides a wider array of columns having potential relevance, compared to conventional tools relying on table evaluation. Direct column evaluation is extended to unstructured data sources. A broad interface extends the reach of search seamlessly across a wide range of structured and unstructured data sources. Disclosed techniques provide superior results with reduced computing resource utilization. Limitations of human expertise are overcome. Further efficiencies are achieved through caching, ranking of columns or results, search refinement, and customized responses.