Big Data Search Relevance Scoring via Preliminary Annotation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data repositories are not effectively searchable using text queries due to the lack of relevant annotations and generic methods for ranking data records, making it difficult to identify statistically significant findings across various data sources and types of analyses.
Innovation Solution
The system integrates diverse biological knowledge bases, annotation text, and statistics to generate findings, allowing for text queries to retrieve relevant results by associating text with electronic records and calculating relevance scores across different data sources, enabling the identification of subsets matching user queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If text search is applied to large data repositories, then text queries can be performed, but the data records lack relevant annotations and cannot be usefully connected to search terms
Solution Approach 1:
The system pre-calculates and stores relevance scores for all data records against all search terms before actual queries are executed. This preliminary action creates a ready-made relevance index that enables immediate text search without losing relevance information during query execution.
Solution Approach 2:
The system divides the large data repository into manageable subsets and performs automatic computation on each subset to generate relevance scores. This segmentation allows the system to process and annotate data records in chunks, making the overall task feasible while maintaining comprehensive coverage.
2Adaptability or versatility
If all data records are connected with relevant terms, then text search coverage is improved, but numerous data records receive the same relevance score making ranking useless
Solution Approach 1:
The system applies different relevance scoring functions to different data records based on their specific characteristics and the nature of their connection to search terms. Each record receives a customized relevance score calculated from its unique attributes, ensuring that no two records receive identical scores unless they are truly equivalent in relevance.
Solution Approach 2:
The system transforms the raw matching data into normalized relevance scores by changing the parameter scale and applying statistical normalization. This parameter transformation ensures that relevance scores are distributed across a meaningful range, enabling effective ranking while maintaining comprehensive search coverage.
3Ease of manufacture
If generic methods are used for relevance score calculation, then computation is simple, but the text association lacks usefulness for ranking
Solution Approach 1:
The system introduces an intermediary relevance scoring layer that sits between the raw data and the search query interface. This intermediary automatically computes meaningful relevance scores using structured methods that bridge the gap between simple computation and useful ranking, transforming basic data matches into actionable search results.
4Speed
If large data repositories are queried at a low level with threshold checks, then query processing is fast, but the data records are not usefully connected to important annotations
Solution Approach 1:
The system pre-computes and stores annotation information and relevance scores for all data records before queries are executed. This preliminary annotation process ensures that when fast threshold queries are performed, the relevant annotation information is already attached to the data records, eliminating the need for slow post-query annotation processing.
Data Source
AI summary
Systems and methods are provided for ranking and querying statistical analysis results of database contents for the purpose of populating search engine query results with novel content.