Sequence Prediction for Enterprise Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Searching through enterprise-scale data stored in distributed datasets across multiple databases with different hierarchies is challenging, as existing methods like TF-IDF often fail to accurately distinguish between similar query strings, reducing the accuracy and usefulness of search results.
Innovation Solution
A computer system generates a predicted sequence by using probabilistic operations based on appearance counts of n-grams in metadata, computing weighted ratios, and selecting additional words to improve search accuracy, thereby reducing irrelevant results without impairing database search performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If TF-IDF technique is used for data retrieval, then search speed is improved, but search accuracy deteriorates
Solution Approach 1:
The system pre-computes and stores n-gram appearance counts and probability scores for all possible word sequences in the dataset metadata before actual search queries are executed. This preliminary processing enables the search engine to quickly retrieve pre-calculated probability information during queries, avoiding real-time computation of sequential patterns while maintaining high accuracy in distinguishing between different query meanings.
2Measurement precision
If sequence prediction with probabilistic operations is implemented, then search accuracy is improved, but computational complexity increases
Solution Approach 1:
The system pre-computes and stores n-gram appearance counts and probability scores for all possible word sequences in the dataset metadata before actual search queries are executed. This preliminary processing enables the search engine to quickly retrieve pre-calculated probability information during queries, avoiding real-time computation of sequential patterns while maintaining high accuracy in distinguishing between different query meanings.
Solution Approach 2:
The system creates a simplified probabilistic model that copies only the essential sequential pattern information (n-gram appearance counts and probability scores) from the full dataset metadata. This copied model is stored separately and used during search operations, reducing the computational burden while preserving the key sequential relationships needed for accurate search results.
3Measurement precision
If appearance count calculation is performed for every query, then prediction accuracy is improved, but search performance deteriorates
Solution Approach 1:
The system pre-computes and stores n-gram appearance counts and probability scores for all possible word sequences in the dataset metadata before actual search queries are executed. This preliminary processing enables the search engine to quickly retrieve pre-calculated probability information during queries, avoiding real-time computation of sequential patterns while maintaining high accuracy in distinguishing between different query meanings.
Data Source
AI summary
A method includes obtaining an input sequence and first and second word sequences of metadata of a plurality of datasets. The method also includes determining a set of scores including first scores and second scores by determining first scores by determining a score of the first scores based on a count of the word sequence of the first word sequences in the metadata without determining a count of the input sequence. The method also includes determining second scores by determining a score of the second scores based on a count of a second sequence of the second word sequences in the metadata. The method also includes determining an additional word for a predicted sequence including the input sequence based on the set of scores and retrieving a set of datasets based on the predicted sequence.