Bayesian Distributed Query Processing Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Bayesian distributed search systems face challenges in achieving accurate relevance scoring across multiple servers due to the use of global scaling factors, which can inaccurately reduce the scores of highly relevant documents, leading to lumping good scores with bad ones and being temperamental based on the number of documents indexed on each server.
Innovation Solution
The system generates partial Bayes calculation data on remote servers and combines it centrally to produce global Bayes calculation data, which is then sent back to remote servers for local Bayesian query calculations, ensuring accurate relevance ranking and merging of results to provide a final list of most relevant documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If global scaling factors are used in Bayesian distributed search systems, then the system can handle large volumes of documents across multiple servers, but the relevance scores of highly relevant documents are inaccurately reduced and good scores are lumped with bad ones
Solution Approach 1:
The patent divides the Bayesian calculation into two segments: local calculations performed at each remote server and global calculations performed at the central server. Each remote server calculates local term statistics independently, then the central server aggregates this data to compute global term statistics. This segmentation allows the system to handle large document volumes while maintaining score accuracy by avoiding the application of global scaling factors to local results.
Solution Approach 2:
The patent introduces partial Bayes calculation data as an intermediary structure that bridges local and global calculations. This intermediary contains the necessary statistics from local servers and is used by the central server to compute global Bayes calculation data, which then guides local relevance calculations without requiring direct global scaling factor application.
2Adaptability or versatility
If distributed query processing is implemented to handle large datasets, then system scalability is improved, but achieving consistent and accurate relevance scoring across all servers becomes difficult
Solution Approach 1:
The patent merges local Bayes calculation data from multiple remote servers at the central server to produce unified global Bayes calculation data. This merging process aggregates term frequency and document count statistics across all servers, ensuring that relevance calculations reflect the complete corpus while maintaining distributed processing architecture for scalability.
Solution Approach 2:
The patent changes the parameters used in Bayesian calculations from local-only statistics to global statistics that incorporate data from all servers. By updating term frequency and document count parameters with global data, the system achieves consistent relevance scoring across distributed servers while maintaining the scalability benefits of distributed architecture.
3Device complexity
If all document indexing and query processing is done on a central server, then relevance scoring is simplified, but the system is not scalable and network traffic increases excessively
Solution Approach 1:
The patent segments query processing into two phases: local processing at remote servers for document matching and global processing at the central server for statistical aggregation. This segmentation allows remote servers to independently process queries using local data while the central server handles coordination and global statistics, achieving both simplicity and scalability.
Solution Approach 2:
The patent applies partial query processing at remote servers, where only the necessary local calculations are performed rather than complete processing. This partial action approach reduces the workload on the central server and enables scalability, while the central server still performs essential global coordination to ensure accurate relevance scoring.
Data Source
AI summary
System and methodology for performing Bayesian-based distributed query processing is provided that solves the problem of how to get each server participating in a Bayesian distributed search system to return the same accurate relevance score for different documents. By performing calculations in a two-step process, accurate Bayesian calculation results are obtained whilst distributing the document indexing and query processing.