Cross-Schema Document Similarity Scoring With Field Mappings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document repositories struggle to effectively identify documents that are semantically similar across different schemas, as existing methods focus on syntactic similarity and fail to intelligently map language from specific fields in the source document to corresponding fields in documents with different schemas, often missing key connections between ideas.
Innovation Solution
A system that uses defined configurations to generate intelligent queries, allowing for semantic similarity searches by identifying document schemas, generating queries based on field mappings, and aggregating weighted scores to provide accurate similarity scores across different document schemas, with the ability to learn from user feedback to improve search performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If syntactic similarity methods are used to compare documents, then the search process is simple and fast, but the accuracy of identifying semantically similar documents across different schemas is poor
Solution Approach 1:
The system performs preliminary actions by pre-defining field mappings between different document schemas and pre-generating query templates. When a search is executed, the system retrieves these pre-established mappings and templates, then applies them to the current search query. This eliminates the need for complex real-time analysis of schema relationships, thereby improving semantic similarity accuracy without proportionally increasing system complexity.
Solution Approach 2:
The patent introduces an intermediary layer consisting of predefined field mappings and query templates that mediate between the search query and the actual document comparison process. This intermediary translates semantic concepts into structured field mappings, allowing the system to achieve accurate semantic similarity identification across different schemas while maintaining manageable system complexity through reusable templates.
2Adaptability or versatility
If simple syntactic comparison is used, then the system complexity is low, but the ability to map language between different schema fields is insufficient
Solution Approach 1:
The system implements universality by creating a unified field mapping framework that can handle multiple document schemas through a common set of query templates and mapping rules. The same infrastructure serves different schema types (e.g., incident reports, service requests, change requests) by retrieving appropriate pre-defined mappings, thereby achieving high adaptability across schemas without requiring separate complex systems for each document type.
Solution Approach 2:
The system achieves versatility through parameter changes by dynamically selecting and applying different field mapping configurations based on the document schema being searched. The underlying query generation mechanism remains consistent, but the specific field mappings and weights are adjusted according to the schema type, allowing the system to adapt to different document structures without changing the core architecture.
3Reliability
If user feedback is collected and used to improve search, then search performance improves over time, but the system requires additional feedback collection and model training mechanisms
Solution Approach 1:
The system implements feedback mechanisms by collecting user interactions (such as document selection, viewing time, and explicit ratings) and using this feedback to retrain the machine learning model. The feedback loop continuously improves search performance by adjusting field weights and mapping priorities based on actual user behavior, while the automated retraining process manages the complexity of incorporating feedback mechanisms.
Solution Approach 2:
The system applies self-service by implementing automated model retraining that uses collected feedback to automatically adjust search parameters and field mappings. The machine learning model autonomously learns from user interactions and improves search performance without requiring manual intervention for each adjustment, thereby managing the complexity of the feedback system through automation.
Data Source
AI summary
A document repository may be searched for documents that are similar to a source document. Multiple queries may be generated based on a type of the source document, and the results may be combined in a unified response. User behavior may then be monitored, and implicit and explicit feedback may be gathered to evaluate the performance of the search. The gathered feedback may indicate how relevant each of the result documents are in comparison to the original source document. This feedback may then be used to adjust search parameters for the source document type, such that the performance of subsequent searches may be improved. A model may also be trained to classify implicit feedback using explicit feedback received from users.