Cognitive Tool Performance Evaluation via Log Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evaluating the performance of question answering cognitive computing tools is challenging due to the difficulty in representing user questions, parsing log files, and correlating performance with other factors, leading to short-sighted assessments and limited measurement of usage and engagement.
Innovation Solution
A system and method that analyzes log files to determine question validity and answer accuracy, using text analytics to categorize questions and format data for visual representation as performance metrics, enabling better understanding of tool usage and training needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If manually reviewed log data is used for evaluation, then detailed user interaction information is obtained, but the data is difficult to parse and understand leading to short-sighted assessments
Solution Approach 1:
The patent introduces an intermediary processing system that automatically parses log files and transforms raw data into structured, visual performance metrics. This intermediary layer bridges the gap between detailed raw log data and human-understandable performance assessments, eliminating the need for manual review while preserving all user interaction information.
Solution Approach 2:
The patent replaces the mechanical manual review process with automated computational systems that parse log files, analyze performance data, and generate visual metrics. This substitution eliminates human labor requirements while improving the comprehensiveness and accuracy of performance evaluations.
2Ease of manufacture
If carefully prepared questions are used for evaluation, then the evaluation process is structured and manageable, but the questions may not fully represent actual user questions
Solution Approach 1:
The patent enables the evaluation system to automatically process and analyze actual user questions from log files without requiring manual curation. The system self-services by extracting, categorizing, and evaluating real user interactions, thereby maintaining both structural manageability and authentic representation of user queries.
Solution Approach 2:
The patent performs preliminary automated processing of log data, including parsing, validation, and categorization of actual user questions before evaluation. This preliminary action prepares the data in a structured manner while preserving the authenticity of real user queries, eliminating the need for manual question preparation.
3Measurement precision
If performance evaluation focuses on individual metrics, then specific performance aspects are measured accurately, but the overall performance picture remains limited and short-sighted
Solution Approach 1:
The patent merges multiple individual performance metrics into a comprehensive visual dashboard that presents both specific measurements and overall performance context simultaneously. By combining detailed question accuracy, user engagement, and system response metrics into an integrated view, the system preserves both precision and holistic understanding.
Solution Approach 2:
The patent adds visual and contextual dimensions to performance data by presenting metrics in graphical formats that show relationships between different performance aspects. This dimensional transformation allows users to see both individual metric precision and overall performance patterns that would be invisible in tabular form.
4Loss of information
If detailed log data is collected for comprehensive analysis, then complete user interaction information is captured, but it becomes difficult to identify trends and correlations
Solution Approach 1:
The patent extracts key performance indicators and meaningful patterns from comprehensive log data, separating essential information from raw data complexity. By extracting only the most relevant metrics and trends while maintaining complete data collection, the system reduces analysis complexity without losing important information.
Solution Approach 2:
The patent creates simplified visual representations and summaries that copy the essential patterns from detailed log data in an easily analyzable format. These visual copies preserve the underlying trends and correlations while presenting them in a form that is simple to interpret and analyze.
Data Source
AI summary
A system and method for evaluating the performance and usage of a cognitive computing tool which answers questions from users. A log file for these interactions includes the questions, the answers and a confidence rating assigned by the tool to each answer. Questions and answers are analyzed to determine validity, accuracy, and categories by subject matter experts or text analytics tools, and the results are added to the log file. Comments and sentiments from users may be analyzed and added to the log file. Additional data about the users, such as identities, demographics, and locations, may be added. Data from the log file may be presented in a dashboard display as metrics, such as trends and comparisons, describing the usage and performance of the cognitive computing tool. Answers may be displayed as they were presented to the users. Selectable filters may be provided to control the data displayed.