LLM Inquiry Classification With Certainty-Based Answer Output

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies struggle to accurately respond to user inquiries in services due to limitations in handling unknown inquiries and relying on predefined categories or tags, leading to reduced answer accuracy.

Innovation Solution

An inquiry answering system that utilizes a large language model to acquire and analyze user inquiries, classify them based on predefined information, and generate model answers with a certainty degree, adjusting output based on this certainty to enhance accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If FAQ information is presented based on predefined categories or tags, then the system can provide structured support, but it cannot respond to unknown inquiries

Engineering Contradiction:

Improveresponse capabilityVSAvoidanswer accuracy

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent introduces an intermediary device that bridges the gap between predefined FAQ categories and user inquiries. This intermediary analyzes the semantic meaning of user questions and matches them with appropriate FAQ items, enabling the system to respond to both known and unknown inquiries while maintaining answer accuracy through confidence degree assessment.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system changes the parameter of inquiry classification from rigid category-based matching to flexible semantic analysis with confidence degree evaluation. By introducing confidence degree as a new parameter, the system can dynamically adjust its response strategy based on how well the inquiry matches available FAQ information.

Inventive Principle:

Principle #35Parameter changes

2Productivity

If chat content is used for advertisement insertion, then the conversation can be monitored, but the accuracy of responding to user inquiries cannot be fully increased

Engineering Contradiction:

Improveservice efficiencyVSAvoidinquiry understanding accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent replaces traditional keyword-based chat analysis with semantic analysis technology. This substitution enables the system to understand the true meaning of user inquiries rather than merely matching keywords, significantly improving inquiry understanding accuracy while maintaining the ability to monitor conversations for service purposes.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Ease of operation

If predefined classification information is used, then the system can organize inquiries systematically, but it cannot accurately classify ambiguous or novel inquiries

Engineering Contradiction:

Improveinquiry organizationVSAvoidclassification accuracy

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The system implements feedback mechanisms where the intermediary device continuously learns from classification outcomes. When confidence degree is low, the system can seek additional information or escalate to human operators, and this feedback loop improves the classification accuracy of ambiguous and novel inquiries over time while maintaining systematic organization.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS20260064730A1Inquiry answering system, inquiry answering method, and information storage medium

Publication Date: 2026.03.05 RAKUTEN GROUP INC

AI summary

Provided is an inquiry answering system including at least one processor configured to: acquire inquiry information relating to an inquiry from a user in a predetermined service; acquire classification information relating to a classification that relates to the inquiry and that is defined in advance in the predetermined service; input the inquiry information and the classification information to a large language model to acquire a model answer relating to the classification and a certainty degree of the classification which are generated by the large language model; and control output of the model answer based on the certainty degree.