Hierarchical Intent Labeling for Consistent ML Training Data

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing intent labeling systems face challenges with large and complex intent spaces, leading to ambiguity and choice overload, where data labelers struggle to uniquely identify intent labels, resulting in the same data being labeled differently.

Innovation Solution

An intent ontology is used to generate intent labels composed of multiple actions or entities subject to a labeling flow hierarchy, presenting a subset of options at a time based on user selection, reducing available choices and improving readability through ordered intent labels.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a large intent space with hundreds or thousands of candidate intents is provided to labelers, then the system can cover more diverse user intents, but labelers experience choice overload and ambiguity, resulting in inconsistent labeling

Engineering Contradiction:
Improveintent coverageVSAvoidlabeling consistency
Core Design Contradiction:
Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent segments the large intent space into a hierarchical structure with parent intents at higher levels and child intents at lower levels. Labelers first select from parent intents, then progressively refine to child intents. This segmentation reduces the number of options presented at each decision point, eliminating choice overload while maintaining comprehensive intent coverage across the entire hierarchy.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent introduces a hierarchical dimension to the intent selection process. Instead of presenting all intents in a single flat list, the system organizes intents across multiple hierarchical levels (parent, child, grandchild intents). This dimensional transformation allows labelers to navigate through layers of abstraction, reducing cognitive load while preserving the ability to select from a comprehensive set of intents.

Inventive Principle:
Principle #17Another dimension (Dimensionality change)

2Adaptability or versatility

If all available intent labels are presented simultaneously to labelers, then complete intent options are available, but labelers struggle to uniquely identify the correct intent due to choice overload

Engineering Contradiction:
Improveintent label completenessVSAvoidintent identification accuracy
Core Design Contradiction:
Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent segments the complete set of intent labels into hierarchical groups, presenting only the relevant subset at each level. Labelers first see parent intents, then progressively see child intents only after selecting a parent. This segmentation ensures that the complete intent space remains available for coverage while preventing choice overload by limiting simultaneous options to manageable subsets.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent implements preliminary action by requiring labelers to first select parent intents before accessing child intent options. This preliminary selection step filters the available intent space, preparing a narrowed-down set of relevant options for the next labeling decision. This sequential approach improves identification accuracy by eliminating irrelevant options before the labeler makes the final intent selection.

Inventive Principle:
Principle #10Preliminary action

3Productivity

If intent labels are generated following the hierarchy order, then the labeling process is systematic, but the readability and natural ordering of intent labels may be compromised

Engineering Contradiction:
Improvelabeling efficiencyVSAvoidintent label readability
Core Design Contradiction:
ProductivityVSEase of operation

Solution Approach 1:

The patent implements dynamics by allowing the intent label ordering to adapt based on context. During the labeling process, intents are presented in hierarchical order for systematic selection. However, when generating the final intent label output, the system dynamically reorders the components to reflect natural language conventions and improve readability, rather than strictly maintaining the hierarchical presentation order.

Inventive Principle:
Principle #15Dynamics

Solution Approach 2:

The patent applies local quality by treating different stages of the labeling process differently. In the selection phase, hierarchical ordering provides systematic structure. In the output phase, natural language ordering provides readability. Each stage has its own optimal ordering strategy applied locally, rather than imposing a single global ordering constraint throughout the entire process.

Inventive Principle:
Principle #3Local quality

Data Source

PatentUS12393786B2Systems and methods for facilitating intent labeling for machine learning model training
Publication Date: 2025.08.19 CAPITAL ONE SERVICES LLC
  • US12393786B2 patent drawing
  • US12393786B2 patent drawing
  • US12393786B2 patent drawing

AI summary

In some embodiments, intent labeling may be facilitated for machine learning model training. In some embodiments, a set of natural language inputs may be obtained. Based on an action set, content related to intent labeling may be presented on a user interface. Based on an entity hierarchy and user selection of an agent-side action of the action set, a first entity subset of entities associated with the agent-side action may be presented on the user interface. Based on the entity hierarchy and user selection of a first entity from the first entity subset, a second entity subset of entities associated with the first entity may be presented on the user interface. Based on user selection of a second entity from the second entity subset, the natural language input may be associated with an ordered intent label that includes the agent-side action, the first entity, and the second entity.