LLM-Based Animation Selection for AR Characters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Augmented Reality (AR) systems struggle to realistically react to the wide variety of objects and stimuli in the physical world, as they rely on pre-defined rules that are impractical to store for all possible scenarios.
Innovation Solution
A client device uses a large language model (LLM) to select animations for an AR character based on the identified objects in the environment, generating a prompt that specifies the status, personality, age, or mood of the AR character to assist the LLM in selecting appropriate actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a rules-based approach is used to dictate AR character animations, then the system is simple to implement, but it cannot react to the wide variety of objects and stimuli in the physical world
Solution Approach 1:
The patent introduces a large language model (LLM) as an intermediary component between the AR system and the character animation system. The LLM receives object detection results and character context, processes this information through natural language understanding, and generates appropriate animation sequences. This intermediary approach allows the system to handle diverse objects without requiring explicit rules for each object type, thereby improving adaptability while keeping the core system architecture relatively simple.
Solution Approach 2:
The system changes the parameter of decision-making from deterministic rules to probabilistic language generation. Instead of using fixed if-then rules, the LLM uses natural language models that can generate different animation responses based on the input context. This parameter change enables the system to handle variability in object detection results and character states, allowing for more versatile and context-aware character behavior.
2Reliability
If pre-defined rules are stored for all possible scenarios, then the AR character can react appropriately to objects, but it becomes impractical to store sufficient rules
Solution Approach 1:
The patent extracts the rule-generation capability from the system and delegates it to the LLM. Instead of storing comprehensive rules in the AR system, the system only stores the LLM interface and character animation definitions. The LLM independently generates the appropriate reaction rules on-demand based on the input context, eliminating the need to store all possible scenario rules while maintaining high reaction accuracy.
Solution Approach 2:
The LLM serves itself by autonomously generating appropriate animation responses without requiring external rule provision. The system provides the LLM with object detection results and character context, and the LLM independently processes this information to generate suitable animation sequences. This self-service approach eliminates the need for extensive pre-programmed rules while maintaining reliable character reactions.
3Adaptability or versatility
If an LLM is used to select animations, then the AR character can dynamically react to the environment, but the system complexity increases
Solution Approach 1:
The LLM serves multiple functions within the AR system: it processes object detection results, understands character context, determines appropriate animations, and generates animation sequences. By consolidating these multiple functions into a single LLM component, the system achieves high adaptability without proportionally increasing complexity. The LLM's multi-functionality allows one component to replace what would otherwise require multiple specialized modules.
Data Source
AI summary
A client device selects animations for an AR character by prompting a large language model (LLM) to select from a set of possible animations for the AR character. The client device captures an image of its environment using a camera and identifies objects that are depicted in the image. The client device generates a prompt for an LLM that instructs the LLM to select from a set of candidate actions for an AR character to perform based on the identified objects. The LLM returns a response to the client device and the client device extracts a set of selected actions from the LLM's response. The client device identifies a set of animations that correspond to the actions selected by the LLM and renders AR content that depicts the AR character performing those actions. The client device augments the captured image to include the AR content and displays the augmented image.