Screen-Aware LLM Guidance for Multifunction Peripheral Navigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users of multifunction peripherals face difficulty in navigating complex screen transitions due to the lack of awareness of the current screen state by language models, leading to inefficient operation methods for displaying desired screens.
Innovation Solution
An information processing apparatus that includes a display, memory, and processor to accept natural language inputs, transmit screen information, and receive outputs from a language model to guide screen operations, utilizing a knowledge base and language model to provide operation methods based on the current screen state.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a language model is used to answer user questions about screen operations, then the user can obtain operation guidance in natural language, but the language model cannot provide accurate navigation because it is not aware of the current screen state
Solution Approach 1:
The system performs preliminary action by capturing and storing screen state information before the user asks a question. The current screen state is acquired and transmitted to the language model in advance, so that when the user asks about navigation, the language model already has the necessary context about where the user is currently located in the screen hierarchy, enabling accurate navigation guidance.
Solution Approach 2:
The system introduces an intermediary mechanism that bridges the user, the language model, and the screen state. A screen state acquisition unit captures the current display state, and this information is transmitted to the language model as context. This intermediary process ensures the language model receives accurate screen state information without requiring the model itself to directly interact with or understand the graphical interface.
2Adaptability or versatility
If multifunction peripherals include a small operation panel with many functions, then the device provides comprehensive functionality, but users need to perform many screen operations to display desired screens
Solution Approach 1:
The system uses the language model as an intermediary that translates the user's natural language intent into the specific sequence of screen operations needed. Instead of requiring users to manually navigate through multiple screens to find functions, they can simply ask the language model how to access a desired function, and the model provides step-by-step navigation instructions based on the current screen state.
Solution Approach 2:
The system implements feedback by continuously monitoring the current screen state and using this information to generate accurate navigation instructions. The screen state acquisition unit captures the current display, and this feedback is transmitted to the language model, which then provides operation guidance that is context-aware and adapted to the user's current position in the interface hierarchy.
Data Source
AI summary
An information processing apparatus includes a display configured to display a screen, an acceptance unit configured to accept a natural language from a user, a transmission unit configured to transmit information about the screen displayed when the acceptance unit accepts the natural language and a prompt using the accepted natural language, the information being transmitted being based on the accepted natural language, and a reception unit configured to receive an output from a language model, the output being based on the transmitted information about the screen and the transmitted prompt. The display is caused to display the received output from the language model, the output indicating an operation method for the information processing apparatus that is displayed on the screen displayed by the display after the acceptance unit accepts the natural language.