Usage Recall Interface for Cross-App Task Continuation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mobile applications lack personalized and efficient content and action recommendations, often requiring manual user intervention and tedious steps for augmented reality interactions, and virtual agents fail to utilize prior user information for seamless task continuation across applications.
Innovation Solution
A system utilizing a neural-network-based behavior analyzer that records user actions across multiple applications, learns personalized models, and executes follow-on actions through a virtual agent, enabling seamless task continuation and augmented reality interactions without manual input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If action recommendations are hardcoded in applications, then users receive consistent action suggestions, but the recommendations are not customized to individual users leading to inferior user experience
Solution Approach 1:
The system performs preliminary actions by automatically tracking and recording user actions across applications in the background before recommendations are needed. This allows the personalized model to be pre-built and updated continuously without requiring manual user input or complex real-time processing when recommendations are generated.
Solution Approach 2:
The system implements self-service by automatically collecting user action data, analyzing behavior patterns, and generating personalized recommendations without requiring manual user configuration or intervention. The behavior analyzer continuously learns from user actions and adapts recommendations autonomously.
2Productivity
If users manually complete steps for augmented reality interactions, then precise control is achieved, but the process becomes tedious and time consuming
Solution Approach 1:
The virtual agent implements self-service by automatically performing augmented reality tasks based on user intent. Instead of requiring users to manually select items, position objects, and configure settings, the system autonomously executes these actions using the personalized behavior model to understand and complete user goals.
Solution Approach 2:
The virtual agent acts as an intermediary between the user and the augmented reality system. It translates high-level user intent into specific actions and coordinates multiple applications and features to complete complex tasks, reducing the number of manual steps users must perform.
3Loss of information
If virtual agents only respond to speech triggers, then simple voice recognition is sufficient, but they cannot utilize previous user information to generate content recommendations for seamless task continuation
Solution Approach 1:
The personalized model implements universality by serving multiple functions: it analyzes user behavior across different applications, generates content recommendations, enables task continuation, and supports both speech and non-speech interactions. This single integrated system replaces multiple separate functions that would otherwise be needed.
Solution Approach 2:
The system performs preliminary actions by continuously tracking and storing user behavior data across applications before virtual agent interactions are needed. This pre-collected information enables the agent to provide contextualized recommendations and continue tasks seamlessly without requiring users to restate their goals or preferences.
Data Source
AI summary
A computer-implemented method for usage recall on a user device is disclosed. The method includes capturing, at intervals, visual content presented on a display of the user device; analyzing the captured visual content by dividing the content into pixels and processing the pixels with a neural network trained to recognize visual elements and their positions to produce an element map; storing, in a local context store, entries associating the captured content with a time, an application or window identifier, and descriptors of the recognized elements; receiving a natural-language user query describing a past activity; retrieving, from the local context store, an entry responsive to the user query; and re-establishing at least part of a prior application state by generating user-interface events directed to a target visual element from the element map. The neural network may comprise deep or recurrent layers with LSTM and attention mechanisms optimized by reinforcement learning.