Activity Timeline Embeddings for Context-Aware Task Resumption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users struggle to quickly resume tasks and retrieve relevant resources due to inadequate search methods on computing devices, exacerbated by numerous storage locations and the challenge of remembering past activities, leading to inefficient manual searches.
Innovation Solution
A machine learning model processes user interaction data, generating embedding vectors to create a timeline of activities, allowing for context-aware searches and proactive content suggestions, enabling the recreation of previous application states and retrieval of relevant documents and websites.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If traditional search methods (keyword-based searches, folder hierarchies) are used, then users can search for files, but the search effectiveness deteriorates due to inadequate context understanding and user intent deciphering
Solution Approach 1:
The patent replaces traditional mechanical search methods (keyword matching, folder navigation) with an AI-based semantic search system. The system uses machine learning models to understand user intent, analyze context from screenshots and audio streams, and retrieve relevant information automatically, eliminating the need for manual keyword searching and folder browsing.
Solution Approach 2:
The patent introduces an intermediary AI system that acts between the user and the computing device's storage systems. This intermediary processes user queries, analyzes context from multiple sources (screenshots, audio, document metadata), and translates them into effective search operations, bridging the gap between simple search interfaces and complex information retrieval.
2Quantity of substance
If users manually search through multiple storage locations, then users can find files, but the time required increases significantly due to the vast amount of information and multiple directories
Solution Approach 1:
The patent implements a self-service search system where the computing device automatically indexes, organizes, and retrieves information without requiring user intervention in the search process. The system proactively monitors user activities, captures screenshots and audio streams, and maintains an updated index of all content across multiple storage locations, enabling instant retrieval without manual browsing.
Solution Approach 2:
The patent performs preliminary actions by continuously monitoring and indexing user activities, screenshots, and audio streams in advance. The system pre-processes and stores context information about user interactions, so when a search is needed, the information is already organized and ready for immediate retrieval, eliminating the need for manual search through vast amounts of data.
3Adaptability or versatility
If users switch between tasks, then users can complete multiple activities, but the ability to resume tasks deteriorates due to loss of track of what was working on
Solution Approach 1:
The patent implements a feedback mechanism that continuously monitors user activities, captures screenshots and audio streams, and stores context information about the current task state. When a user switches tasks or closes a window, the system provides feedback by preserving the task context and making it easily accessible for resumption, allowing users to quickly return to where they left off without having to re-orient themselves.
Data Source
AI summary
Machine learning techniques are leveraged to provide personalized assistance on a computing device. In some configurations a timeline of a user's interactions with the computing device is generated. For example, screenshots and audio streams may be saved as entries in the timeline. Context—the state of the computing device when the entry is created, such as which documents and websites are open—is also stored. Entries in the timeline are processed by a model to generate embedding vectors. The timeline may be searched by finding the embedding vector that is closest to an embedding vector derived from a search query. The user may select a query result, causing the associated context to be restored. For example, if the query is “show me all documents related to my upcoming trip to Japan”, the query result may open documents and websites that were open when booking a flight to Japan.