Cross-App Screen Annotation for Mobile Context Preservation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile applications lack consistency in user experience due to varying features, link structures, and user interfaces, and cannot be crawled, resulting in lost context for personalization and assistance.
Innovation Solution
A system that captures screen images on mobile devices at intervals, analyzes content using recognition tools, and generates annotation data to provide context, allowing for personalized assistance and consistent user experience across applications by disambiguating and ranking entities, and indexing content for search and automation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If mobile applications use varying user interfaces and features, then application functionality and versatility are improved, but user experience consistency deteriorates
Solution Approach 1:
The patent applies universality by creating a cross-application annotation system that works across different mobile applications with varying interfaces. The system provides consistent annotation capabilities (entity detection, highlighting, actions) regardless of the specific application being used, making the system universally applicable across diverse apps while preserving their individual functionalities
2Reliability
If mobile applications are closed and cannot be crawled, then application security and integrity are improved, but context availability for personalization deteriorates
Solution Approach 1:
The patent uses an intermediary approach by introducing an annotation layer that sits between the user and the application content. This annotation system can access and analyze application content without requiring the application itself to be crawlable or open, thus maintaining application security while providing context for personalization through the intermediary annotation mechanism
3Adaptability or versatility
If screen capture images are analyzed and annotated, then user experience personalization is improved, but device processing complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the processing into distinct modules: screen capture, image analysis, entity detection, annotation generation, and annotation display. This modular segmentation allows the system to handle complex personalization tasks through multiple specialized components rather than a single complex processor, managing processing complexity through functional decomposition
Data Source
AI summary
Systems and methods are provided for sharing a screen from a mobile device. For example, a method includes receiving, at a second mobile device, an image of a screen captured from a first mobile device and determining whether to trigger an automated action. The method may also include displaying, responsive to not triggering the automated action, annotation data generated for the image with the image on a display of the second mobile device, the annotation data including at least one visual cue corresponding to content in the image relevant to a user of the second mobile device. The method may further include, responsive to triggering the automated action, determining that a mobile application associated with the image is installed on the second mobile device and replaying user input actions received with the image on the second mobile device starting from a reference screen associated with the mobile application.