Mobile Action Cards for Contextual Entity Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Navigating between applications on mobile devices is cumbersome due to limited screen size and the lack of external input devices, making it error-prone and time-consuming, especially since mobile applications perform specific functions and do not conventionally interact with each other.
Innovation Solution
A system that identifies entities on a mobile device screen, issues queries for these entities, ranks them based on search results, and generates action cards with suggested actions, allowing users to perform relevant actions directly from the current context without navigating through multiple applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If mobile applications perform specific functions and do not conventionally interact, then application functionality is specialized and reliable, but navigation between applications becomes cumbersome and time-consuming
Solution Approach 1:
The patent introduces an intermediary system that captures screenshots, identifies entities within them, and generates action cards that mediate between different applications. This intermediary layer enables cross-application interactions without requiring traditional navigation, thus improving both navigation speed and ease of operation while maintaining application specialization.
Solution Approach 2:
The action card interface provides universal access to multiple applications and functions through a single unified interface. Instead of requiring users to navigate to different applications for different actions, the action cards consolidate multiple application functions into one accessible location, improving navigation efficiency while maintaining the specialized functionality of individual applications.
2Productivity
If screen size is limited in mobile devices, then device portability is improved, but navigation and interaction become error-prone and time-consuming
Solution Approach 1:
The patent transitions from traditional horizontal/vertical navigation on limited screens to a new dimension of interaction through action cards that present actions in a structured, easily selectable format. This dimensional change in interaction design allows users to access multiple applications and functions without the error-prone swiping and navigation required by conventional mobile interfaces, improving both efficiency and accuracy.
3Productivity
If external input devices are limited in mobile devices, then device simplicity is improved, but navigation and content interaction become more difficult
Solution Approach 1:
The system performs automatic entity identification and action generation without requiring external input devices. The screenshot capture, entity recognition, and action card generation occur automatically, allowing users to interact with content through simple selection rather than complex input operations, thus improving interaction speed and ease while maintaining device simplicity.
Data Source
AI summary
Implementations provide an interface for initiating actions related to content the user is viewing on a mobile device. A method includes performing recognition on content captured from a display of a mobile device and identifying a plurality of entities in the content. The method may include issuing a query for each of the plurality of entities, ranking the plurality of entities based on the search results, generating an action card for at least some of the highest ranked entities, and providing the action cards for display on the mobile device. Alternatively or additionally, the method may include determining at least one contact in a contacts data store that corresponds to an entity and generating an action card for the entity, the action card having a first action that uses first information from the contacts data store and a second action that uses second information from the contacts data store.