Cross-App Drag-and-Drop with Entity Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile devices face challenges in seamlessly sharing information between applications due to their closed environments, limiting user interaction and requiring cumbersome navigation or memorization to accomplish tasks.
Innovation Solution
Implementing an improved drag-and-drop operation that uses an entity-based model and search-based approaches, allowing users to select any on-screen entity for drag-and-drop, with machine-learning suggestions for drop locations, enabling efficient information transfer across mobile applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional mobile applications are used with specific functions, then each application performs its designated function reliably, but information sharing between applications becomes cumbersome and requires memorization or navigation back and forth
Solution Approach 1:
The patent implements a universal drag-and-drop mechanism that works across different mobile applications, allowing information to be transferred from any source application to any destination application. This multi-functional approach enables users to share information between applications without being constrained by application-specific protocols or navigation patterns, directly resolving the contradiction between maintaining application specificity and enabling information sharing.
Solution Approach 2:
The system introduces an intermediary drag-and-drop framework that mediates between source and destination applications. This intermediary layer handles the complexity of cross-application communication, entity recognition, and data transformation, allowing users to simply perform drag-and-drop gestures without needing to understand or manage the underlying complexity of information sharing between different applications.
2Loss of information
If users navigate back and forth between applications to accomplish tasks, then information can be accessed from different applications, but time is lost due to repeated navigation
Solution Approach 1:
The system performs preliminary actions by pre-recognizing entities within the drag area using graph-based data stores and machine learning models before the user completes the drag operation. This preliminary entity identification and validation occurs in the background, so when the user drops the information at the destination, the system can immediately execute the appropriate action without requiring the user to navigate back and forth or manually search for the information in the destination application.
Solution Approach 2:
The system implements feedback mechanisms where machine learning models track user actions and drop locations, then provide suggestions for future drag-and-drop operations. This feedback loop learns from user behavior patterns to anticipate information sharing needs, reducing the time required for navigation and information retrieval by suggesting relevant drop locations based on past interactions.
3Quantity of substance
If drag-and-drop uses traditional text-based selection, then text information can be transferred, but image-based entities without text elements cannot be selected or transferred
Solution Approach 1:
The patent replaces traditional text-based selection mechanics with a graph-based entity recognition system that uses machine learning to identify and categorize entities regardless of their format. Instead of relying on text selection, the system analyzes the drag area content through graph-based data stores and machine learning models to recognize entities such as images, contacts, and other data types, enabling the transfer of both text-based and image-based entities without text elements.
4Measurement precision
If the system provides detailed entity information from graph-based data stores, then accuracy of transferred information improves, but user input requirements increase
Solution Approach 1:
The system applies partial action by providing just enough entity information from the graph-based data store to enable accurate information transfer without requiring the user to review or confirm every detail. The machine learning model determines the appropriate level of information extraction based on the entity type and destination context, providing accurate information automatically while minimizing the user's input burden by only requiring the simple drag-and-drop gesture.
Data Source
AI summary
Implementations provide an improved drag-and-drop operation on a mobile device. For example, a method includes identifying a drag area in a user interface of a first mobile application in response to a drag command and receiving a drop location in a second mobile application that differs from the first mobile application. The method may also include determining that a drop location is a text input control and the drag area is not text-based, performing a search for a text description of the drag area, and pasting the text description into the text input control. The method may also include determining that a drop location is an image input control and that the drag area is text based, performing a search using the drag area for a responsive image, and pasting the responsive image into the image input control.