Remote Mobile App Control via Screen Capture and Input Forwarding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing communication technologies lack the ability to effectively address customer issues with mobile applications in real-time due to limitations in shared visual context and input functionality.
Innovation Solution
A computer-implemented method and system that enables remote control of mobile applications during communication sessions by establishing a connection between a network device and a terminal device, allowing for real-time data streaming, input event detection, and control data transmission to modify the application's state.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If messaging functionality is used for customer support, then communication can be established, but shared visual context and input functionality are insufficient to adequately address mobile application issues
Solution Approach 1:
The patent creates a visual copy of the mobile application interface by capturing screen content and transmitting it to the terminal device. This allows the agent to see exactly what the customer sees, providing shared visual context without requiring the customer to physically show their device. The screen capture functionality copies the visual information from the network device to the terminal device, resolving the information loss problem.
Solution Approach 2:
The patent introduces a communication server as an intermediary that facilitates the connection between the network device (customer) and terminal device (agent). This intermediary enables the transmission of screen content, input events, and control data between devices, allowing remote control capabilities while maintaining the ease of operation through automated mediation rather than direct device-to-device communication.
2Productivity
If remote control of mobile applications is implemented, then issue resolution capability is improved, but system complexity increases
Solution Approach 1:
The communication server provides multiple functions within a single system: establishing communication sessions, transmitting screen content, forwarding input events, and delivering control data. This multi-functionality reduces overall system complexity by consolidating what could be separate specialized components into a single versatile platform, while still enabling comprehensive remote control capabilities for improved issue resolution.
Solution Approach 2:
The system enables self-service remote control by automatically capturing input events from the terminal device and translating them into appropriate control actions on the network device. The automated event detection and control execution mechanisms reduce the need for manual intervention and complex coordination, allowing the system to manage itself while providing enhanced issue resolution capabilities.
3Loss of information
If real-time data streaming is used for screen content transmission, then visual feedback is improved, but network bandwidth consumption increases
Solution Approach 1:
Instead of continuous uninterrupted streaming, the system uses periodic screen captures at defined intervals or trigger events. This approach maintains visual feedback quality by regularly updating the displayed content while reducing network bandwidth consumption by transmitting data in discrete packets rather than as a continuous stream, optimizing the trade-off between quality and resource usage.
Data Source
AI summary
Systems and methods for real-time, remote-control of mobile applications are provided. A communication session between a network device and a terminal device can be established. The network device can be configured to execute a mobile application. For example, the mobile application can include an input element at an initial state. The mobile application can be remotely controlled by the terminal device. Further, a data stream including content data can be received and transmitted during the communication session. For example, the content data can include an object presented by the mobile application. The content data can be displayed on the terminal device. An input event associated with the content data can be detected. Control data can be generated based on the detected input event. The control data can be received at the network device and the initial state of the input element can be modified.