Screen Mirroring Interaction via Object Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mirroring technologies limit user interaction with multiple mirrored images, as they primarily transmit video and audio data, preventing direct control over mirrored content using keyboard, mouse, or touch inputs, and do not allow interaction with multiple mirrored screens simultaneously.
Innovation Solution
A method and apparatus that enable a sink device to receive information about objects and applications from source devices, allowing users to select and interact with objects across multiple mirroring images by determining the selected objects based on user input and providing services related to these objects, such as executing applications or uploading content, through a communication unit and processor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If mirroring technology transmits only video and audio data, then the system complexity is low, but user interaction capability with mirrored content is limited
Solution Approach 1:
The patent segments the mirroring system into distinct functional modules: a mirroring module that handles video/audio transmission, and an interaction module that processes control data. This segmentation allows the system to maintain low complexity for basic mirroring while adding interaction capabilities independently through the separate interaction module, which captures keyboard, mouse, and touch inputs and transmits them to the mirrored device.
Solution Approach 2:
The interaction module is designed with multi-functionality to handle multiple types of user inputs (keyboard, mouse, touch) and multiple types of control data (click, drag, drop, text input). This universal design allows a single module to provide comprehensive interaction capabilities across different input devices and interaction scenarios, improving ease of operation without requiring separate complex systems for each function.
2Ease of operation
If only single mirroring screen is displayed, then the display device is simple, but user ability to interact with multiple mirrored images is prevented
Solution Approach 1:
The patent extends the display from a single two-dimensional screen to multiple two-dimensional screens arranged in a spatial configuration. The interaction module captures user inputs and determines which mirrored image or region is being interacted with based on spatial coordinates, enabling cross-screen interaction. This dimensional extension allows users to interact with multiple mirrored images simultaneously while maintaining the simplicity of individual display devices.
3Adaptability or versatility
If control data transmission is added to mirroring, then interaction capability improves, but data transmission complexity increases
Solution Approach 1:
The patent merges the video/audio data transmission channel with the control data transmission channel into a unified communication system. The interaction module transmits control data (keyboard inputs, mouse movements, touch coordinates) through the existing communication infrastructure used for mirroring, rather than requiring completely separate transmission pathways. This combining approach enables enhanced adaptability for interaction while avoiding the complexity of completely separate data transmission systems.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A method for providing a service related to a plurality of mirroring images displayed on a screen of a sink device is provided. The method includes determining a first object in a first mirroring image that is displayed on the screen of the sink device, based on a user input, as the first object is moved to a location in a second mirroring image that is displayed on the screen of the sink device, determining a second object that corresponds to the location to which the first object has been moved, determining a service related to the first object and the second object, from among services provided via the second mirroring image, and outputting a result of the determined service.