DMS-based method and apparatus for remote precise control of device screen, and medium

Abstract

The present invention relates to the technical field of remote control. Disclosed are a DMS-based method and apparatus for remote precise control of a device screen, and a medium. The method of the present invention comprises: firstly, reporting the resolution of a mainboard to a server end; then respectively calculating a resolution scaling ratio rateA between the resolution of a video stream transmitted by WebRTC and the resolution of the device mainboard, and a coordinate scaling ratio rateB between the mainboard and a display screen for projection; by means of the calculated resolution scaling ratio rateA and coordinate scaling ratio rateB, calculating a final coordinate point; and finally, issuing a simulated touch event on the basis of the final coordinate point, and injecting the simulated event into an Android system to simulate a touch operation of a user, thereby achieving the function of remotely and precisely controlling a device to control a device screen. In actual remote operation, a sliding trajectory of a mouse can cover the range of an entire screen, which means that coordinates of the entire screen can be accurately controlled.

Description

A method, apparatus, and medium for remote precision control of equipment screens based on DMS. Technical Field

[0001] This invention relates to the field of remote control technology, and more particularly to a method, apparatus and medium for remote and precise control of a device screen based on DMS. Background Technology

[0002] With the continuous advancement of technology, computers play an extremely important role in people's lives and work. Even with the enrichment of mobile phone functions, there are still some areas where mobile phones cannot perform the tasks that computers can. Because computers are bulky and inconvenient to carry, and laptops have certain requirements that cannot be met by desktop computers for certain workers, remote control functionality was invented, allowing users to remotely control their computers to complete various tasks via mobile phone.

[0003] Because the screen resolutions of mobile devices and remotely controlled computer devices are different, coordinate offsets may occur when remotely operating the device in the background, making it impossible to operate the device accurately. As shown in Figure 1, it is clear from Figure 1 that there is a large deviation between the coordinates of the background operation and the actual coordinates on the screen, which can easily lead to errors in remote control.

[0004] In view of the above problems, how to design a solution to the control deviation caused by the difference in resolution between the mobile terminal and the remote control terminal is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned issues, this invention incorporates a cloud-based remote device control function within the AirgoDMS APP (DMS for short). The principle is as follows: DMS transmits the device screen's image as a video stream to the cloud via WebRTC technology. The cloud calculates a scaling factor (rateA) by comparing the resolution of the video stream transmitted by DMS with the device's motherboard resolution. The cloud then calculates a new coordinate point (point1) by comparing the mouse's coordinates (point0) with rateA. The background system sends point1 to DMS. DMS calculates a scaling factor (rateB) by comparing the motherboard and screen resolutions. Finally, it calculates the coordinates (point1) with rateB, converting them into coordinates that the Android system can accurately recognize, thereby enabling precise device control.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The present invention provides a method for remote and precise control of a device screen based on DMS, comprising the following steps:

[0007] S1. Report the motherboard's resolution to the server.

[0008] S2. Calculate the resolution scaling ratio rateA between the resolution of the video stream transmitted via WebRTC and the resolution of the device's motherboard.

[0009] S3. Calculate the coordinate scaling ratio rateB between the motherboard and the projection screen;

[0010] S4. Calculate the final coordinate points using the resolution scaling rate A and the coordinate scaling rate B;

[0011] S5. Send simulated touch events to control the device screen based on the final coordinates.

[0012] Preferably, in step S1, a LoginEntity object is created and initialized, which is used to store the motherboard's resolution information; then, the LoginEntity object is sent as a parameter to the server via a Socket connection.

[0013] Preferably, in step S2, the resolution scaling ratio rateA between the transmitted video stream resolution and the device motherboard resolution is calculated using the width videoWidth, the height videoHeight, the width boardWidth, and the height boardHeight of the device motherboard resolution.

[0014] Preferably, in step S3, the coordinate scaling ratio rateB is calculated by comparing the motherboard resolution reported to the server with the actual resolution of the device. The coordinate values ​​are adjusted by the coordinate scaling ratio rateB to ensure that elements can be correctly displayed or positioned on devices with different resolutions.

[0015] Preferably, in step S4, the final coordinate point X2 is calculated through the following steps:

[0016] Let the original mouse operation coordinates be X1(x1,y1), then the final coordinates X2(x2,y2) are calculated as follows: x2=x1·rateA·rateB y2=y1·rateA·rateB

[0017] The above formula is used to calculate the coordinates X2(x2,y2) that the Android system accurately recognizes. X2(x2,y2) is then passed into the simulated touch event to achieve precise control of the device screen.

[0018] Preferably, in step S5, the simulated touch events sent from the background are converted into Android touch events and injected into the Android system via injectTouch. The coordinate positions in the simulated touch events are calculated by step S4. Finally, the simulated events are injected into the Android system using InputManager to simulate the user's touch operation, thereby realizing the function of remote and precise control of the device.

[0019] Preferably, in step S5, the control accuracy evaluation coefficient is calculated by collecting the coordinate set of multiple final coordinate points X2 within a preset time and the coordinate set of mouse operation coordinate points X1.

[0020] More preferably, the control accuracy evaluation coefficient is calculated using the following formula:

[0021] Where ε represents the control accuracy evaluation coefficient, n represents the number of coordinate points collected within the preset time, and Q k M represents the resolution scaling evaluation value. k M0 represents the actual value of the coordinate scaling ratio, and T represents the evaluated value of the coordinate scaling ratio. k X represents the response time between two adjacent final coordinate points during remote control. n1 This represents the set of coordinates of the mouse operation points X1 within a preset time, and the set of coordinates of the final coordinate point X2 of the set of mouse operation points X1.

[0022] The calculated control accuracy evaluation coefficient ε is compared with the set control accuracy evaluation coefficient ε0. If ε0>ε, it means that the control accuracy deviation is within the allowable range and no calibration is required. If ε0<ε, it means that there is a large deviation in control accuracy and calibration is required.

[0023] In a second aspect, the present invention also provides a remote precision control device screen based on DMS, comprising:

[0024] The data upload module is used to report the motherboard's resolution to the server.

[0025] The resolution scaling ratio calculation module is used to calculate the resolution scaling ratio between the resolution of the video stream transmitted by WebRTC and the resolution of the device's motherboard.

[0026] The coordinate scaling ratio calculation module is used to calculate the coordinate scaling ratio between the motherboard and the projection screen;

[0027] The final coordinate point calculation module is used to calculate the coordinate position of the final coordinate point;

[0028] Control module; used to control the device screen by issuing simulated touch events based on the final coordinate point.

[0029] A third aspect of the present invention also provides a readable storage medium storing computer instructions that, when executed by a processor, implement the aforementioned method for remotely and precisely controlling a device screen based on a DMS.

[0030] The beneficial effects of this invention are as follows: This invention designs a cloud-based remote control function for the device within the DMS (Distributed Management System). The DMS transmits the device screen's image to the cloud as a video stream via WebRTC technology. The cloud calculates the resolution of the video stream transmitted by the DMS and the resolution of the device's motherboard to obtain a scaling factor, rateA. The cloud then calculates the coordinates of the mouse operation, point0, with rateA to obtain a new coordinate point, point1. The background then sends coordinate point1 to the DMS. In the DMS, the resolutions of the motherboard and the screen are calculated to obtain a scaling factor, rateB. The coordinates of point1 and rateB are then calculated and converted into coordinates that the Android system can accurately recognize. Furthermore, the mouse's trajectory can cover the entire projection screen area, meaning that precise control of the entire screen's coordinates is achieved, effectively reducing errors caused by remote control. Attached Figure Description

[0031] Figure 1 is an example of the deviation between the remote operation coordinates of the DMS mouse and the screen coordinates.

[0032] Figure 2 is a schematic flowchart of the remote precision control device screen method based on DMS according to the present invention.

[0033] Figure 3 is a screenshot of the screen of the remote precision control device based on DMS according to the present invention. Detailed Implementation

[0034] Please refer to Figure 2. In one aspect, the present invention provides a method for remotely and precisely controlling a device screen based on a DMS (Distributed Control System), comprising the following steps:

[0035] S1. Report the motherboard's resolution to the server.

[0036] DMS is a system service responsible for device display management. It manages the screen's displayed content by interacting with other components in the system (such as SurfaceFlinger, WindowManager, etc.). In scenarios involving remote and precise control of a device screen, the first step is to establish a connection between the remote device and the target device. This can be achieved through network protocols (such as TCP / IP, WebSocket, etc.).

[0037] Remote connections are authenticated, and only authorized users can access and control the target device's screen. Data can only be read after the user is authorized.

[0038] First, the resolution information stored on the phone's motherboard needs to be reported to the server. This step involves two parts of code:

[0039] Part 1: A `LoginEntity` object is created and initialized using `getLoginEntity`. This object stores motherboard resolution information and other data; it is returned as the return value. The code is as follows:

[0040] Part Two: Using loginV2, the loginEntity entity object obtained in Part One of the code above is sent to the server via a Socket connection as a parameter. The loginEntity object stores the motherboard resolution information. The code is as follows:

[0041] The second part of the code sends the LoginEntity object as a parameter to the server via a Socket connection.

[0042] S2. Calculate the resolution scaling ratio rateA between the resolution of the video stream transmitted via WebRTC and the resolution of the device's motherboard.

[0043] In the DMS system, the real-time image of the target device screen is acquired using the interface provided by DMS. In this embodiment of the invention, the screen content is captured using SurfaceFlinger and encoded into a format suitable for network transmission (such as JPEG, H.264, or WebRTC video stream), and the screen resolution is obtained.

[0044] In one feasible implementation, the capture resolution and frame rate can be adjusted as needed to balance performance and transmission bandwidth.

[0045] Since the resolution of WebRTC video streams is affected by factors such as network environment and server configuration, the resolution of the video stream is not fixed. Therefore, it is necessary to calculate the resolution with respect to the motherboard to ensure that the coordinates of subsequent operations have a corresponding mapping relationship with the motherboard.

[0046] This invention utilizes the width (videoWidth), height (videoHeight) of the WebRTC video stream, the width (boardWidth), and the height (boardHeight) of the device's motherboard resolution. The computeWebRTCToScreenScale function determines a suitable scaling ratio by calculating the ratio of these two dimensions. This ratio can be used to adjust the coordinates of mouse operations to fit the screen without distortion or exceeding the screen's boundaries.

[0047] The specific calculation code is as follows:

[0048] The resolution scaling ratio rateA between the transmitted video stream resolution and the device motherboard resolution is calculated.

[0049] S3. Calculate the coordinate scaling ratio rateB between the motherboard and the projection screen;

[0050] This invention calculates the coordinate scaling ratio rateB between the display screen and the target screen using getScreenCoordinateRatio, enabling accurate mapping or transformation of coordinates on devices with different resolutions. This method determines the coordinate scaling ratio by comparing the motherboard resolution obtained from middleware with the device screen resolution.

[0051] The code is as follows:

[0052] This method is primarily used to handle coordinate mapping issues across different device resolutions. It calculates a coordinate scaling ratio by comparing the reported resolution with the device's actual resolution. This scaling ratio can be used to adjust coordinate values, ensuring that elements can be correctly displayed or positioned on devices with different resolutions.

[0053] For example, if the reported motherboard resolution is 2K, but the actual screen resolution is 4K, then all coordinate values ​​need to be magnified by two times.

[0054] S4. Calculate the final coordinate points using the resolution scaling rate A and the coordinate scaling rate B;

[0055] The following steps are used to calculate the final coordinates of point X2:

[0056] Let the original mouse operation coordinates be X1(x1,y1), then the final coordinates X2(x2,y2) are calculated as follows: x2=x1·rateA·rateB y2=y1·rateA·rateB

[0057] The above formula is used to calculate the coordinates X2(x2,y2) that the Android system accurately recognizes. X2(x2,y2) is then passed into the simulated touch event to achieve precise control of the device screen.

[0058] S5. Send simulated touch events to control the device screen based on the final coordinates.

[0059] This invention injects simulated events as touch events into the Android system of a mobile phone using `injectTouch`. It receives a `WebRtcInjectEventEntity` object, which contains detailed information about the simulated touch event, such as the action type, X and Y coordinates, etc. The final X and Y coordinates are calculated through the steps above. Internally, the method creates an Android touch event `MotionEvent` object based on this information and injects it into the system using `InputManager`. The code is as follows:

[0060] The captured screen image is transmitted over a network to a remote device and displayed in real time on the remote device.

[0061] Meanwhile, the remote device can receive user input commands (such as mouse clicks, keyboard input, etc.) and send these commands back to the target device via the network.

[0062] After the target device receives the remote command, DMS parses it and passes it to the corresponding system component (such as WindowManager or SurfaceFlinger).

[0063] System components perform corresponding operations based on instructions, such as moving the cursor, clicking a button, or opening an application.

[0064] After the target device performs the operation, the screen content is captured again and transmitted to the remote device to achieve real-time updates of the screen content.

[0065] This real-time feedback mechanism ensures that users can precisely control the screen of the target device.

[0066] In S5, the control accuracy evaluation coefficient is calculated by collecting the coordinate set of multiple final coordinate points X2 within a preset time and the coordinate set of mouse operation coordinate points X1.

[0067] More preferably, the control accuracy evaluation coefficient is calculated using the following formula:

[0068] Where ε represents the control accuracy evaluation coefficient, n represents the number of coordinate points collected within the preset time, and Q k M represents the resolution scaling evaluation value. k M0 represents the actual value of the coordinate scaling ratio, and T represents the evaluated value of the coordinate scaling ratio. k X represents the response time between two adjacent final coordinate points during remote control. n1 This represents the set of coordinates of the mouse operation points X1 within a preset time, and the set of coordinates of the final coordinate point X2 of the set of mouse operation points X1.

[0069] The calculated control accuracy evaluation coefficient ε is compared with the set control accuracy evaluation coefficient ε0. If ε0>ε, it means that the control accuracy deviation is within the allowable range and no calibration is required. If ε0<ε, it means that there is a large deviation in control accuracy and calibration is required.

[0070] The control accuracy evaluation coefficient ε is an important data point for evaluating the relative positional changes between the original coordinates and the final coordinates of the remote equipment. The smaller the control accuracy evaluation coefficient ε of the remote equipment, the higher its accuracy and the smaller the deployment error.

[0071] It's also important to note that efficiency and accuracy are often at odds in image capture. High-precision capture may consume more computing resources and bandwidth, while high-efficiency capture may compromise image quality.

[0072] To balance these two aspects, it is necessary to consider using appropriate image compression techniques, optimizing capture algorithms, and selecting suitable hardware acceleration techniques during the design phase. For example, using device-based GPU image processing can significantly improve capture efficiency, while a proper image compression algorithm can reduce data volume without significantly affecting visual quality.

[0073] Remote control requires real-time transmission of user input commands (such as mouse clicks and keyboard input) to the target device, and real-time feedback of the target device's screen image to the user. Network latency directly affects the response speed of the operation; since real-time transmission of screen images requires a high-bandwidth and low-latency network environment, especially at high resolutions or high frame rates, the latency will be more noticeable; in addition, unstable factors such as network packet loss and jitter will cause data retransmission, further increasing latency. Therefore, it is also necessary to ensure a smooth and stable network environment when using the method of this invention.

[0074] The main function of this invention is to convert simulated touch events sent from the background into Android touch events and inject them into the Android system. The coordinate positions in the simulated touch events are calculated according to the formula provided above to adapt to the screen resolutions of different devices.

[0075] Finally, the method of this invention injects this event into the mobile phone system using InputManager, allowing the application to simulate the user's touch operation, thereby realizing the function of remote and precise control of the device, as shown in Figure 3. Figure 3 clearly shows that the mouse's trajectory covers the entire screen, indicating precise control of the screen's coordinates, thus achieving precise control of the device screen. It can optimize the efficiency of screen capture and command transmission based on network conditions and device performance.

[0076] Technical support personnel can use this method to remotely assist users in solving problems and operate directly on the user's device; employees can access company equipment from home via remote control software to perform daily work; education and training: teachers can remotely control students' devices to conduct teaching demonstrations.

[0077] In addition, the remote precision control device based on DMS provided by this invention also requires attention to the following:

[0078] Security: During remote control, it is necessary to strictly ensure the security of data transmission to prevent information leakage.

[0079] User privacy: Ensure that remote control actions comply with relevant laws and regulations and respect user privacy.

[0080] The above methods can be used to achieve remote and precise control of device screens based on DMS, meeting various practical application needs.

[0081] In a second aspect, the present invention also provides a remote precision control device screen based on DMS, comprising:

[0082] The data upload module is used to report the motherboard's resolution to the server.

[0083] The resolution scaling ratio calculation module is used to calculate the resolution scaling ratio between the resolution of the video stream transmitted by WebRTC and the resolution of the device's motherboard.

[0084] The coordinate scaling ratio calculation module is used to calculate the coordinate scaling ratio between the motherboard and the projection screen;

[0085] The final coordinate point calculation module is used to calculate the coordinate position of the final coordinate point;

[0086] Control module; used to control the device screen by issuing simulated touch events based on the final coordinate point.

[0087] A third aspect of the present invention also provides a readable storage medium storing computer instructions that, when executed by a processor, implement the aforementioned method for remotely and precisely controlling a device screen based on a DMS.

[0088] Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media may be appropriately added to or subtracted from the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0089] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for remote and precise control of a device screen based on DMS, characterized in that, Includes the following steps: S1. Report the motherboard's resolution to the server; S2. Calculate the resolution scaling ratio rateA between the resolution of the video stream transmitted via WebRTC and the resolution of the device's motherboard. S3. Calculate the coordinate scaling ratio rateB between the motherboard and the projection screen; S4. Calculate the final coordinate points using the resolution scaling rate A and the coordinate scaling rate B; S5. Send simulated touch events to control the device screen based on the final coordinates.

2. The method for remotely and precisely controlling a device screen based on DMS according to claim 1, characterized in that, In step S1, a LoginEntity object is created and initialized. The LoginEntity object is used to store the motherboard's resolution information. Then, the LoginEntity object is sent as a parameter to the server via a Socket connection.

3. The method for remotely and precisely controlling a device screen based on DMS according to claim 1, characterized in that, In step S2, the resolution scaling ratio rateA between the transmitted video stream resolution and the device motherboard resolution is calculated using the width videoWidth, the height videoHeight, the width boardWidth, and the height boardHeight of the device motherboard resolution.

4. The method for remotely and precisely controlling a device screen based on DMS according to claim 1, characterized in that, In step S3, the coordinate scaling ratio rateB is calculated by comparing the motherboard resolution reported to the server with the actual resolution of the device. The coordinate values ​​are adjusted by the coordinate scaling ratio rateB to ensure that elements can be correctly displayed or positioned on devices with different resolutions.

5. The method for remotely and precisely controlling a device screen based on DMS according to claim 1, characterized in that, In step S4, the final coordinate point X2 is calculated through the following steps: Let the original mouse operation coordinates be X1(x1,y1), then the final coordinates X2(x2,y2) are calculated as follows: x2 = x1·rateA·rateB y2 = y1·rateA·rateB The above formula is used to calculate the coordinates X2(x2,y2) that the Android system accurately recognizes. X2(x2,y2) is then passed into the simulated touch event.

6. The method for remotely and precisely controlling a device screen based on DMS according to claim 1, characterized in that, In step S5, the simulated touch events sent from the background are converted into Android touch events and injected into the Android system via injectTouch. The coordinate positions in the simulated touch events are calculated by step S4. Finally, the simulated events are injected into the Android system using InputManager to simulate the user's touch operation.

7. The method for remotely and precisely controlling a device screen based on DMS according to claim 1, characterized in that, In step S5, the control accuracy evaluation coefficient is calculated by collecting the coordinate set of multiple final coordinate points X2 within a preset time and the coordinate set of mouse operation coordinate points X1.

8. A method for remotely and precisely controlling a device screen based on DMS according to claim 7, characterized in that, The control accuracy evaluation coefficient is calculated using the following formula: Where ε represents the control accuracy evaluation coefficient, n represents the number of coordinate points collected within the preset time, and Q k M represents the resolution scaling evaluation value. k M0 represents the actual value of the coordinate scaling ratio, and T represents the evaluated value of the coordinate scaling ratio. k X represents the response time between two adjacent final coordinate points during remote control. n1 This represents the set of coordinates of the mouse operation points X1 within a preset time, and the set of coordinates of the final coordinate point X2 of the set of mouse operation points X1. The calculated control accuracy evaluation coefficient ε is compared with the set control accuracy evaluation coefficient ε0. If ε0>ε, it means that the control accuracy deviation is within the allowable range and no calibration is required. If ε0<ε, it means that there is a large deviation in control accuracy and calibration is required.

9. A remote precision control device screen based on DMS, characterized in that, include: The data upload module is used to report the motherboard's resolution to the server. The resolution scaling ratio calculation module is used to calculate the resolution scaling ratio between the resolution of the video stream transmitted by WebRTC and the resolution of the device's motherboard. The coordinate scaling ratio calculation module is used to calculate the coordinate scaling ratio between the motherboard and the projection screen; The final coordinate point calculation module is used to calculate the coordinate position of the final coordinate point; Control module; Used to send simulated touch events to control the device screen based on the final coordinates.

10. A readable storage medium, characterized in that, The readable storage medium stores computer instructions that, when executed by a processor, implement a method for remotely and precisely controlling a device screen based on DMS as described in any one of claims 1-8.

Images