Device interaction method, device and system, storage medium, and program product

Abstract

Disclosed are a device interaction method, device and system, a storage medium, and a program product. The method comprises: receiving slave device picture data, the slave device picture data being used for generating a display picture of a slave device; forwarding the slave device picture data to a master device so as to be displayed on the master device; generating a virtual interaction interface on the basis of the slave device picture data, the virtual interaction interface comprising the display picture of the slave device; when a user of the slave device is detected in a display area, displaying the virtual interaction interface; in response to a modification operation of the user, modifying the display picture of the slave device in the virtual interaction interface; and separately sending the modified slave device picture data to the master device and the slave device so as to be displayed on the master device and the slave device.

Description

Device interaction methods, devices, systems, storage media and program products Technical Field

[0001] This disclosure relates to the field of display technology, and in particular to a device interaction method, interaction device, interaction system, storage medium, and program product. Background Technology

[0002] In related technologies, Android devices connect to the all-in-one machine via Bluetooth and send files to the all-in-one machine for opening and modification. After modification, the files are then transferred back from the all-in-one machine to the Android device, resulting in low interaction efficiency. Summary of the Invention

[0003] This disclosure provides a device interaction method, interaction device, interaction system, storage medium, and program product, aiming to at least partially solve the problem of low interaction efficiency.

[0004] In a first aspect of this disclosure, a device interaction method is provided, applied to a terminal device. The method may include: receiving slave device screen data sent by at least one slave device, the slave device screen data being used to generate a display screen of the slave device; forwarding the slave device screen data sent by the slave device to a master device to display the display screen of the slave device on the master device; generating a virtual interactive interface corresponding to the slave device based on the slave device screen data sent by the slave device, the virtual interactive interface including the corresponding display screen of the slave device; displaying the virtual interactive interface corresponding to the slave device when a user of the slave device is detected in a display area; modifying the display screen of the slave device in the virtual interactive interface in response to a modification operation by a user of the terminal device; and sending the modified slave device screen data to the master device and the slave device respectively to display the modified display screen on the master device and the slave device.

[0005] In some implementations, modifying the display screen of the slave device in the virtual interactive interface in response to a user's modification operation on the terminal device may include: acquiring an image of the interactive area and performing gesture recognition to determine a gesture operation; when the gesture operation belongs to a first gesture operation, mapping the gesture operation to a screen operation, the first gesture operation being used to modify the display screen of the slave device; and modifying the display screen of the slave device in the virtual interactive interface in response to the screen operation.

[0006] In some embodiments, the method may further include: storing the slave device screen data received by the terminal device in a database; modifying the display screen of the slave device in the virtual interactive interface includes: when the slave device screen data is not loaded in the cache, writing the slave device screen data in the database into the cache; and / or, when the slave device screen data is loaded in the cache, modifying the slave device screen data in the cache, and modifying the slave device screen data in the database based on the modified slave device screen data in the cache, so as to modify the display screen of the slave device in the virtual interactive interface.

[0007] In some embodiments, the method may further include: cleaning up the cache and / or the slave device screen data in the database.

[0008] In some embodiments, the method may further include: receiving slave device screen data sent by the master device, the slave device screen data sent by the master device being used to generate a modified display screen of the slave device; forwarding the slave device screen data sent by the master device to the slave device to display the modified display screen on the slave device; and updating the display screen of the slave device in the virtual interactive interface based on the slave device screen data sent by the master device.

[0009] In some implementations, generating a virtual interactive interface corresponding to the slave device based on the slave device screen data sent by the slave device may include: generating a virtual interactive interface corresponding to the slave device based on the slave device screen data sent by the slave device when the slave device screen data is received for the first time; and / or updating the display screen of the slave device in the virtual interactive interface based on the slave device screen data sent by the slave device when the slave device screen data is received for a subsequent time.

[0010] In some implementations, the slave device screen data in the terminal device may use the same format file as the slave device screen data in the master device and the slave device screen data in the slave device.

[0011] In some implementations, the virtual interactive interface may further include at least one of the following information: the device identifier of the corresponding slave device, the user tag of the user of the corresponding slave device, and the connection status with the corresponding slave device.

[0012] In some implementations, when a user of the slave device is detected within the display area, displaying the virtual interactive interface corresponding to the slave device may include: acquiring an image of the display area and performing human body recognition to determine a first user located within the display area; determining the slave device matched by the first user based on the matching relationship between the slave device and the user; and displaying the virtual interactive interface corresponding to the slave device matched by the first user.

[0013] In some embodiments, the method may further include: obtaining the location of the first user; displaying the virtual interactive interface corresponding to the slave device matched by the first user includes: displaying the virtual interactive interface corresponding to the slave device matched by the first user in the surrounding area of ​​the first user based on the location of the first user, wherein the distance between the virtual interactive interface and the corresponding first user is less than a first preset threshold.

[0014] In some implementations, the virtual interactive interface may be located above the corresponding slave device or the first user.

[0015] In some implementations, obtaining the location of the first user may include: acquiring images of the display area at set intervals and performing human body recognition to obtain the location of the first user; the relative position of the virtual interactive interface and the corresponding first user remains unchanged.

[0016] In some embodiments, the method may further include: changing the display position of the virtual interactive interface in response to a second gesture operation by a user of the terminal device, wherein the second gesture operation is used to drag the virtual interactive interface.

[0017] In some implementations, the terminal device may connect to the master device and the slave device in at least one of the following ways: Bluetooth, and local area network.

[0018] In some embodiments, the method may further include: scanning a first slave device located in the surrounding area of ​​the terminal device and establishing a connection, wherein the distance between the first slave device and the terminal device is less than a second preset threshold.

[0019] In some embodiments, the method may further include: obtaining the location of the first slave device; acquiring an image of the area surrounding the terminal device and performing human body recognition to obtain the location of a second user located within the area surrounding the terminal device; determining a first slave device that matches the second user based on the location of the first slave device and the location of the second user; and disconnecting from the first slave device when no second user matches the first slave device.

[0020] In some implementations, the method may further include: obtaining the device identifier of the first slave device and the user tag of the second user; binding the user tag of the second user with the matching device identifier of the first slave device to generate a matching relationship between the first slave device and the second user.

[0021] In some embodiments, the method may further include: re-establishing a connection with the first slave device after a failure to establish a connection with the first slave device; and alerting the user of the terminal device after a failure to re-establish a connection with the first slave device.

[0022] In some implementations, when the terminal device receives slave device screen data sent by at least one of the slave devices, forwarding the slave device screen data to the master device to display the display screen of the slave device on the master device may include: responding to a third gesture operation by the user of the terminal device to forward the slave device screen data sent by the second slave device to the master device so that the master device displays the display screen of the second slave device, wherein the third gesture operation is used to select a user of one of the slave devices as the user of the second slave device.

[0023] In a second aspect of this disclosure, a device interaction method is provided, applied to a master device. The method may include: receiving slave device screen data sent by a terminal device; the slave device screen data being sent by the slave device to the terminal device to generate a display screen of the slave device; and displaying the display screen of the slave device based on the slave device screen data.

[0024] In some implementations, displaying the display screen of the slave device may include: simultaneously displaying the display screen of the master device and the display screen of the slave device; or, switching the display screen of the master device to the display screen of the slave device.

[0025] In some embodiments, the method may further include: modifying the display screen of the slave device in response to a screen operation by a user of the master device; acquiring the modified slave device screen data and sending it to the terminal device, so that the terminal device forwards the modified slave device screen data to the slave device, and the terminal device and the slave device display the modified display screen.

[0026] In a third aspect of this disclosure, a device interaction method is provided, applied to a slave device. The method may include: displaying a screen based on the slave device's screen data; sending the slave device's screen data to a terminal device, so that the terminal device forwards the slave device's screen data to a master device, and the terminal device and the master device display the screen of the slave device.

[0027] In some embodiments, the method may further include: modifying the display screen of the slave device in response to a screen operation by a user of the slave device; acquiring the modified slave device screen data and sending it to the terminal device, so that the terminal device forwards the modified slave device screen data to the master device, and the terminal device and the master device display the modified display screen.

[0028] In a fourth aspect of this disclosure, an interactive device is provided, which may include a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method provided in the first aspect, or the steps of the method provided in the second aspect, or the steps of the method provided in the third aspect.

[0029] In a fifth aspect of this disclosure, an interactive system is provided, the system including a terminal device, a master device, and at least one slave device, the terminal device being connected to the master device and the slave device respectively, the terminal device implementing the steps of the method provided in the first aspect, the master device implementing the steps of the method provided in the second aspect, and the slave device implementing the steps of the method provided in the third aspect.

[0030] In a sixth aspect of this disclosure, a computer-readable storage medium is provided on which a computer program may be stored, wherein the computer program, when executed by a processor, implements the steps of the method provided in the first aspect, or the steps of the method provided in the second aspect, or the steps of the method provided in the third aspect.

[0031] In a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method provided in the first aspect, or the steps of the method provided in the second aspect, or the steps of the method provided in the third aspect.

[0032] According to one or more embodiments of this disclosure, the device interaction method, interaction device, interaction system, storage medium, and program product provide a slave device displaying a screen based on the slave device screen data and sending the slave device screen data to a terminal device. The terminal device forwards the slave device screen data to the master device to display the slave device's screen on the master device. On the other hand, it generates a virtual interactive interface corresponding to the slave device based on the slave device screen data. When a user of the slave device is detected in the display area, the virtual interactive interface corresponding to the slave device is displayed. The virtual interactive interface includes the corresponding slave device's screen. Moreover, after the user of the terminal device modifies the slave device's screen in the virtual interactive interface, the terminal device also sends the modified slave device screen data to both the master device and the slave device. The modified screen is then displayed synchronously on both the master device and the slave device. In this way, the terminal device, the master device, and the slave device can synchronously display the slave device's screen, effectively improving interaction efficiency. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 is a schematic diagram of an application scenario of a device interaction method according to some embodiments of the present disclosure.

[0035] Figure 2 is a schematic diagram of a real-world scenario shown in Figure 1.

[0036] Figure 3 is a schematic diagram of another actual scenario shown in Figure 1.

[0037] Figure 4 is a flowchart illustrating a device interaction method according to some embodiments of the present disclosure.

[0038] Figure 5 is a schematic diagram of the display results of a terminal device according to some embodiments of the present disclosure.

[0039] Figure 6 is a schematic diagram of a display result of a master device according to some embodiments of the present disclosure.

[0040] Figure 7 is a schematic diagram of another display result of a main device according to some embodiments of the present disclosure.

[0041] Figure 8 is a flowchart of the device interaction method in Figure 4 on the terminal device side.

[0042] Figure 9 is a schematic diagram of a virtual interactive interface according to some embodiments of the present disclosure.

[0043] Figure 10 is a flowchart of step S205 in Figure 8.

[0044] Figure 11 is a schematic flowchart of data processing according to some embodiments of the present disclosure.

[0045] Figure 12 is a flowchart illustrating a device interaction method according to some embodiments of the present disclosure.

[0046] Figure 13 is a schematic diagram of the device management process according to some embodiments of the present disclosure.

[0047] Figure 14 is a flowchart of step S204 in Figure 8.

[0048] Figure 15 is a flowchart illustrating the display process of a virtual interactive interface according to some embodiments of the present disclosure.

[0049] Figure 16 is a flowchart of the device interaction method in Figure 4 on the main device side.

[0050] Figure 17 is a schematic flowchart illustrating the operation of a main device according to some embodiments of the present disclosure.

[0051] Figure 18 is a schematic diagram of the device interaction method in Figure 4 on the device side.

[0052] Figure 19 is a schematic diagram of the overall architecture of an interactive system according to some embodiments of the present disclosure.

[0053] Figure 20 is an internal structural diagram of an interactive device according to some embodiments of the present disclosure.

[0054] Explanation of reference numerals in the attached diagram: 11: Terminal device; 12: Master device; 13: Slave device; 21: Host; 22: Participant; 31: Teacher; 32: Student. Detailed Implementation

[0055] To enable those skilled in the art to more clearly understand this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0056] Figure 1 is a schematic diagram of an application scenario of the device interaction method in one or more embodiments of this disclosure. Referring to Figure 1, the interaction device includes a terminal device 11, a master device 12, and at least one slave device 13. The terminal device 11 is connected to both the master device 12 and the slave device 13. Both the master device 12 and the slave device 13 can transmit data to the terminal device 11. The terminal device 11 can forward data transmitted by the master device 12 to the slave device 13, or forward data transmitted by the slave device 13 to the master device 12. The terminal device 11 can also transmit the same data to both the master device 12 and the slave device 13. In this way, the terminal device 11, the master device 12, and the slave device 13 can display the same screen simultaneously, or after any one device modifies the screen, the other two devices can not display the modified screen.

[0057] For example, terminal device 11 can be an AR (Augmented Reality) glasses, AR headset, or other AR devices. Main device 12 can be an all-in-one device. Slave device 13 can be an Android device such as a tablet or smartphone. Terminal device 11 connects to main device 12 and slave device 13 using at least one of the following methods: Bluetooth and local area network (LAN). For example, terminal device 11 connects to both main device 12 and slave device 13 via Bluetooth. Alternatively, terminal device 11 connects to both main device 12 and slave device 13 via LAN. Yet another example is that terminal device 11 connects to both main device 12 and slave device 13 simultaneously via both Bluetooth and LAN.

[0058] Figure 2 is a schematic diagram of a real-world scenario shown in Figure 1. Referring to Figure 2, the meeting host 21 and multiple participants 22 are participating in the same meeting. The meeting host 21 can operate the terminal device 11, and each participant 22 can operate a slave device 13. The master device 12 can be operated by either the meeting host 21 or a participant 22.

[0059] For example, a participant 22 can operate their slave device 13 to share the display screen presented by the slave device 13 with the terminal device 11. The terminal device 11 then presents the display screen of the slave device 13 to the host 21 and forwards it to the host device 12. The host device 12 then presents the display screen of the slave device 13 to the other participants 22. In this way, the display screen of the slave device 13 is simultaneously displayed on three devices: the terminal device 11, the host device 12, and the slave device 13. The host 21 and the other participants 22 can see the same display screen of the slave device 13 through different devices, allowing them to discuss the topics, tasks, solutions, and other content displayed on the slave device 13.

[0060] When participant 22 modifies the display screen of their slave device 13, terminal device 11 and master device 12 simultaneously display the modified display screen of slave device 13. When master participant 21 modifies the display screen of slave device 13 presented on terminal device 11, master device 12 and slave device 13 simultaneously display the modified display screen of slave device 13. When other participants 22 modify the display screen of slave device 13 presented on master device 12, terminal device 11 and slave device 13 simultaneously display the modified display screen of slave device 13.

[0061] Figure 3 is a schematic diagram of another actual scenario shown in Figure 1. Referring to Figure 3, a teacher 31 and multiple students 32 are engaged in interactive teaching. The teacher 31 can operate the terminal device 11, and each student 32 can operate a slave device 13. The master device 12 can be operated by either the teacher 31 or the students 32.

[0062] For example, a student 32 can operate their slave device 13 to share the display screen presented by the slave device 13 with the terminal device 11. The terminal device 11 then presents the display screen of the slave device 13 to the teacher 31 and forwards it to the master device 12. The master device 12 then presents the display screen of the slave device 13 to the other students 32. In this way, the display screen of the slave device 13 is displayed simultaneously on three devices: the terminal device 11, the master device 12, and the slave device 13. The teacher 31 and the students 32 can see the same display screen of the slave device 13 through different devices, so that the teacher 31 and the students 32 can discuss the difficult problems, solutions, and other content displayed on the slave device 13.

[0063] When student 32 modifies the display screen of their slave device 13, terminal device 11 and master device 12 simultaneously display the modified display screen of slave device 13. When teacher 31 modifies the display screen of slave device 13 presented on terminal device 11, master device 12 and slave device 13 simultaneously display the modified display screen of slave device 13. When other students 32 modify the display screen of slave device 13 presented on master device 12, terminal device 11 and slave device 13 simultaneously display the modified display screen of slave device 13.

[0064] Figure 4 is a flowchart illustrating the device interaction method in one or more embodiments of this disclosure. Referring to Figure 4, a first aspect of this disclosure provides a device interaction method, which includes the following steps S101 to S107.

[0065] Step S101: The slave device displays the screen based on the slave device screen data and sends the slave device screen data to the terminal device.

[0066] In one possible embodiment, a user of the slave device can share the display screen of their slave device when needed. In response to the user's sharing action, the slave device sends its screen data to the terminal device. In another possible embodiment, the slave device receives a sharing request from the terminal device and, based on the request, sends its screen data to the terminal device. The terminal device can obtain the screen data of a specified slave device by sending a sharing request to that specific slave device.

[0067] The above two methods apply to the initial transmission of device screen data from the slave device to the terminal device. After the slave device initially transmits the screen data to the terminal device, the user of the slave device may modify the display screen of their slave device. In another possible embodiment, in response to the user's modification operation, the slave device not only modifies the display screen but also sends the modified slave device screen data to the terminal device.

[0068] For example, the operation mode of the device can be screen operation, that is, the user's sharing and modification operations on the device can be screen operations.

[0069] For example, the number of devices can be one or multiple, and is not limited to one-to-one interaction, which can also improve interaction efficiency.

[0070] In step S102, the terminal device forwards the slave device screen data sent by the slave device to the master device, and generates a virtual interactive interface corresponding to the slave device based on the slave device screen data sent by the slave device.

[0071] For example, the virtual interactive interface includes the display screen of the corresponding slave device.

[0072] When the slave device first sends its screen data to the terminal device, the terminal device receives the slave device screen data for the first time, but does not yet have a corresponding virtual interface for the slave device. In one possible embodiment, the terminal device can generate a corresponding virtual interface for the slave device based on the slave device screen data sent by the slave device.

[0073] When the slave device subsequently sends its screen data to the terminal device again due to modifications or other reasons, the terminal device will not be receiving the slave device's screen data for the first time, and the terminal device will already have a virtual interactive interface corresponding to the slave device. In another possible embodiment, the terminal device can update the slave device's display screen in the virtual interactive interface based on the slave device's screen data. This only modifies the slave device's display screen in the virtual interactive interface, while other elements such as gesture interactions in the virtual interactive interface remain unchanged. This is faster than regenerating the virtual interactive interface, which is beneficial for real-time display on the terminal device and improves the user experience.

[0074] For example, when multiple slave devices send slave device screen data to a terminal device, the terminal device can forward the slave device screen data sent by each slave device to the master device, and generate a virtual interactive interface corresponding to each slave device based on the slave device screen data. That is, the terminal device forwards the slave device screen data sent by each slave device to the master device respectively, and generates a virtual interactive interface corresponding to each slave device based on the slave device screen data sent by each slave device.

[0075] Step S103: When the terminal device detects a user of the slave device in the display area, it displays the virtual interactive interface corresponding to the slave device.

[0076] For example, the display area of ​​a terminal device is the area where the terminal device is currently displaying the image. This can be an area in the real environment overlaid with a virtual image, typically located in the perimeter of the terminal device. Taking AR glasses as an example, the display area of ​​the terminal device is the area currently seen by the user through the AR glasses.

[0077] After receiving screen data from the slave device, the terminal device can first generate a virtual interactive interface corresponding to the slave device and store it in the database. Once the terminal device detects the user of the slave device within its display area, it retrieves the corresponding virtual interactive interface from the database and displays it. At this point, the user of the slave device appears in the user's field of vision. The terminal device overlays the virtual interactive interface of the slave device onto the user's real environment, allowing the user to simultaneously see the slave device, the user, and the corresponding virtual interactive interface.

[0078] The terminal device presents a virtual image within the user's real environment, achieving a fusion of virtual and real worlds. For slave devices detected within the display area, a corresponding virtual interactive interface is displayed to facilitate interaction. Figure 5 is a schematic diagram of the display result of the terminal device in this disclosure. Referring to Figure 5, corresponding virtual interactive interfaces are superimposed on the slave devices and the user in the real environment.

[0079] Step S104: The master device displays the display screen of the slave device based on the screen data of the slave device.

[0080] The main device can display a screen based on the main device's own screen.

[0081] In one possible embodiment, the master device can simultaneously display both the master device's display screen and the slave device's display screen. For example, the size of the master device's display screen can be reduced to make room for the slave device's display screen. Figure 6 is a schematic diagram of one display result of the master device in this disclosure. Referring to Figure 6, the master device's display screen and the display screens of each slave device are arranged on the master device's display screen.

[0082] In another possible embodiment, the master device can switch between the display screen of the master device and the display screen of the slave device. Figure 7 is a schematic diagram of another display result of the master device in this disclosure. Referring to Figure 7, the display screen of a slave device basically occupies the entire display screen of the master device. The display screen of the master device or other slave devices can be switched by clicking the device icon at the edge of the display screen, or by swiping the display screen.

[0083] Step S105: In response to the user's modification operation, the terminal device modifies the display screen of the slave device in the virtual interactive interface and sends the modified slave device screen data to the master device and the slave device respectively.

[0084] Step S106: The master device displays the modified display screen based on the slave device screen data sent by the terminal device.

[0085] Step S107: The slave device displays the modified display screen based on the slave device screen data sent by the terminal device.

[0086] The aforementioned device interaction method involves the slave device displaying its own screen data and sending this data to the terminal device. Upon receiving the slave device's screen data, the terminal device forwards it to the master device and simultaneously generates a virtual interactive interface corresponding to the slave device. This virtual interface, which includes the corresponding slave device's screen, is displayed when a user is detected within the display area. The master device, upon receiving the slave device's screen data from the terminal device, displays the slave device's screen data. Furthermore, if the user modifies the slave device's screen in the virtual interactive interface, the terminal device sends the modified screen data to both the master and slave devices, simultaneously displaying the modified screen on both. This synchronized display of the slave device's screen by the terminal, master, and slave devices effectively improves interaction efficiency and enhances user experience. Moreover, the entire interaction process requires no manual operation from the user, allowing for seamless interaction and saving time and effort.

[0087] For example, the terminal device can be operated via gestures, meaning that user actions such as modification can be performed using gestures. Interacting with both the master and slave devices through gestures allows users to share the slave device's display to the master device for presentation, and also to modify the slave device's display.

[0088] Virtual interactive interfaces can intuitively display the screen from the device, and users can modify the display screen through gesture operations, which is convenient and intuitive.

[0089] In some embodiments, the user of the master device can also modify the display screen of the slave device. The method may further include the following steps: the master device, in response to the user's modification operation, modifies the display screen of the slave device and sends the modified slave device screen data to the terminal device; the terminal device forwards the slave device screen data sent by the master device to the slave device and updates the display screen of the slave device in the virtual interactive interface based on the slave device screen data sent by the master device; the slave device displays the modified display screen based on the slave device screen data sent by the terminal device.

[0090] For example, the main device can be operated via screen operation, meaning that the user's modification and switching operations on the main device can be performed via screen operation.

[0091] In addition to the slave device, both the terminal device and the master device can modify the display screen of the slave device and display it synchronously on the other two devices, simplifying the interaction process between devices. The terminal device automatically forwards the screen data received from the slave device, including forwarding the slave device screen data sent by the master device to the slave device, and forwarding the slave device screen data sent by the slave device to the master device. This allows the terminal device, master device, and slave device to update the display screen of the slave device in real time, achieving synchronous display of the terminal device, master device, and slave device.

[0092] In some embodiments, the slave device screen data in the terminal device, the slave device screen data in the master device, and the slave device screen data in the slave device can use the same format file to facilitate the transmission of slave device screen data between devices.

[0093] For example, the device screen data can be a dbat format file. On one hand, it is compatible with the AR platform of the terminal device, as well as the Android platforms of the master and slave devices, facilitating the transfer of slave device screen data between the terminal device and the master / slave device. On the other hand, dbat format files have a small file size, enabling fast and smooth real-time transmission and updates, thus improving the user experience.

[0094] The above is a brief introduction to the interaction process between terminal devices, master devices, and slave devices. The following sections will provide a detailed introduction to terminal devices, master devices, and slave devices respectively.

[0095] Figure 8 is a flowchart of the device interaction method in Figure 4 on the terminal device side. Referring to Figure 8, the second aspect of this disclosure provides a device interaction method, the execution subject of which is a terminal device, including the following steps S201 to S206.

[0096] Step S201: Receive slave device screen data sent by at least one slave device. The slave device screen data is used to generate the display screen of the slave device.

[0097] Step S202: The slave device screen data sent by the slave device is forwarded to the master device so that the slave device's display screen is displayed on the master device.

[0098] Step S203: Based on the slave device screen data sent by the slave device, generate a virtual interactive interface corresponding to the slave device. The virtual interactive interface includes the display screen of the corresponding slave device.

[0099] For example, the virtual interactive interface may also include at least one of the following information: the device identifier of the corresponding slave device, the user tag of the corresponding slave device user, and the connection status with the corresponding slave device, to facilitate the management of the slave device. The device identifier may include, but is not limited to, at least one of UDID (Unique Device Identifier), device username, and device code. The user tag may include, but is not limited to, at least one of user avatar, user physical characteristics, user location, and custom name. Figure 9 is a schematic diagram of the virtual interactive interface in this disclosure. Referring to Figure 9, the virtual interactive interface can display the slave device's screen, the slave device's device identifier, the slave device user's user tag, and the connection status with the slave device.

[0100] Step S204: When a user of a slave device is detected in the display area, the virtual interactive interface corresponding to the slave device is displayed.

[0101] Step S205: In response to the user's modification operation on the terminal device, modify the display screen of the device in the virtual interactive interface.

[0102] Step S206: The modified slave device screen data is sent to the master device and slave device respectively, so that the modified display screen is displayed on the master device and slave device.

[0103] In the aforementioned device interaction method, after the terminal device receives screen data from at least one slave device, the slave device screen data is used to generate the slave device's display screen. On one hand, the slave device screen data sent by the slave device is forwarded to the master device for display on the master device. On the other hand, based on the slave device screen data sent by the slave device, a virtual interactive interface corresponding to the slave device is generated. When a user of the slave device is detected in the display area, the virtual interactive interface corresponding to the slave device is displayed. The virtual interactive interface includes the corresponding slave device's display screen. Furthermore, if the user of the terminal device modifies the slave device's display screen in the virtual interactive interface, the terminal device will also send the modified slave device screen data to both the master device and the slave device. The modified display screen is then displayed synchronously on both the master device and the slave device. By connecting the master device and the slave device through the terminal device, and simultaneously displaying the slave device's display screen on the terminal device, the master device, and the slave device, the interaction efficiency can be effectively improved.

[0104] In some embodiments, step S203 may include the following steps: when receiving slave device screen data sent by the slave device for the first time, generating a virtual interactive interface corresponding to the slave device based on the slave device screen data sent by the slave device; when receiving slave device screen data sent by the slave device for a subsequent time, updating the display screen of the slave device in the virtual interactive interface based on the slave device screen data sent by the slave device.

[0105] In other embodiments, step S203 may also include the following steps: when receiving slave device screen data sent by the slave device for the first time, generating a virtual interactive interface corresponding to the slave device based on the slave device screen data sent by the slave device; when receiving slave device screen data sent by the slave device for a subsequent time, regenerating the virtual interactive interface corresponding to the slave device based on the slave device screen data sent by the slave device.

[0106] Figure 10 is a flowchart of step S205 in Figure 8. Referring to Figure 10, in one possible embodiment, the terminal device modifies the display screen of the slave device. The method may also include the following steps S301 to S303.

[0107] Step S301: Acquire an image of the interactive area and perform gesture recognition to determine the gesture operation.

[0108] Step S302: When the gesture operation belongs to the first gesture operation, the gesture operation is mapped to the screen operation. The first gesture operation is used to modify the display screen of the slave device.

[0109] Step S303: In response to screen operation, modify the display screen of the device in the virtual interactive interface.

[0110] For example, the first gesture operation is a swipe operation, and the mapped screen operation is to annotate on the device's display screen in the virtual interactive interface.

[0111] In some embodiments, the method may further include the following step: storing the data received by the terminal device from the device screen into a database.

[0112] For example, the database can be SQLite (a lightweight database), which is lightweight, occupies little space (only a few hundred kilobytes), has fast data read and write speeds, and does not require a separate server process. It can directly interact with applications, simplifying configuration and deployment. Moreover, SQLite is cross-platform and can run on most platforms, which is beneficial for data transfer between terminal devices and master and slave devices.

[0113] For example, storing the slave device screen data received by the terminal device into a database may include: storing the slave device screen data into the database when it is received for the first time; and replacing the slave device screen data in the database with the latest received slave device screen data when it is received again.

[0114] Accordingly, step S301 may include the following steps: when the slave device screen data is not loaded in the cache, write the slave device screen data in the database into the cache; modify the slave device screen data in the cache, and modify the slave device screen data in the database based on the modified slave device screen data in the cache, so as to modify the display screen of the slave device in the virtual interactive interface.

[0115] Step S301 may also include the following steps: when slave device screen data is loaded in the cache, modify the slave device screen data in the cache, and modify the slave device screen data in the database based on the modified slave device screen data in the cache, so as to modify the display screen of the slave device in the virtual interactive interface.

[0116] For example, step S301 may include the following steps: when modifying the display screen of the slave device in the virtual interactive interface for the first time, writing the slave device screen data in the database into the cache and modifying the slave device screen data in the cache; when modifying the display screen of the slave device in the virtual interactive interface for the second time, modifying the slave device screen data in the cache.

[0117] In the above embodiments, by first writing the slave device screen data from the database into the cache and then modifying the slave device screen data in the cache, the speed is faster than modifying the slave device screen data from the database. This is beneficial for the terminal device to display in real time and improves the user experience.

[0118] Figure 11 is a schematic diagram of the data processing flow in this disclosure. Referring to Figure 11, for example, after the application interacting on the terminal device starts, it first initializes the database and cache, and then checks whether the cache stores the device screen data. If the cache stores the device screen data, it retrieves the device screen data from the cache and returns it to the application. If the cache does not store the device screen data, it first writes the device screen data from the database into the cache, then retrieves the device screen data from the cache and returns it to the application. If the application updates the device screen data in response to user operation, it first updates the device screen data in the cache, then updates the device screen data in the database, thereby storing the updated device screen data in real time and transmitting the updated device screen data in real time.

[0119] For example, the method may further include the step of clearing slave device screen data in the cache and / or database. The clearing may be performed only on the cached slave device screen data, only on the database slave device screen data, or simultaneously on both the cached and database slave device screen data.

[0120] For example, a cleanup button can be added to the virtual interface to respond to user cleanup requests and clear cached slave device screen data. Alternatively, terminal devices can use JobScheduler or WorkManager to periodically check cached slave device screen data for useless data and clean it up. Furthermore, the cache can use the LRU (Least Recently Used) algorithm to clean up the least frequently used slave device screen data. Custom caching strategies can also be defined to clean up the least frequently used slave device screen data only when the cached data exceeds a set threshold.

[0121] In the above embodiments, by promptly clearing the slave device screen data in the cache and database, the storage space of the terminal device can be saved, ensuring the smooth operation of the application.

[0122] Figure 12 is a flowchart of a device interaction method in another embodiment of this disclosure. Referring to Figure 12, in another possible embodiment, the master device modifies the display screen of the slave device. The method may also include the following steps S401 to S403.

[0123] Step S401: Receive the slave device screen data sent by the master device. The slave device screen data sent by the master device is used to generate the modified display screen of the slave device.

[0124] Step S402: Forward the slave device screen data sent by the master device to the slave device so that the modified display screen can be displayed on the slave device.

[0125] Step S403: Update the display screen of the slave device in the virtual interactive interface based on the slave device screen data sent by the master device.

[0126] The above describes the interaction between the terminal device and the master and slave devices. A connection needs to be established before interaction can occur. The following describes the connection establishment process between the terminal device and the master and slave devices.

[0127] In some embodiments, the method may further include the following steps: scanning a first slave device located in the peripheral area of ​​the terminal device and establishing a connection, wherein the distance between the first slave device and the terminal device is less than a second preset threshold.

[0128] For example, the surrounding area of ​​the terminal device can be the physical space where the terminal device is located, such as a conference room for multi-person meetings or a classroom for interactive teaching.

[0129] For example, after startup, the terminal device can automatically scan for, identify, and classify surrounding devices, and establish connections with them based on preset rules, thereby automatically establishing connections with both master and slave devices. The preset rules can be device identifiers for the master and slave devices to exclude other devices. Alternatively, the preset rules can be device identifiers set according to user preferences, such as identifiers of devices the user wants to connect to or devices the user does not want to connect to.

[0130] In some embodiments, the method may further include the following steps: obtaining the location of a first slave device; acquiring images of the area surrounding the terminal device and performing human body recognition to obtain the location of a second user located within the area surrounding the terminal device; determining a first slave device that matches the second user based on the location of the first slave device and the location of the second user; and disconnecting from the first slave device when no second user matches the first slave device.

[0131] In the above embodiments, the locations of slave devices and users are matched. If the location of a slave device is close to the location of a user, it means that the user is a user of that slave device. The slave device and the user are then bound together, indicating the matching relationship between the slave device and the user. If a slave device has no matched user, it means that no user is using that slave device, no interaction will occur, and the connection can be disconnected.

[0132] For example, the terminal device can be configured with a camera or image sensor to capture images. The terminal device can also be configured with a gyroscope and accelerometer, etc., to determine the position of the terminal device and whether the user is wearing the terminal device, etc.

[0133] In some implementations, the method may further include the following steps: obtaining the device identifier of the first slave device and the user tag of the second user; binding the user tag of the second user with the device identifier of the matching first slave device to generate a matching relationship between the first slave device and the second user.

[0134] For example, the device identifier and location sent by the first slave device can be received to obtain the device identifier and location of the first slave device. By acquiring images of the area surrounding the terminal device and performing human body recognition, in addition to obtaining the location of the second user, a user tag for the second user can also be generated. The user tag contains user features obtained from human body recognition, such as face and appearance. When determining the first slave device to match the second user, the user tag of the second user is bound to the device identifier of the first slave device, and the resulting matching relationship between the first slave device and the second user is the matching relationship between the slave device and the user. The matching relationship between the slave device and the user is stored in a database and can be retrieved when needed.

[0135] For example, the virtual interactive interface can display the device identifier of the slave device, the user's user tag, and the connection status with the slave device. Users on each slave device are monitored in real time; if a slave device changes from having users to having no users, the connection to that slave device is disconnected. Conversely, if a slave device changes from having no users to having users, the connection is re-established. Through automatic connection establishment and disconnection, interactions with each slave device can be managed seamlessly. This enables multi-device recognition and management on AR devices, achieving rapid management and control. Connections are largely automated, requiring minimal user intervention, reducing user steps and learning costs, making it convenient and efficient for users.

[0136] In some embodiments, the method may further include the following steps: re-establishing a connection with the first slave device after a failure to establish a connection with the first slave device; and alerting the user of the terminal device after a failure to re-establish a connection with the first slave device.

[0137] For example, the device connection status can be automatically detected, and repair can be automatically performed if the initial connection fails. If the automatic repair fails, the user will be notified to handle the issue promptly. The user can be notified in at least one of the following ways: displaying a connection failure message on the virtual interface or the terminal device vibrating.

[0138] Figure 13 is a flowchart of device management in this disclosure. Please refer to Figure 13. For example, after the terminal device is started, it automatically scans the surrounding devices, identifies and establishes a connection. If the first connection fails, it re-establishes the connection. If the re-connection fails, it reminds the user. After the connection is successful, it obtains the matching user tag. If the user of the terminal device is not wearing the terminal device, it disconnects and enters a sleep state.

[0139] The following section describes some scenarios during the interaction process of terminal devices.

[0140] Figure 14 is a flowchart of step S204 in Figure 8. Referring to Figure 14, in some embodiments, step S204 may include steps S501 to S503 as follows.

[0141] Step S501: Acquire an image of the display area and perform human body recognition to determine the first user located within the display area.

[0142] For example, when using a face as a user feature, face recognition can be performed on the image of the display area, and the user identity obtained based on the face recognition result can determine the first user located within the display area. When using a human body shape as a user feature, feature extraction can be performed on the image of the display area, and the feature set obtained based on the feature extraction result can determine the first user located within the display area.

[0143] Step S502: Based on the matching relationship between the slave device and the user, determine the slave device matched with the first user.

[0144] For example, the database can store the mapping between device identifiers of slave devices and user tags of users. This mapping can bind slave devices and users one-to-one, and a slave device and user bound together is a matching pair. By retrieving this mapping from the database, the matching relationship between slave devices and users can be determined.

[0145] Step S503: Display the virtual interactive interface corresponding to the slave device matched by the first user.

[0146] In one possible embodiment, the method may further include the step of obtaining the location of a first user. This step can be performed simultaneously with step S501, that is, acquiring an image of the display area and performing human body recognition to determine the first user located within the display area and obtaining the location of the first user.

[0147] Accordingly, step S503 may include the following steps: based on the location of the first user, displaying a virtual interactive interface corresponding to the slave device matching the first user within the surrounding area of ​​the first user, wherein the distance between the virtual interactive interface and the corresponding first user is less than a first set threshold.

[0148] For example, the virtual interactive interface can be located above the corresponding slave device or the first user, providing an intuitive operating interface and interaction method, allowing management of multiple devices without switching perspectives.

[0149] For example, obtaining the location of the first user may include the following steps: capturing images of the display area at set intervals and performing human body recognition to obtain the location of the first user.

[0150] Accordingly, the relative position between the virtual interactive interface and the corresponding first user can remain unchanged.

[0151] In the above embodiments, by acquiring the location of the first user at set intervals, the display position of the virtual interactive interface can be adjusted in a timely manner when the location of the first user changes, so that the relative position of the virtual interactive interface and the corresponding first user remains unchanged.

[0152] In another possible embodiment, the method may further include the following steps: changing the display position of the virtual interactive interface in response to a second gesture operation by a user of the terminal device, wherein the second gesture operation is used to drag the virtual interactive interface. When the display position of the virtual interactive interface deviates from that of the user of the corresponding slave device, the user can manually adjust the display position of the virtual interactive interface to facilitate interaction by the user of the terminal device.

[0153] For example, in response to a second gesture operation by a user on a terminal device, changing the display position of the virtual interactive interface may include the following steps: acquiring an image of the interactive area and performing gesture recognition to determine the gesture operation; when the gesture operation is a second gesture operation, causing the display position of the virtual interactive interface to change with the position of the gesture operation, and taking the position where the gesture operation disappears as the display position of the virtual interactive interface.

[0154] Figure 15 is a flowchart illustrating the display process of the virtual interactive interface in this disclosure. Referring to Figure 15, after the application on the terminal device starts, initialization is performed first. Initialization includes, but is not limited to, initializing the code library, checking the access permissions of the camera or sensor, and device compatibility. Then, the camera and sensor are activated, and planar detection is performed based on the code library to acquire and identify images of the area surrounding the terminal device. This allows the plane and the user tag and location of the second user located within the area surrounding the terminal device to be determined. After scanning the first slave device located within the area surrounding the terminal device and establishing a connection, the user identifier and location of the first slave device are acquired, and the slave device with the matching location is bound to the user. Then, based on the user's location, the display position of the virtual interactive interface corresponding to the matching slave device is determined. The ARKit platform's faceAnchor or the ARCore platform's Face Detection can be used to track changes in the user's position so that the display position of the virtual interactive interface follows the user's position changes. When the user on the terminal device manually adjusts the position of the virtual interactive interface, the display position of the virtual interactive interface is changed. After exiting the application, the camera and sensor are turned off, and the data is cleared.

[0155] In some embodiments, when the terminal device receives screen data from at least one slave device, step S202 may include the following steps: in response to a third gesture operation by the user of the terminal device, forwarding the screen data from the second slave device to the master device so that the master device displays the screen of the second slave device, wherein the third gesture operation is used to select a user of a slave device as the user of the second slave device.

[0156] For example, in response to a user's third gesture operation on a terminal device, forwarding the slave device screen data sent by the second slave device to the master device may include the following steps: acquiring an image of the interactive area and performing gesture recognition to determine the gesture operation; when the gesture operation belongs to a third gesture operation, forwarding the slave device screen data sent by the second slave device to the master device. For example, the third gesture operation includes a combination of a double-click operation and a quick throw operation; the user first double-clicks to select the display screen of the slave device, and then quickly throws to share the selected display screen.

[0157] Figure 16 is a flowchart of the device interaction method in Figure 4 on the main device side. Referring to Figure 16, the third aspect of this disclosure provides a device interaction method, the execution subject of which is the main device, including the following steps S601 to S602.

[0158] Step S601: Receive the slave device screen data sent by the terminal device. The slave device screen data is sent from the slave device to the terminal device and is used to generate the display screen of the slave device.

[0159] Step S602: Based on the slave device screen data, display the slave device's display screen.

[0160] In one possible embodiment, step S602 may include the following steps: simultaneously displaying the display screen of the master device and the display screen of the slave device.

[0161] In another possible embodiment, step S602 may include the following step: switching the display screen of the master device to the display screen of the slave device.

[0162] In some embodiments, the method may further include the following steps: modifying the display screen of the slave device in response to a user's screen operation on the master device; acquiring the modified slave device screen data and sending it to the terminal device, so that the terminal device forwards the modified slave device screen data to the slave device, and the terminal device and the slave device display the modified display screen.

[0163] Figure 17 is a flowchart illustrating the operation of the master device in this disclosure. Referring to Figure 17, for example, after the master device starts up, it initializes the display screen and sets the distribution area of ​​each display screen. It loads and displays the display screens of the master device and the slave device. When a user modifies the display screen of the slave device on the master device, the modified display screen is displayed, and the modified slave device screen data is forwarded to the terminal device for display on both the terminal device and the slave device. When a user modifies the display screen of the slave device on either the terminal device or the slave device, the master device receives the slave device screen data sent by the terminal device and displays the modified display screen based on the received slave device screen data. Finally, the master device is shut down and its data is cleared.

[0164] Figure 18 is a schematic flowchart of the device interaction method in Figure 4 on the slave device side. Referring to Figure 18, the fourth aspect of this disclosure provides a device interaction method, the execution subject of which is a slave device, including the following steps S701 to S702.

[0165] Step S701: Display the screen based on the data from the device screen.

[0166] In step S702, the slave device screen data is sent to the terminal device, so that the terminal device forwards the slave device screen data to the master device, and the terminal device and the master device display the display screen of the slave device.

[0167] In some embodiments, the method may further include the following steps: modifying the display screen of the slave device in response to a user's screen operation on the slave device; acquiring the modified slave device screen data and sending it to the terminal device, so that the terminal device forwards the modified slave device screen data to the master device, and the terminal device and the master device display the modified display screen.

[0168] According to a fifth aspect embodiment of this disclosure, an interactive system is provided. Referring to FIG1, the system includes a terminal device 11, a master device 12, and at least one slave device 13, wherein the terminal device 11 is connected to the master device 12 and the slave device 13 respectively.

[0169] Figure 19 is a schematic diagram of the overall architecture of the interactive system in one or more embodiments of this disclosure. Referring to Figure 19, the system architecture includes a device layer, a front-end UI (User Interface) layer, a control layer, a communication layer, and a data layer. Each layer can be divided into AR and Android types. In the device layer, AR devices include terminal devices 11, and Android devices include master devices 12 and slave devices 13. In the front-end UI layer, the AR front-end UI includes AR platforms (such as ARCore, ARKit), 3D modeling and rendering, screen graphics drawing components (such as SurfaceView), and image recognition. The Android front-end UI includes floating windows (such as Dialog), interface layout files (such as layout), interactive windows (such as Activity), and basic interface frameworks (such as View). In the control layer, AR control includes SLAM (simultaneous localization and mapping), a cloud service processor for managing and monitoring multiple devices (such as Google Cloud IoT), object recognition and tracking, gesture recognition, and sensor data acquisition. Android control includes Bluetooth data monitoring, WiFi data monitoring, display screen data processing, and display screen control. At the communication layer, both AR and Android can use Bluetooth and WiFi for communication. At the data layer, both AR and Android can use databases and caches for data storage.

[0170] For example, a workbench for the main device 12 and slave device 13 can be built first in the Android development environment, and then this workbench can be migrated to AR. Using the AR platform within the Android development environment, existing functions are integrated into the AR interface based on the Android code, keeping the business logic unchanged, and the interface is adjusted according to AR interaction requirements. For instance, design tools such as Figma, Sketch, and Adobe XD can be used to design an AR interface adapted for gesture control, and collaboration tools such as Invision and Marvel App can be used to design gestures for interaction. New gesture operations can be mapped to existing screen operations to ensure compatibility with Android input and gesture control. Furthermore, gesture recognition can be implemented using machine learning libraries (such as TensorFlow and OpenCV), and performance optimization tools (such as Android Profiler) can be used to optimize gesture response time.

[0171] The code can be refactored. On one hand, modular programming can be used, employing design patterns (such as Singleton and Factory patterns) to refine and differentiate functional modules, encapsulating the code, implementing inheritance and polymorphism, and allowing for the reuse of functions like drawing on the canvas. On the other hand, new AR-specific functional modules can be developed using an AR platform. ARCore's Augmented Images, Augmented Faces, and Sceneform gesture listening can be utilized.

[0172] A sixth aspect of this disclosure provides an interactive device, which can be a terminal device, a master device, or a slave device. The interactive device includes a processor and a memory, the memory storing computer program code that, when executed by the processor, causes the processor to perform a device interaction method according to one or more embodiments of this disclosure.

[0173] Figure 20 is an internal structural diagram of an interactive device according to one or more embodiments of this disclosure. Referring to Figure 20, the interactive device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be implemented through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a device interaction method. The display unit is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the device casing, or an external keyboard, touchpad, or mouse, etc.

[0174] Those skilled in the art will understand that the structure shown in FIG20 is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include ROM (Read-Only Memory), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, ReRAM (Resistive Random Access Memory), MRAM (Magnetoresistive Random Access Memory), FRAM (Ferroelectric Random Access Memory), PCM (Phase Change Memory), graphene memory, etc. Volatile memory can include RAM (Random Access Memory) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] In this disclosure, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0178] In the description of this disclosure, it should be understood that the terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “rear,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this disclosure.

[0179] In this disclosure, unless otherwise expressly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0180] Furthermore, the use of terms such as "first" and "second" in this disclosure is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include one or more features. In the description of this disclosure, "multiple" means two or more, unless otherwise explicitly specified.

[0181] Although embodiments of the present disclosure have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A device interaction method, applied to a terminal device, comprising: Receive at least one slave device screen data sent by a slave device, the slave device screen data being used to generate the display screen of the slave device; The slave device screen data sent by the slave device is forwarded to the master device so that the display screen of the slave device can be displayed on the master device; Based on the slave device screen data sent by the slave device, a virtual interactive interface corresponding to the slave device is generated, and the virtual interactive interface includes the corresponding display screen of the slave device; When a user of the slave device is detected within the display area, the virtual interactive interface corresponding to the slave device is displayed; In response to a user's modification operation on the terminal device, the display screen of the virtual interactive interface on the device is modified; The modified slave device screen data is sent to the master device and the slave device respectively, so that the modified display screen is displayed on the master device and the slave device.

2. The method according to claim 1, wherein, The modification of the display screen of the virtual interactive interface in response to the user's modification operation on the terminal device includes: Capture images of the interactive area and perform gesture recognition to determine the gesture operation; When the gesture operation belongs to the first gesture operation, the gesture operation is mapped to a screen operation, and the first gesture operation is used to modify the display screen of the slave device; In response to the screen operation, the display screen of the device in the virtual interactive interface is modified.

3. The method according to claim 2, further comprising: The terminal device stores the screen data received from the device into a database; Modifying the display screen of the virtual interactive interface from the device includes: If the slave device screen data is not loaded in the cache, the slave device screen data in the database is written into the cache; And / or, When the slave device screen data is loaded in the cache, the slave device screen data in the cache is modified, and based on the modified slave device screen data in the cache, the slave device screen data in the database is modified, so as to modify the display screen of the slave device in the virtual interactive interface.

4. The method according to claim 3, further comprising: Clean up the cache and / or the slave device screen data in the database.

5. The method according to any one of claims 1-4, further comprising: The system receives slave device screen data sent by the master device, which is used to generate the modified display screen of the slave device. The master device forwards the slave device screen data sent to the slave device so that the modified display screen is displayed on the slave device; Based on the slave device screen data sent by the master device, update the display screen of the slave device in the virtual interactive interface.

6. The method according to any one of claims 1-4, wherein, The step of generating a virtual interactive interface corresponding to the slave device based on the slave device screen data sent by the slave device includes: Upon first receiving the slave device screen data sent by the slave device, a virtual interactive interface corresponding to the slave device is generated based on the slave device screen data sent by the slave device; and / or, When receiving slave device screen data sent by the slave device for the first time, the display screen of the slave device in the virtual interactive interface is updated based on the slave device screen data sent by the slave device.

7. The method according to any one of claims 1-4, wherein, The slave device screen data in the terminal device uses the same file format as the slave device screen data in the master device and the slave device screen data in the slave device.

8. The method according to any one of claims 1-4, wherein, The virtual interactive interface also includes at least one of the following information: the device identifier of the corresponding slave device, the user tag of the user of the corresponding slave device, and the connection status with the corresponding slave device.

9. The method according to any one of claims 1-4, wherein, When a user of the slave device is detected within the display area, the virtual interactive interface corresponding to the slave device is displayed, including: The system acquires images of the display area and performs human body recognition to identify the first user located within the display area. Based on the matching relationship between the slave device and the user, the slave device matched by the first user is determined; Displays the virtual interactive interface corresponding to the slave device matched by the first user.

10. The method of claim 9, further comprising: Obtain the location of the first user; The virtual interactive interface displaying the slave device matched by the first user includes: Based on the location of the first user, a virtual interactive interface corresponding to the slave device matching the first user is displayed in the surrounding area of ​​the first user, and the distance between the virtual interactive interface and the corresponding first user is less than a first set threshold.

11. The method according to claim 10, wherein, The virtual interactive interface is located above the corresponding slave device or the first user.

12. The method according to claim 10, wherein, Obtaining the location of the first user includes: Images of the display area are captured and human body recognition is performed at set intervals to obtain the location of the first user; The relative position of the virtual interactive interface and the corresponding first user remains unchanged.

13. The method according to any one of claims 1-4, further comprising: In response to a second gesture operation by the user of the terminal device, the display position of the virtual interactive interface is changed, wherein the second gesture operation is used to drag the virtual interactive interface.

14. The method according to any one of claims 1-4, wherein, The terminal device connects to the master device and the slave device using at least one of the following methods: Bluetooth and local area network.

15. The method according to any one of claims 1-4, further comprising: Scan the first slave device located in the surrounding area of ​​the terminal device and establish a connection, wherein the distance between the first slave device and the terminal device is less than a second set threshold.

16. The method of claim 15, further comprising: Obtain the location of the first slave device; The system acquires images of the area surrounding the terminal device and performs human body recognition to obtain the location of a second user located within the area surrounding the terminal device. Based on the location of the first slave device and the location of the second user, determine the first slave device that matches the second user; Disconnect from the first slave device when no second user is matched with the first slave device.

17. The method of claim 16, further comprising: Obtain the device identifier of the first slave device and the user tag of the second user; The user tag of the second user is bound to the device identifier of the matching first slave device to generate a matching relationship between the first slave device and the second user.

18. The method of claim 15, further comprising: After failing to establish a connection with the first slave device, re-establish a connection with the first slave device; The user of the terminal device is notified after the failure to re-establish a connection with the first slave device.

19. The method according to any one of claims 1-4, wherein, When the terminal device receives slave device screen data sent by at least one of the slave devices, the step of forwarding the slave device screen data to the master device for displaying the slave device's screen on the master device includes: In response to a third gesture operation by a user of the terminal device, the screen data of the slave device sent by the second slave device is forwarded to the master device so that the master device displays the screen of the second slave device. The third gesture operation is used to select a user of the slave device as the user of the second slave device.

20. A device interaction method, applied to a master device, comprising: Receive device screen data sent by the terminal device; The slave device screen data is sent from the slave device to the terminal device to generate the display screen of the slave device; Based on the slave device screen data, the display screen of the slave device is displayed.

21. The method according to claim 20, wherein, The display screen of the slave device includes: Simultaneously display the display screen of the master device and the display screen of the slave device; or... Switch the display screen of the master device to the display screen of the slave device.

22. The method according to claim 20 or 21, further comprising: In response to the user's screen operation on the master device, the display screen of the slave device is modified; The modified display screen data of the slave device is acquired and sent to the terminal device, so that the terminal device forwards the modified display screen data of the slave device to the slave device, and the terminal device and the slave device display the modified display screen.

23. A device interaction method applied to a slave device, comprising: Based on the display of images from the device screen; The slave device screen data is sent to the terminal device, so that the terminal device forwards the slave device screen data to the master device, and the terminal device and the master device display the display screen of the slave device.

24. The method of claim 23, further comprising: In response to the user's screen operation on the slave device, the display screen of the slave device is modified; The modified display screen data of the slave device is acquired and sent to the terminal device, so that the terminal device forwards the modified display screen data of the slave device to the master device, and the terminal device and the master device display the modified display screen.

25. An interactive device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method of any one of claims 1-19, or the steps of the method of any one of claims 20-22, or the steps of the method of claim 23 or 24.

26. An interactive system comprising a terminal device, a master device, and at least one slave device, wherein the terminal device is connected to the master device and the slave device respectively, the terminal device implements the steps of the method according to any one of claims 1-19, the master device implements the steps of the method according to any one of claims 20-22, and the slave device implements the steps of the method according to claim 23 or 24.

27. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of any one of claims 1-19, or the steps of the method of any one of claims 20-22, or the steps of the method of claim 23 or 24.

28. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method of any one of claims 1-19, or the steps of the method of any one of claims 20-22, or the steps of the method of claim 23 or 24.

Images