Screen casting method and system, electronic device, storage medium, and program product

Abstract

A screen casting method and system, an electronic device, a storage medium, and a program product. In a screen casting scenario, a target device can "integrate" display information sent from a host device into local display information of the target device, thereby improving user experience in screen casting scenarios. For example, a first device comprises a screen casting auxiliary chip, a main chip, and a display screen, the screen casting auxiliary chip is connected to the main chip, and the main chip is connected to the display screen. The screen casting auxiliary chip receives first display information sent from a second device, and sends the first display information to the main chip. The main chip processes the first display information into second display information locally generated by the first device, and displays the second display information by means of the display screen.

Description

A screen mirroring method, system, electronic device, storage medium, and program product.

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202510382350.7, filed on March 28, 2025, with the title “A screen projection method, system, electronic device, storage medium and program product”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of terminal technology, and in particular to a screen projection method, system, electronic device, storage medium, and program product. Background Technology

[0004] Screen mirroring technology provides convenient services for users in various aspects of work, leisure, and entertainment. Taking screen mirroring from device A to device B as an example, device A is called the host computer, and device B is called the slave computer. Please refer to Figure 1, which is a schematic diagram of a current slave computer structure. As shown in Figure 1, the display screen of the slave computer is connected to the main chip and the screen mirroring auxiliary chip through a multiplexer (MUX). In non-screen mirroring scenarios, the main chip is connected to the display screen through the MUX and displays the local display information through the display screen. In screen mirroring scenarios, the screen mirroring auxiliary chip is connected to the display screen through the MUX. After receiving the display information sent by the host computer, the screen mirroring auxiliary chip displays the display information through the display screen.

[0005] Therefore, it can be seen that there are currently two display transmission paths in the lower-level machine: the local display information generated by the main chip and the display information sent by the upper-level machine are transmitted through different display transmission paths.

[0006] Clearly, the display information sent by the host computer and the local display information of the slave computer are not well integrated, which affects the user experience in the screen projection scenario. Summary of the Invention

[0007] This application provides a screen mirroring method, system, electronic device, storage medium, and program product, which helps to improve the user experience in screen mirroring scenarios.

[0008] Firstly, a screen mirroring method is provided, which can be applied to a first device. The first device can be referred to as a lower-level device. For example, the first device can be a tablet computer or a laptop computer. The first device includes a screen mirroring auxiliary chip, a main chip, and a display screen. The screen mirroring auxiliary chip is connected to the main chip, and the main chip is connected to the display screen. The screen mirroring auxiliary chip receives first display information sent by a second device. The second device can be referred to as a higher-level device. The screen mirroring auxiliary chip sends the first display information to the main chip. The main chip processes the first display information into second display information generated by the first device itself and displays the second display information through the display screen.

[0009] As described in the background section, in current lower-level machines (e.g., the lower-level machine shown in Figure 1), the projection auxiliary chip and the main chip are connected to the display screen, but there is no connection between the projection auxiliary chip and the main chip. Therefore, there are two display transmission paths inside the current lower-level machine, as shown in display transmission path 1 and display transmission path 2 in Figure 1. After receiving the first display information sent by the upper-level machine, the projection auxiliary chip sends the first display information to the display screen for display through display transmission path 2. The first display information does not pass through the main chip.

[0010] Compared with the prior art, the embodiments of this application include at least the following two differences:

[0011] On the one hand, the structure of the lower-level machine provided in this application embodiment differs from the structure of lower-level machines in the prior art. In this application embodiment, the projection auxiliary chip of the lower-level machine is connected to the main chip, and the main chip is connected to the display screen. Therefore, the lower-level machine only needs one display transmission path, namely the display transmission path 1 from the main chip to the display screen. After receiving the first display information sent by the upper-level machine, the projection auxiliary chip does not directly send the first display information to the display screen for display, because there is no display transmission path 2 from the projection auxiliary chip to the display screen. Instead, it sends the first display information to the main chip, and the main chip sends the first display information for display through the display transmission path 1. Therefore, in this application embodiment, the lower-level machine only needs one display transmission path, saving the overhead of the display transmission path.

[0012] On the other hand, in this embodiment, after the main chip receives the first display information from the host computer, it can process the first display information into second display information generated by the first device itself, and display the second display information through the display screen. In this way, the first display information sent by the host computer can be well integrated into the lower device, which helps to improve the screen projection experience.

[0013] Optionally, the screen projection auxiliary chip is connected to the main chip, which can include a direct connection or an indirect connection. An indirect connection can include: the screen projection auxiliary chip and the main chip are connected through an intermediate module, which may be various components and is not limited thereto.

[0014] Optionally, the screen mirroring auxiliary chip and the main chip can be two separate chips, or the screen mirroring auxiliary chip can be integrated into the main chip.

[0015] In one possible design, the main chip processes the first display information into second display information generated by the first device itself, and displays the second display information through the display screen. This includes: the main chip processes the first display information into second display information of a first application of the first device, where the first application creates a first window and displays the second display information within that window. The first application is a local application of the first device. In other words, the lower-level device displays the upper-level device's display information through the application window of a local application, seamlessly integrating the upper-level device's display information into the lower-level device and improving the screen projection experience.

[0016] In one possible design, the main chip processes the first display information into second display information for a first application of the first device. This includes: the main chip generating second display information based on the first display information, the second display information being located in the application package of the first application, and the first application displaying the second display information in the first window based on the application package. In other words, the lower-level device integrates the display information sent by the upper-level device into the application package of its local application, allowing the local application to display the upper-level device's display information through its application window. This method effectively integrates the upper-level device's display information into the lower-level device, improving the screen projection experience.

[0017] In one possible design, the method further includes: after the main chip determines that the screen mirroring of the second device has ended, deleting the second display information from the application package. This saves storage space on the lower-level machine.

[0018] In one possible design, the first window includes a first AI button, and the method further includes: receiving an operation on the first AI button, determining a target area within the first window; and performing AI processing on the displayed content within the target area, wherein the AI ​​processing includes at least one of translation, search, object recognition, copying, and sharing. Therefore, through the first application on the lower-level device, the user can translate, search, recognize, copy, and share the displayed information on the upper-level device, resulting in a better user experience.

[0019] In one possible design, the target area is a user-specified area. Therefore, users can specify a target area according to their needs, enabling the first device to translate, search, recognize objects, copy, and share the content displayed within that target area, resulting in a better user experience.

[0020] In one possible design, the first window includes a second AI button, and the second displayed information is a game screen. The method further includes: receiving an operation on the second AI button and displaying a game guide corresponding to the game screen. For example, the host computer is a game console, and the slave computer is a laptop computer. The game console sends the game screen to the tablet computer for display. The tablet computer displays the game screen through the application window of a local application and can also query game guides, which helps to improve the gaming experience.

[0021] Optionally, the first AI button and the second AI button can be the same AI button or different AI buttons.

[0022] In one possible design, the size of the first window matches the display size of the second device, or the first window is a default size. If the first window's size matches the second device's display size, then the user experience of operating the first window is not significantly different from operating on the second device itself, and the user will not feel uncomfortable. If the first window is a default size, for example, the same as the window size of other local applications on the first device, then it means that the display information from the host computer is well integrated into the slave computer.

[0023] In one possible design, the method further includes: receiving a first operation on the first window, and performing at least one of the following processes on the first window: moving, scaling, maximizing, minimizing, and closing. Therefore, in this embodiment, the first device displays the host computer's display information through the application window of a local screen mirroring application, and the application window of the screen mirroring application supports processes such as moving, scaling, maximizing, and minimizing. From the user's perspective, during the user's operation of the first device, there is no significant difference in the operational experience between the screen mirroring application and other local applications on the first device. In other words, the host computer's display information is well integrated into the lower-level device, improving the screen mirroring experience.

[0024] In one possible design, before the screen projection assist chip receives the first display information sent by the second device, the method further includes: the main chip outputting a first prompt message, which prompts whether to use the first device as the screen projection receiver to receive the screen projection from the second device; after receiving the user's confirmation operation, the main chip sends a notification message to the second device, which notifies the second device to project its screen onto the first device. In this way, users can configure which device (first or second) acts as the host computer and which as the slave computer according to their needs, resulting in a better user experience.

[0025] In one possible design, the main chip includes a video interface supporting a first protocol. Before the projection assist chip sends the first display information to the main chip, the method further includes: the projection assist chip performing protocol conversion on the first display information to convert the protocol of the first display information to the first protocol; and the projection assist chip sending the protocol-converted first display information to the main chip through the video interface. In this way, the projection assist chip can send the first display information sent by the host computer to the main chip, so that the main chip can process the first display information into second display information generated locally, thus integrating the display information.

[0026] In one possible design, the video interface is the Mobile Industry Processor Interface (MIPI) interface. It should be noted that the MIPI interface is merely one example; other interfaces are also possible and not limited to this one.

[0027] In one possible design, the first device further includes a control chip and a MUX. The control chip is connected to the MUX, and the MUX is connected to both the projection assistance chip and the main chip. Before the projection assistance chip receives the first display information sent by the second device, the method further includes: if the control chip determines that the device itself is the projection receiver, it switches the MUX to a first state; in the first state, the MUX connects the projection assistance chip to the second device, enabling the projection assistance chip to receive the first display information sent by the second device. Therefore, by adjusting the state of the MUX through the control chip, projection from one device to another, or from one device to another, can be achieved, offering greater flexibility and a better user experience.

[0028] In one possible design, the method further includes: if the control chip determines that the local device is the screen projection sender, it switches the MUX to a second state; in the second state, the MUX connects the main chip and the second device, and the main chip is configured to project the screen to the second device. Therefore, the control chip can adjust the state of the MUX to achieve screen projection from one device to another, or from one device to another, which is flexible and provides a better user experience.

[0029] In one possible design, the method further includes: the screen projection assist chip receiving first audio information sent by the second device; the screen projection assist chip sending the first audio information to the main chip; the main chip processing the first audio information into second audio information generated by the first device itself, and outputting the second audio information. In this way, the first audio information sent by the host computer can be well integrated into the lower-level device, which helps to improve the screen projection experience.

[0030] In one possible design, the main chip processes the first audio information into second audio information generated by the first device itself and outputs the second audio information. This includes: the main chip processes the first audio information into second audio information of a first application on the first device, whereby the first application is used to output the second audio information, and the first application is a local application of the first device. In other words, the lower-level device outputs the audio information from the upper-level device through a local application, thereby seamlessly integrating the audio information from the upper-level device into the lower-level device and improving the screen projection experience.

[0031] In one possible design, the main chip processes the first audio information into second audio information for a first application of the first device. This includes: the main chip generating second audio information based on the first audio information, the second audio information being located in the application package of the first application, and the first application outputting the second audio information based on the application package. In other words, the lower-level device integrates the audio information sent by the upper-level device into the application package of its local application, allowing the local application to output the upper-level device's audio information through the application package. This method effectively integrates the upper-level device's audio information into the lower-level device, improving the screen mirroring experience.

[0032] In one possible design, the method further includes: after the main chip determines that the screen mirroring of the second device has ended, deleting the second audio information from the application package. This helps save memory space on the lower-level machine.

[0033] In one possible design, the main chip includes an audio interface that supports a second protocol. Before the projection assist chip sends the first audio information to the main chip, the method further includes: the projection assist chip performing protocol conversion on the first audio information to convert the protocol of the first audio information to the second protocol; the projection assist chip sending the protocol-converted first audio information to the main chip through the audio interface. In this way, the projection assist chip can send the first audio information sent by the host computer to the main chip, so that the main chip can process the first audio information into second audio information generated locally, thus achieving audio information integration.

[0034] In one possible design, the audio interface is an integrated circuit-built-in audio I2S interface. It should be noted that the I2S interface is merely one example; other interfaces are also possible and are not limited to this one.

[0035] In one possible design, the method further includes: the projection assist chip receiving third display information sent by a third device; the projection assist chip sending the third display information to the main chip; the main chip processing the third display information into fourth display information for the first application, the first application creating a second window and displaying the fourth display information in the second window. Therefore, there can be multiple host computers, allowing the local application on the slave computer to create multiple windows, each window corresponding to one host computer for displaying the display information of that host computer.

[0036] Secondly, a screen projection method is also provided, applied to a communication system. The communication system includes a first device and a second device. The first device includes a screen projection auxiliary chip, a main chip, and a display screen. The screen projection auxiliary chip is connected to the main chip, and the main chip is connected to the display screen. The method includes: the second device sending first display information to the first device; the screen projection auxiliary chip in the first device receiving the first display information sent by the second device and sending the first display information to the main chip; the main chip processing the first display information into second display information generated by the first device itself and displaying the second display information through the display screen.

[0037] Optionally, the screen projection auxiliary chip is connected to the main chip, which can include a direct connection or an indirect connection. An indirect connection can include: the screen projection auxiliary chip and the main chip are connected through an intermediate module, which may be various components and is not limited thereto.

[0038] Optionally, the screen mirroring auxiliary chip and the main chip can be two separate chips, or the screen mirroring auxiliary chip can be integrated into the main chip.

[0039] In one possible design, the main chip processes the first display information into second display information generated natively by the first device, and displays the second display information through the display screen, including: the main chip processes the first display information into second display information of a first application of the first device, the first application being used to create a first window and display the second display information in the first window, the first application being a native application of the first device.

[0040] In one possible design, the main chip processes the first display information into second display information for a first application of the first device, including: the main chip generating second display information based on the first display information, the second display information being located in the application package of the first application, and the first application displaying the second display information in the first window based on the application package.

[0041] In one possible design, the method further includes: after the main chip determines that the second device has finished casting the screen, it deletes the second display information from the application package.

[0042] In one possible design, the first window includes a first artificial intelligence (AI) button, and the method further includes: receiving an operation on the first AI button, determining a target area within the first window; and performing AI processing on the displayed content within the target area, wherein the AI ​​processing includes at least one of translation, search, object recognition, copying, and sharing.

[0043] In one possible design, the target area is a user-specified area.

[0044] In one possible design, the first window includes a second AI button, and the second displayed information is a game screen. The method further includes: receiving an operation on the second AI button and displaying a game guide corresponding to the game screen.

[0045] In one possible design, the size of the first window matches the display size of the second device, or the size of the first window is the default size.

[0046] In one possible design, the method further includes: receiving a first operation on the first window, and performing at least one of moving, scaling, maximizing, minimizing, and closing the first window.

[0047] In one possible design, before the screen projection assist chip receives the first display information sent by the second device, the method further includes: the main chip outputting a first prompt message, the first prompt message being used to prompt whether to use the first device as a screen projection receiver to receive the screen projection from the second device; after receiving the user's confirmation operation, the main chip sends a notification message to the second device, the notification message being used to notify the second device to project its screen onto the first device.

[0048] In one possible design, the main chip includes a video interface that supports a first protocol. Before the projection assist chip sends the first display information to the main chip, the method further includes: the projection assist chip performing protocol conversion on the first display information to convert the protocol of the first display information into the first protocol; and the projection assist chip sending the protocol-converted first display information to the main chip through the video interface.

[0049] In one possible design, the video interface is a Mobile Industry Processor Interface (MIPI) interface.

[0050] In one possible design, the first device further includes a control chip and a MUX, the control chip being connected to the MUX, the MUX being connected to the projection assist chip and the main chip respectively, and before the projection assist chip receives the first display information sent by the second device, the method further includes: if the control chip determines that the device is the projection receiver, it switches the MUX to a first state; in the first state, the MUX connects the projection assist chip to the second device, so that the projection assist chip receives the first display information sent by the second device.

[0051] In one possible design, the method further includes: if the control chip determines that the local device is the screen projection sending end, then the MUX is switched to a second state; in the second state, the MUX connects the main chip and the second device, and the main chip is configured to project the screen to the second device.

[0052] In one possible design, the method further includes: the screen projection assist chip receiving first audio information sent by the second device; the screen projection assist chip sending the first audio information to the main chip; the main chip processing the first audio information into second audio information generated by the first device itself, and outputting the second audio information.

[0053] In one possible design, the main chip processes the first audio information into second audio information generated by the first device itself and outputs the second audio information, including: the main chip processes the first audio information into second audio information of a first application of the first device, the first application being used to output the second audio information, and the first application being a local application of the first device.

[0054] In one possible design, the main chip processes the first audio information into second audio information for a first application of the first device, including: the main chip generating second audio information based on the first audio information, the second audio information being located in the application package of the first application, and the first application outputting the second audio information based on the application package.

[0055] In one possible design, the method further includes: after the main chip determines that the screen mirroring of the second device has ended, deleting the second audio information from the application package.

[0056] In one possible design, the main chip includes an audio interface that supports a second protocol. Before the screen projection auxiliary chip sends the first audio information to the main chip, the method further includes: the screen projection auxiliary chip performing protocol conversion on the first audio information to convert the protocol of the first audio information to the second protocol; and the screen projection auxiliary chip sending the protocol-converted first audio information to the main chip through the audio interface.

[0057] In one possible design, the audio interface is an integrated circuit-built-in audio I2S interface.

[0058] In one possible design, the method further includes: the screen projection assist chip receiving third display information sent by a third device; the screen projection assist chip sending the third display information to the main chip; the main chip processing the third display information into fourth display information for the first application, the first application being used to create a second window and display the fourth display information in the second window.

[0059] Thirdly, an electronic device is also provided for performing the method as described in the first aspect above.

[0060] Fourthly, a communication system is also provided, comprising: a first device and a second device;

[0061] A first device is used to perform the steps of the first device as described in the second aspect above;

[0062] The second device is used to perform the steps of the second device as described in the second aspect above.

[0063] Fifthly, a computer-readable storage medium is also provided for storing a computer program that, when run on a computer, causes the computer to perform the methods provided in the first or second aspect above.

[0064] In a sixth aspect, a computer program product is also provided, comprising a computer program that, when run on a computer, causes the computer to perform the methods provided in the first or second aspect above.

[0065] In a seventh aspect, a chip is also provided for performing the method as described in the first aspect above.

[0066] Eighthly, a chip system is also provided, comprising: a first chip and a second chip;

[0067] A first chip is used to perform the steps of the first device as described in the second aspect above;

[0068] The second chip is used to perform the steps of the second device as described in the second aspect above.

[0069] For the technical effects that can be achieved in aspects two through eight above, please refer to the description of the technical effects that can be achieved in the corresponding design schemes in aspect one above. This application will not repeat them here. Attached Figure Description

[0070] Figure 1 is a schematic diagram of a lower-level machine provided in an embodiment of this application;

[0071] Figure 2 is a schematic diagram of a communication system provided in an embodiment of this application;

[0072] Figure 3 is a schematic diagram of a screen projection scenario provided in an embodiment of this application;

[0073] Figure 4A is a schematic diagram of a lower-level machine provided in an embodiment of this application;

[0074] Figure 4B is another schematic diagram of a lower-level machine provided in an embodiment of this application;

[0075] Figure 4C is another schematic diagram of a lower-level machine provided in an embodiment of this application;

[0076] Figure 5 is a schematic flowchart of a screen projection method provided in an embodiment of this application;

[0077] Figure 6 is a schematic diagram of the role determination process provided in an embodiment of this application;

[0078] Figure 7 is a schematic diagram of the application package for screen projection in a lower-level machine according to an embodiment of this application;

[0079] Figure 8 is a schematic diagram of the display interface of a screen mirroring application provided in an embodiment of this application;

[0080] Figures 9A to 9C are schematic diagrams of the first window of a screen mirroring application provided in an embodiment of this application;

[0081] Figure 10 is another schematic diagram of the first window of the screen casting application provided in an embodiment of this application;

[0082] Figures 11 and 12 are schematic diagrams of the AI ​​function of a screen casting application provided in an embodiment of this application;

[0083] Figure 13 is a schematic diagram of another structure of the lower-level machine provided in an embodiment of this application;

[0084] Figure 14 is a schematic flowchart of another screen projection method provided in an embodiment of this application;

[0085] Figure 15 is a schematic diagram of another structure of the lower-level machine provided in an embodiment of this application;

[0086] Figure 16 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0087] The technical solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0088] The technical solutions provided in this application are applicable to communication systems. For example, please refer to Figure 2, which is a schematic diagram of a communication system provided in this application. The communication system includes a first device and a second device. The communication system is applicable to screen projection scenarios. Taking screen projection from the second device to the first device as an example, the first device can also be called: a lower-level device, a screen projection receiver, a screen-projected device, an auxiliary device, a destination device, or various other names. Correspondingly, the second device can also be called: a host device, a screen projection sender, a screen projection device, a main device, a source device, or various other names.

[0089] Figure 2 uses a mobile phone as an example of a second device. It should be noted that the second device can be any device other than a mobile phone. For example, it can also be a tablet, laptop, personal computer (PC), or other terminal device; or it can be a wearable device such as a watch or wristband; or it can be a game console, scanner, camera, or other device. Optionally, the game console can be any type of game console, including but not limited to: shooting, fighting, racing, music, puzzle, racing, basketball, football, etc. wait series or Xbox Xbox Series wait A series of home video game consoles; or, Switch wait The second device can be a series of handheld game consoles; or it can be an arcade game console located in public places such as game halls, amusement parks, cinemas, shopping malls, and supermarkets; or it can be a motion-sensing game console that captures the player's body movements / gestures through sensors and converts these movements / gestures into actions in the game; or it can be a virtual reality game console that works in conjunction with virtual reality (VR) glasses or VR gloves; or it can be a mobile game console, i.e., a mobile phone with various types of game applications installed. In short, the embodiments of this application do not limit the type of the second device.

[0090] Figure 2 uses a laptop computer as an example of the first device. It should be noted that the first device can be any other device besides a laptop computer. For example, the first device can be a mobile phone, tablet computer, PC, foldable phone, or other terminal device; or it can be a smart screen, television, or other large-screen device; or it can be a projector, conference screen, or other office equipment; or it can be a vehicle display screen, and so on. In short, the embodiments of this application do not limit the type of the first device.

[0091] Optionally, screen mirroring between the second device and the first device can be one-way or two-way.

[0092] Taking one-way screen mirroring as an example, it can include one-way screen mirroring from a second device to a first device, or one-way screen mirroring from a first device to a second device. One example of one-way screen mirroring from a second device to a first device is that the second device mirrors to the first device, but the first device does not mirror back to the second device. Another example of one-way screen mirroring from a first device to a second device is that the first device mirrors to the second device, but the second device does not mirror back to the first device. Continuing with the example of one-way screen mirroring from a second device to a first device, in this case, the mirroring roles of the first and second devices can be interchanged; for example, one-way screen mirroring from the second device to the first device can switch to one-way screen mirroring from the first device to the second device.

[0093] Taking two-way screen mirroring as an example, it means that while the second device is mirroring its screen to the first device, the first device is also mirroring its screen to the second device. Optionally, one-way screen mirroring can be switched to two-way screen mirroring, and vice versa.

[0094] For ease of understanding, the following explanation will mainly use the example of one-way screen projection from the second device to the first device.

[0095] Optionally, the second device can project its screen to the first device, which may include wired or wireless projection.

[0096] Wired screen mirroring refers to screen mirroring based on wired communication technology. For example, a second device includes a second interface, and a first device includes a first interface. The first interface and the second interface are connected via a data cable, establishing a wired connection between the two devices. Thus, the second device can send display information to the first device for display via the data cable. For instance, the second interface can be any of the following: Universal Serial Bus (USB), Universal Serial Bus Type-C (Type-C), DisplayPort (DP), High Definition Multimedia Interface (HDMI), Video Graphics Array (VGA), or Digital Visual Interface (DVI). The first interface can be any of the following: USB, Type-C, DP, HDMI, VGA, or DVI. The first and second interfaces can be of the same type or different types; there is no limitation.

[0097] Wireless screen mirroring refers to screen mirroring based on wireless communication technology. For example, a second device establishes a wireless communication connection with a first device, and the second device sends display information to the first device for display via wireless communication technology. Exemplary wireless communication technologies may include 2G / 3G / 4G / 5G / 6G or future higher-level mobile communication technologies, or short-range communication technologies such as Wireless Fidelity (Wi-Fi), Bluetooth, Near Field Communication (NFC), and Ultra-Wideband (UWB), without limitation.

[0098] The following embodiments in this article mainly illustrate the example of wired screen projection from a second device to a first device. The second device is referred to as the host computer, and the first device as the slave computer.

[0099] As described in the background section, Figure 1 is a schematic diagram of a current lower-level machine structure. As shown in Figure 1, the current lower-level machine has two display transmission paths: Display Transmission Path 1 and Display Transmission Path 2, indicated by dashed lines in the figure. Display Transmission Path 1 is the path from the main chip to the display screen. The main chip can send the local display information of the lower-level machine to the display screen through Display Transmission Path 1. Display Transmission Path 2 is the path between the projection auxiliary chip and the display screen. The projection auxiliary chip receives the first display information sent by the upper-level machine and then sends the first display information to the display screen through Display Transmission Path 2 for display.

[0100] In Figure 1, the MUX can control whether the display transmission path 2 is on or off. Optionally, the display transmission path 1 can remain on, or the MUX can also control whether the display transmission path 1 is on or off.

[0101] For example, in a non-screen projection scenario, display transmission path 1 is activated. The MUX can control display transmission path 2 to be deactivated. Therefore, the main chip can send the local display information of the lower-level machine to the display screen through display transmission path 1 for display. The local display information of the lower-level machine can include: the lower-level machine's desktop, the application interface of the lower-level machine's local applications, etc. For example, please refer to Figure 3(a), which is a schematic diagram of the lower-level machine's display interface in a non-screen projection scenario. The display interface includes the lower-level machine's desktop and application windows (e.g., the file manager window).

[0102] In a scenario where the second device projects its screen to the first device, the MUX can control the display transmission path 2 to be activated. Optionally, the display transmission path 1 can be activated or deactivated. Taking the deactivated display transmission path 1 as an example, the first device's display screen does not show its own display information, only the first display information sent by the host computer. Taking the activated display transmission path 1 as an example, in this case, both display transmission paths are activated, and the main chip can send the lower device's own display information to the display screen through display transmission path 1 for display. After receiving the first display information sent by the host computer, the projection auxiliary chip sends the first display information to the display screen through display transmission path 2. This process does not involve the main chip. For example, the first display information sent by the host computer can be an image stream, where each frame is a screenshot of the host computer's screen. For example, as shown in Figure 3(b), the projection auxiliary chip displays the first display information sent by the host computer through area A on the display screen. Generally, area A is the default area; in other words, after receiving the first display information, the display screen defaults to displaying the first display information through area A. For example, area A is the display area in the lower right corner of the lower-level device. Therefore, area A, which originally displays the local display information of the lower-level device, is configured to display the first display information sent by the upper-level device in the screen projection scenario.

[0103] It should be noted that, taking Figure 3(b) as an example, the first display information of the host computer is displayed after area A, and the display position and size of the first display information cannot be adjusted. From the user's perspective, the lower-level computer's display screen has an area (i.e., area A) displaying the first display information of the host computer, but this area cannot be moved or scaled, resulting in poor flexibility. It should be understood that the window of the lower-level computer's local application is movable and scalable, making it more convenient for users to operate. Therefore, the fact that area A cannot be adjusted creates a significant difference between area A and the window of the lower-level computer's local application from the user's perspective, giving the user the feeling that the first display information of the host computer is not well integrated into the lower-level computer, thus affecting the user experience.

[0104] It is understandable that the root cause of the above problem is that the lower-level machine has the structure shown in Figure 1. In this structure, the first display information sent by the upper-level machine and the local display information of the lower-level machine are sent through different display channels, which causes the first display information of the upper-level machine and the local display information of the lower-level machine to be unable to be integrated.

[0105] In view of this, this application provides a screen projection method. In this method, the lower-level machine can process the first display information sent by the upper-level machine into the second display information generated by the lower-level machine, so that the first display information of the upper-level machine can be well integrated into the lower-level machine, thereby improving the user experience in the screen projection scenario.

[0106] Please refer to Figure 4A, which is a schematic diagram of a lower-level machine (i.e., a first device) provided in an embodiment of this application. As shown in Figure 4A, the first device includes a projection auxiliary chip, a main chip, and a display screen. The projection auxiliary chip is connected to the main chip, and the main chip is connected to the display screen.

[0107] The screen projection assist chip receives first display information sent by the second device and then sends the first display information to the main chip. Optionally, before sending the first display information to the main chip, the screen projection assist chip can also preprocess the first display information. Preprocessing may include protocol conversion of the first display information so that the protocol of the first display information is converted into a protocol supported by the main chip. For example, if the first device connects to the second device through a Type-C interface, the first display information received by the first device through the Type-C interface is encapsulated by the protocol corresponding to the Type-C interface. The screen projection assist chip can convert the protocol of the first display information into a protocol supported by the main chip, such as the Mobile Industry Processor Interface (MIPI) protocol. Optionally, the screen projection assist chip can be a single chip or a chip system composed of multiple chips, without limitation. For example, the screen projection assist chip can be one or more bridge chips. In addition, the screen projection assist chip can be a standalone chip or a non-standalone chip; for example, the screen projection assist chip is integrated inside the main chip and is part of the main chip.

[0108] The main chip is responsible for processing various tasks in the first device and can be understood as the nerve center and command center of the first device. For example, the main chip can be a system-on-a-chip (SOC) of the first device. The main chip may include one or more processors. For instance, the main chip may integrate one or more of the following processors: central processing unit (CPU), image signal processor (ISP), digital signal processor (DSP), application processor (AP), modem processor, graphics processing unit (GPU), baseband processor, neural network processing unit (NPU), etc. The main chip can run various applications (APPs), such as instant messaging applications, camera applications, browser applications, audio playback applications, video playback applications, etc. After receiving the first display information sent by the screen projection auxiliary chip, the main chip can process the first display information into second display information generated by the first device itself, and then display the second display information through the display screen. The processing procedure will be described in detail later.

[0109] Please compare Figure 4A with Figure 1 mentioned earlier. In Figure 1, the lower-level machine has two display transmission paths (display transmission path 1 and display transmission path 2). The first display information sent by the upper-level machine and the local display information of the lower-level machine are transmitted through different display transmission paths. Unlike Figure 1, in Figure 4A, the lower-level machine only needs one display transmission path, namely display transmission path 1 from the main chip to the display screen (indicated by the dashed line in the figure). After the projection auxiliary chip receives the first display information sent by the upper-level machine, it does not directly send the first display information to the display screen for display, because there is no display transmission path 2 from the projection auxiliary chip to the display screen. Instead, it sends the first display information to the main chip. The main chip processes the first display information into the second display information generated by the first device itself, and then sends the second display information to the display screen through display transmission path 1. Therefore, compared with Figure 1, in Figure 4A, only one display transmission path is needed inside the lower-level machine, that is, there is no need to deploy a MUX between the display screen and the main chip, which can save the overhead of the display transmission path.

[0110] Please refer to Figure 4B, which is another structural schematic diagram of the lower-level machine (i.e., the first device) provided in an embodiment of this application. Figure 4B can be understood as a refinement of Figure 4A. As shown in Figure 4B, the first device includes a first interface, a signal amplifier, a control chip, a MUX, a screen projection auxiliary chip, a main chip, a display screen, and a speaker. The first interface is connected to the signal amplifier, the signal amplifier is connected to the control chip, the control chip is connected to the MUX and the main chip respectively, the MUX is connected to the screen projection auxiliary chip and the main chip respectively, the screen projection auxiliary chip is connected to the main chip, and the main chip is connected to the display screen and the speaker respectively.

[0111] The first interface is used to connect a data cable, the other end of which connects to the second interface of the second device. The types of the first and second interfaces have already been illustrated earlier and will not be repeated here. For example, both the first and second interfaces are Type-C interfaces.

[0112] A signal amplifier is used to amplify the signal transmitted by the second device to ensure signal quality. Optionally, signal amplification may include power amplification and / or frequency amplification. This application does not limit the type of signal amplifier in its embodiments. It should be noted that a signal amplifier may or may not be present, therefore, the signal amplifier is represented by a dashed line in Figure 4B.

[0113] The control chip analyzes the capabilities of the second device, such as determining whether it supports screen mirroring. If the second device does not support screen mirroring (e.g., it's a portable hard drive), the control chip does not need to control the MUX to enter DP in or DP out state (explained later). Instead, it connects the second device to the main chip, allowing the main chip to read / write data from the portable hard drive. If the second device supports screen mirroring, the control chip can also determine which of the first and second devices acts as the host and which as the slave device. This process is also known as "role determination," which will be explained in detail later. If the first device acts as the slave device receiving screen mirroring from the second device, the control chip controls the MUX to enter DP in state; if the first device acts as the host device mirroring to the second device, the control chip controls the MUX to enter DP out state. The DP in and DP out states of the MUX will be explained in detail later. As mentioned earlier, the control chip can analyze the capabilities of the second device. One possible approach is that after the first and second devices are connected, the control chip of the first device sends capability query information to the second device to request a query about the second device's capabilities, including whether the second device supports screen mirroring. The control chip of the first device then determines whether the second device supports screen mirroring based on the capability information returned by the second device. Alternatively, the second device can proactively send capability information to the first device, which can be used to determine whether the second device supports screen mirroring. Optionally, the capability information may include at least one of the following: the second device's device type, device name, device model, manufacturer, and screen mirroring indication information. The screen mirroring indication information indicates whether the second device supports screen mirroring; for example, a screen mirroring indication of 1 indicates that the second device supports screen mirroring, and a screen mirroring indication of 0 indicates that the second device does not support screen mirroring. As an example, the control chip can be a power delivery (PD) chip.

[0114] The MUX is connected to the control chip, the projection auxiliary chip, and the main chip, respectively. The MUX has two states: DP in and DP out. If the first device acts as a lower-level device receiving projection from the second device, the MUX enters the DP in state. This means the control chip connects to the projection auxiliary chip through the MUX, allowing the control chip to send the initial display information from the second device to the projection auxiliary chip. In other words, when the MUX is in the DP in state, the second device projects its screen to the first device, and the information flow is as follows: second device → second interface → first interface → signal amplifier → control chip → MUX → projection auxiliary chip → main chip. If the first device acts as a higher-level device projecting its screen to the second device, the MUX enters the DP out state. This means the control chip connects to the main chip through the MUX, allowing the main chip to send the display information to be projected to the second device to the control chip, which then forwards it to the second device. In other words, when the MUX is in the DP out state, the first device projects its screen to the second device, and the information flow from the first device to the second device can include: the main chip of the first device -> MUX -> control chip -> first interface -> second interface -> second device. Whether the MUX is in the DP in or DP out state is controlled by the control chip. As mentioned earlier, the control chip can perform a "role determination" process, that is, determine which of the first and second devices acts as the host machine and which acts as the slave machine. If the second device acts as the host machine and the first device acts as the slave machine, the control chip controls the MUX to switch to the DP in state; if the second device acts as the slave machine and the first device acts as the host machine, the control chip controls the MUX to switch to the DP out state.

[0115] The screen mirroring assist chip is used to receive first display information sent by the second device. Optionally, it may also receive first audio information sent by the second device. For ease of description, the first display information and the first audio information sent by the second device are collectively referred to as "screen mirroring information". After receiving the screen mirroring information from the second device, the screen mirroring assist chip can perform at least one of the following processing on the screen mirroring information:

[0116] (1) Convert the projection information to a protocol, such as converting it to a protocol supported by the projection assist chip. For example, if the projection assist chip supports the DP protocol and the first interface is a Type-C interface, then the projection assist chip can convert the projection information from the Type-C protocol to the DP protocol. For example, the projection assist chip integrates a Type-C to DP protocol conversion module, which can convert the projection information from the Type-C protocol to the DP protocol.

[0117] (2) Separate the screen projection information to separate the first display information and the first audio information. It should be understood that if the screen projection information includes the first display information but does not include the first audio information, then separation is not necessary.

[0118] (3) Convert the separated first display information into a display protocol supported by the main chip, such as the MIPI protocol. Taking the video interface on the main chip as the MIPI interface as an example, in order for the main chip to receive the first display information sent by the host computer, the projection auxiliary chip converts the protocol of the first display information into the MIPI protocol.

[0119] (4) Convert the separated first audio information into an audio protocol supported by the main chip, such as the integrated circuit sound (I2S) protocol. Taking the audio interface on the main chip as an I2S interface as an example, in order for the main chip to receive the first audio information sent by the host computer, the projection auxiliary chip converts the protocol of the first audio information into the I2S protocol.

[0120] As shown in Figure 4B, the main chip includes a video interface used to establish a video transmission path between the projection auxiliary chip and the main chip. For example, the projection auxiliary chip sends the first display information sent by the host computer to the main chip through this video interface. Exemplarily, the video interface can be a MIPI interface, such as a camera serial interface (CSI), referred to as the MIPI CSI interface. It should be noted that the MIPI CSI interface serves as the communication interface between the main chip and the camera. After the camera acquires image information, it sends the image information to the main chip through the MIPI CSI interface. In this embodiment, the main chip reuses the MIPI CSI interface to communicate with the projection auxiliary chip, that is, it receives the first display information from the host computer sent by the auxiliary projection signal through the MIPI CSI interface. This method of reusing existing interfaces on the main chip does not require adjusting the chip layout of the main chip, resulting in lower overhead. Of course, a new interface can also be added to the main chip to receive the first display information sent by the projection auxiliary chip. After receiving the first display information sent by the screen projection auxiliary chip, the main chip can perform integration processing, such as processing the first display information into second display information generated by the first device itself, and then sending the second display information for display. The specific processing procedure will be explained later.

[0121] As shown in Figure 4B, the main chip includes an audio interface used to establish audio transmission communication between the projection auxiliary chip and the main chip. For example, the projection auxiliary chip sends the first audio information sent by the host computer to the main chip through this audio interface. Exemplarily, the audio interface can be an I2S interface. It should be noted that the I2S interface serves as the communication interface between the main chip and the microphone. After the microphone collects the sound signal, it sends the sound signal to the main chip through the I2S interface. In this embodiment, the main chip reuses the I2S interface to communicate with the projection auxiliary chip, that is, it receives the first audio information sent by the projection auxiliary chip through the I2S interface. This method of reusing existing interfaces on the main chip does not require adjusting the chip layout of the main chip, resulting in lower overhead. Of course, a new interface can also be added to the main chip to receive the first audio information sent by the projection auxiliary chip. After receiving the first audio information sent by the projection auxiliary chip, the main chip can perform integration processing, for example, processing the first audio information into second audio information generated by the first device itself, and then outputting the second audio information. The specific processing procedure will be described later.

[0122] Please refer to Figure 4C, which is a schematic diagram of another structure of the lower-level machine (i.e., the first device) provided in an embodiment of this application. The difference from Figure 4B is that in Figure 4B, the projection auxiliary chip and the main chip are two independent chips, while in Figure 4C, the projection auxiliary chip is built into the main chip as a module within the main chip, called the projection auxiliary module. The function of the projection auxiliary module is the same as that of the projection auxiliary chip in Figure 4B. For example, the projection auxiliary module is used to convert the projection information sent by the upper-level machine into the DP protocol supported by the projection auxiliary module, and to separate the converted projection information into first display information and first audio information. Then, the protocol of the first display information is converted into the MIPI protocol supported by the main chip, and the first display information is sent to the main chip. The protocol of the separated first audio information can also be converted into the I2S protocol supported by the main chip, and the first audio information is sent to the main chip.

[0123] Based on the above, please refer to Figure 5, which is a flowchart illustrating a screen projection method provided in an embodiment of this application. This method can be applied to scenarios where a second device projects its screen to a first device, wherein the first device has the structure shown in Figure 4A, Figure 4B, or Figure 4C. As shown in Figure 5, the process includes:

[0124] S100, the first device is connected to the second device.

[0125] Optionally, the connection method may include wired connection or wireless connection; if it is a wired connection, the second device will project its screen to the first device via wired connection; if it is a wireless connection, the second device will project its screen to the first device via wireless connection.

[0126] S101, the first device receives the first display information sent by the second device.

[0127] As an example, the first display information sent by the second device may include an image stream. For instance, each frame in the image stream may be a screenshot from the second device.

[0128] As another example, the first display information sent by the second device may include: display elements and the display positions of the display elements. For instance, the display elements may be various display elements included on the screen of the second device, and the display positions may be the display locations of the display elements on the screen of the second device.

[0129] Optionally, prior to S101, the first device can also determine whether the second device supports screen mirroring. For details on this, please refer to the description of the control chip in Figure 4A above, which will not be repeated here.

[0130] Optionally, before S101, the first device can also perform a role determination process, that is, determine which of the first device and the second device will act as the host computer and which will act as the slave computer.

[0131] In one possible implementation, the first device and / or the second device can display a prompt message to inquire which device will act as the host computer and which device will act as the slave computer. Taking the prompt message displayed by the first device as an example, the first device displays the prompt message in response to the access of the second device. The prompt message can be in the form shown in Figure 6(a), where the first device displays "Second device (e.g., Huawei mobile phone XXX) has been detected accessing the network. Please select a slave computer." Alternatively, it can display two options: the first device and the second device, allowing the user to select one as the slave computer. Or, the prompt message can also be in the form shown in Figure 6(b), where the first device displays a window, which can be a settings application window. This window includes a "Display Mode" option and a description of "Display Mode": "When connecting other devices, this device acts as the display," so that the user understands the function of "Display Mode." If the user wants the first device to act as the slave computer, they can choose to enable "Display Mode." Alternatively, the query message can also be in the form shown in Figure 6(c), where the first device pops up a control center interface in the lower right area. This interface includes a "Display Mode" icon. When the first device detects that this icon is selected, it enters "Display Mode," meaning the first device acts as a slave device. It should be noted that the query message can also take other forms besides those shown in Figure 6, which are not listed here.

[0132] Considering that pop-up prompts might disturb the user, in another possible implementation, the first device and / or the second device can independently determine which device acts as the host and which as the slave device without prompting the user. As an example, the first device can determine which device acts as the host and which as the slave device based on historical records. For instance, if the first device determines that the second device has previously projected its screen to the first device, it will determine the second device as the host and the first device as the slave device. As another example, before the first device connects to the second device, the user may have pre-configured the first device as the slave device, for example, by enabling "Display Mode" in the settings application window in Figure 6(b). In this case, after the first device connects to the second device, the first device determines the first device as the slave device by querying the settings application, without needing to prompt the user again. As another example, after the second device is connected to the first device, the first device can compare the display size of the first device with the display size of the second device. If the display size of the first device is larger than the display size of the second device, the first device is determined to be the slave device. If the display size of the second device is larger than the display size of the first device, the second device is determined to be the slave device.

[0133] After the first device determines that the second device is the host computer and the first device is the slave computer, it can send a notification message to the second device to notify the second device to project its screen onto the first device, that is, in S101, the second device sends the first display information to the first device. In this embodiment, the projection from the second device to the first device can be either mirror projection or non-mirror projection. Taking mirror projection as an example, the second device sends its current display information to the first device. If the second device's current display information is updated, it sends the updated display information to the first device. In short, the current display information on the second device's screen is consistent with the display information projected onto the first device. Taking non-mirror projection as an example, the display information projected from the second device to the first device can be different from the second device's current display information. In this way, the user can handle other services on the second device without affecting the projection onto the first device.

[0134] S102, the first device processes the first display information into second display information generated by the first device itself.

[0135] In one possible implementation, the first device processes the first display information sent by the second device into second display information generated by a local application of the first device. The local application could be, for example, a screen mirroring application on the first device; that is, the first device can process the first display information sent by the second device into second display information generated by the screen mirroring application.

[0136] Optionally, the screen mirroring application can be a system application or a third-party application, without limitation. The screen mirroring application can be in the form of an application (APP) or a non-APP form. Taking a non-APP form as an example, it can exist in various forms such as a built-in function of the first device's operating system, a mini-program, a service, or a plugin. It is understood that, in addition to the screen mirroring application, the first device can also include other applications, such as image capture applications, instant messaging applications, office software applications, short video applications, shopping applications, audio playback applications, news applications, etc., or one or more of these. Image capture applications, for example, can be camera applications.

[0137] Optionally, the screen mirroring application can be an application that comes pre-installed on the first device or an application downloaded by the user. If the first device does not come pre-installed with a screen mirroring application, a situation might arise in actual use where the first device is used as a slave device for the first time, but the screen mirroring application is not downloaded on it. In this case, the first device can output a prompt message to remind the user to download the screen mirroring application; optionally, it can also display the download address of the screen mirroring application for the user's convenience.

[0138] Continuing with the screen mirroring application as an example, the first device processes the first display information into second display information generated by the screen mirroring application, which may include: the first device generating second display information based on the first display information, and the second display information being located in the application package of the screen mirroring application.

[0139] One possible approach is for the first device to directly write (or store) the first display information into the application package of the screen mirroring application to obtain the second display information. In this case, the first display information and the second display information can be the same.

[0140] Another possible approach is for the first device to generate second display information based on the first display information and write the second display information into the application package of the screen mirroring application. In this case, the second display information is different from the first display information. The generation of the second display information by the first device based on the first display information may include: the first device processing the first display information to obtain the second display information; the processing may include sharpness enhancement, scaling, etc. For example, regarding sharpness enhancement, if the resolution of the first display information sent by the second device is low, or if there is loss during transmission from the second device to the first device resulting in low sharpness of the first display information, the first device can enhance the sharpness of the first display information. If the first display information is an image stream, the first device can use image processing technology to enhance the image sharpness. For example, the image processing technology may be an AI algorithm model for enhancing image sharpness. Regarding scaling, if the first display information sent by the second device is an image stream containing screenshots of the second device, but the image size is not suitable for the window size of the screen mirroring application, the image is scaled (e.g., proportionally scaled) to fit the window size of the screen mirroring application.

[0141] For example, please refer to Figure 7, which is a schematic diagram of the application package of the screen mirroring application in the first device. As shown in Figure 7, the application package of the screen mirroring application includes File 1 and File 2. File 1 can be the file corresponding to the main process of the screen mirroring application; for example, File 1 can be in .exe, .hap, or .apk format. The first device calls File 1 to start the screen mirroring application and run its main process. After receiving the first display information sent by the second device, the first device can generate second display information based on the first display information, which is located in File 2. It should be noted that the first display information sent by the host computer can be a real-time information stream; therefore, the second display information in File 2 can also be a real-time information stream. For example, File 2 can be in .dat format or other formats. File 2 can receive calls from the main process of the screen mirroring application. For example, after the screen mirroring application starts, the main process begins to run. The main process can call File 2 to obtain the second display information within File 2 through function calls or interface calls, and then display the second display information. Optionally, after the screen mirroring ends, the first device can delete the second display information in file 2 to save memory space and make room for the next screen mirroring.

[0142] Therefore, in this way, the first device processes the first display information sent by the second device into the second display information of the first device's local application, as if the display information was generated by the first device's local application itself, thereby integrating the first display information of the host computer into the lower device.

[0143] S103, the first device displays the second display information.

[0144] For example, the first device runs a screen mirroring application, which creates a first window and displays the second display information through that first window. Alternatively, the screen mirroring application obtains the second display information based on an application package and displays it in the first window. Continuing with Figure 7 as an example, after the first device starts the screen mirroring application, the application runs file 1, i.e., the main process. The main process can be used to create the first window and call the second display information in file 2 through function calls or interface calls, displaying the second display information through the first window.

[0145] In some embodiments, the first device starts the screen mirroring application only after step 102 is completed, that is, after the second display information in the application package of the screen mirroring application is generated according to the first display information. In this case, since the second display information is already in the application package, the screen mirroring application can quickly display the second display information after starting, resulting in a better user experience. For example, please refer to Figures 8(a) and 8(b). In Figure 8(a), the first device pops up an inquiry message in response to the access of the second device. After the first device detects the operation to enable "display mode", it receives the first display information sent by the second device, generates the second display information in the application package of the screen mirroring application according to the first display information, and then starts the screen mirroring application. Since the second display information is already in the application package, the screen mirroring application can quickly display the second display information after starting, for example, displaying the first window of the screen mirroring application as shown in Figure 8(b), which includes the second display information.

[0146] In other embodiments, after the first device launches the screen mirroring application, it executes step 102, which generates the second display information in the application package of the screen mirroring application based on the first display information. In this case, since the screen mirroring application is launched first and then the second display information is generated, the second display information can be displayed after a period of time after the screen mirroring application is launched. For example, please refer to Figures 8(a), (c), and (b). In Figure 8(a), the first device pops up an inquiry message in response to the access of the second device. After the first device detects the operation to enable "display mode", it first launches the screen mirroring application, for example, displaying the first window of the screen mirroring application as shown in Figure 8(c), which displays: "Preparing, please wait". After the second display information is generated, the second display information as shown in Figure 8(b) is displayed in the window.

[0147] Optionally, the second device may also send the first audio information to the first device. In this embodiment, the first device may further process the first audio information sent by the second device into second audio information generated by the first device itself, and output the second audio information.

[0148] In one possible implementation, the first device processes the first audio information sent by the second device into second audio information generated by a local application on the first device. The local application could be, for example, a screen mirroring application on the first device; that is, the first device can process the first audio information sent by the second device into second audio information generated by the screen mirroring application. For information on screen mirroring applications, please refer to the preceding description.

[0149] The first device processes the first audio information into second audio information generated by the screen mirroring application. This can include: the first device generating second audio information based on the first audio information, with the second audio information located in the application package of the screen mirroring application. One possible approach is for the first device to write (or store) the first audio information into the application package of the screen mirroring application to obtain the second audio information. Optionally, the second audio information may be the same as or different from the first audio information. For example, after the first audio information is written into the application package of the screen mirroring application, the screen mirroring application can process the first audio information to obtain the second audio information. The processing may include noise reduction, voice enhancement, etc.

[0150] Continuing with Figure 7 as an example, the application package of the screen mirroring application includes file 3. After the first device receives the first audio information sent by the second device, it generates second audio information based on the first audio information, which is located in file 3. It should be noted that the first audio information sent by the host computer can be a real-time information stream; therefore, the second audio information in file 2 can also be a real-time information stream. For example, file 3 can be in .dat format or other formats. File 2 and file 3 can be the same file or different files, without limitation. File 3 can receive calls from the main process of the screen mirroring application. For example, after the screen mirroring application starts, the main process begins to run. The main process can call file 3 to obtain the second audio information in file 3 through function calls or interface calls, and then output the second audio information. Optionally, after screen mirroring ends, the first device can delete the second audio information in file 3 to make room for the next screen mirroring session.

[0151] Therefore, in this way, the first device processes the first audio information sent by the second device into the second audio information of the first device's local application, as if the first device's local application itself generated the audio information, thereby integrating the first audio information of the host computer into the lower device.

[0152] In the above embodiment, the screen mirroring application displays the second display information through a first window. Next, the first window of the screen mirroring application will be described.

[0153] In some embodiments, the size of the first window of the screen mirroring application can be determined according to the display size of the second device. For example, referring to Figure 9A, the size of the first window of the screen mirroring application in the first device matches the display size of the second device. This matching can include: the sizes being exactly the same, or the sizes not being exactly the same but having the same aspect ratio. Taking the first display information sent by the second device to the first device including an image stream as an example, where each frame of the image stream is a screenshot of the second device, if the size of the first window of the screen mirroring application is exactly the same as the display size of the second device, then the first device can simply display the image stream within the first window of the screen mirroring application without scaling the images within the image stream. If the size of the first window of the screen mirroring application is not exactly the same as the display size of the second device but has the same aspect ratio, then the first device can scale the images in the image stream proportionally and then display the processed image stream within the first window of the screen mirroring application. Optionally, the second device may support screen mirroring of multiple sizes (i.e., resolutions). For example, the second device supports screen mirroring of multiple resolutions such as 1280*720 and 720×720. If the second device projects its screen to the first device at a resolution of 1280*720, then the window size of the projection application on the first device will be adapted to 1280:720. If the second device projects its screen to the first device at a resolution of 720*720, then the window size of the projection application on the first device will be adapted to 720*720. In this case, the first device can determine the projection resolution of the second device in advance. One possible approach is that after the first and second devices are connected, the second device determines a projection resolution from among several options and then notifies the first device of the projection resolution. One possible way for the second device to determine a projection resolution from among several options is that the second device displays multiple resolution options, allowing the user to select the projection resolution from among them. After the second device notifies the first device of the projection resolution, the first device creates the first window of the projection application based on that resolution, so that the window size of the projection application is adapted to the resolution.

[0154] In other embodiments, the size of the first window of the casting application is a default size, which may differ from the display size of the second device. For example, referring to Figure 9B, the default window size of the casting application is square, but the display of the second device is rectangular. In this case, before casting, the second device can process the display information to be cast to the first device, processing it to fit the default window size of the casting application, and then send the processed display information to the first device. One possible approach is, as shown in Figure 9C, where the second device includes a local display screen. In this case, if casting to the first device is desired, the second device can create a virtual display (VD) and map the display information on the local display screen to the VD. The display information within the VD is configured to be cast to the first device. Before creating the VD, the second device can obtain the default window size of the casting application on the first device and then create the VD based on that size, ensuring that the size of the VD matches the default window size of the casting application on the first device. One possible mapping method is mirror mapping / casting, where each frame of the image on the local display screen is captured and mapped to the VD. Since the VD size is inconsistent with the local display size, the screenshot can be cropped to fit the VD size. Another possible mapping method is non-mirroring mapping / casting, where the local display shows the interface of application 'a'. If the interface of application 'a' is to be cast to the first device, the second device can move application 'a' to the VD to run, while the local display shows the interface of application 'b' (e.g., the desktop). After application 'a' is moved to the VD to run, application 'a' itself can adjust the layout of the displayed information according to the size of the VD to fit the VD size.

[0155] The following explanation uses the first window of the screen mirroring application shown in Figure 9B as an example.

[0156] The first window of the screen mirroring application can include multiple areas, such as a title bar and a workspace; optionally, it can also include a scroll bar. The title bar can be located at the top of the first window. The width of the title bar can be equal to the width of the first window, as shown in Figure 10(a), or smaller than the width of the first window, as shown in Figure 10(b). In Figure 10(a), when the width of the title bar is equal to the width of the first window, the title bar can include the application name, application icon, and one or more buttons, such as a maximize button, a minimize button, a close button, and may also include an AI control and an audio mode button. The functions of the AI ​​button and the audio mode button will be explained later. In Figure 10(b), when the title bar is smaller, the title bar may not need to include the application name, application icon, etc., and the number of buttons can be relatively small, such as including the close / exit button, AI button, and audio mode button as shown in Figure 10(b). To avoid obscuring the interface, the title bar can be hidden when needed and hidden when not needed. The workspace is used to display secondary information. A scroll bar is used when a window cannot display all its contents. A vertical or horizontal scroll bar appears on the right or bottom border of the window to allow you to view other content in the window.

[0157] In this embodiment, the first device receives an operation on the first window and performs at least one of the following processes: moving, scaling, maximizing, minimizing, and closing the first window. For example, if the first device receives an operation to move the first window, it moves the display position of the first window. If the first device receives an operation to shrink / enlarge the first window, it shrinks / enlarges the first window. If the first device receives an operation on the maximize button, it maximizes the first window, for example, by displaying it in full screen. If the first device receives an operation on the minimize button, it minimizes the first window, for example, by switching it to the background. If the first device receives an operation on the close button (or exit button), it closes the first window. Optionally, closing the first window may include: exiting the screen casting application, or not exiting the screen casting application, only closing the first window of the screen casting application, while leaving other windows of the screen casting application open.

[0158] As mentioned earlier, the first window of the screen mirroring application includes an audio mode button. This button controls whether the second device sends audio information to the first device, or whether the first device outputs (i.e., plays) the audio information sent by the second device. For example, if the first device determines that the audio mode button is in a first state (e.g., selected state), it can send a notification message to the second device to notify the second device to stop sending audio information to the first device, or the second device can continue sending audio information to the first device, but the first device will not play the audio information. If, based on user operation, the first device determines that the audio mode button is in a second state (e.g., unselected state), it will send a notification message to the second device to notify the second device to send audio information to the first device, and the first device will play the audio information after receiving it. Alternatively, if the second device continuously sends audio information to the first device before the audio mode button is in the second state, but the first device does not play the audio information, then after the audio mode button switches to the second state, the first device will play the audio information.

[0159] As mentioned earlier, the first window of the screen mirroring application includes an AI button, which is used to perform AI processing on the second displayed information within the first window of the screen mirroring application. The AI ​​processing includes at least one of the following: copying, sharing, translating, searching, object recognition, and strategy lookup.

[0160] For example, as shown in Figure 11(a), the first window of the screen mirroring application displays second display information, including text and images. In response to an operation on the AI ​​button, the screen mirroring application enters an editable state, allowing the user to define a target area within the first window. For example, as shown in Figure 11(b), the user selects the target area (the area enclosed by the circle) by drawing a circle. As shown in Figure 11(c), in response to the user's circle operation, one or more buttons are displayed in the first window of the screen mirroring application, such as a copy button, share button, translate button, search button, object recognition button, and guide button. Taking the copy button as an example, after the first device detects an operation on the copy button, it copies the displayed content within the target area. The first device can then paste this displayed content into other applications on the first device. Taking the share button as an example, after the first device detects an operation on the share button, it can display a quick share button, such as the application icon of an instant messaging application on the first device or a contact icon within an instant messaging application. In this way, information from the second device can be quickly shared to the instant messaging application on the first device. Taking the search button as an example, after the first device detects an operation on the search button, it launches the browser application within the first device to search for the displayed content in the target area. Taking the object recognition button as an example, after the first device detects an operation on the object recognition button, it launches the object recognition application within the first device (e.g., Huawei Smart Object Recognition) to recognize the displayed content in the target area. Taking the translation button as an example, after the first device detects an operation on the translation button, it translates the displayed content in the target area.

[0161] For example, as shown in Figure 12(a), the first window of the screen mirroring application displays second display information, such as a game screen. In response to an operation on the AI ​​button, the screen mirroring application enters an editable state, allowing the user to define a target area within the first window. For example, as shown in Figure 12(b), the user selects the target area (i.e., the area enclosed by the circle) by drawing a circle. As shown in Figure 12(c), in response to the user's circle operation, one or more buttons are displayed in the first window of the screen mirroring application, including a strategy button. After receiving an operation on the strategy button, the first device displays the game strategy as shown in Figure 12(d). Optionally, the game strategy can be a walkthrough for the current level. It is understood that the first device can query the game strategy before displaying it, for example, by querying it in a browser application or other applications, and then display the game strategy. Figure 12 is applicable to screen mirroring scenarios where the host computer is a game console and the slave computer is a PC. The host computer can be various types of game consoles, which have been exemplified previously and will not be repeated here.

[0162] It should be noted that in Figures 11 and 12, the target area is a user-specified area as an example. In other embodiments, the target area can be an area automatically determined by the screen casting application. For example, the target area could be the entire display area within the screen casting application window, or the middle display area, etc., without requiring the user to specify the target area. Alternatively, there is no need to determine the target area; the target area defaults to the entire display area within the window.

[0163] Furthermore, in Figures 11 and 12, the copy, share, translate, search, object recognition, and strategy guide buttons pop up after detecting a user's selection action. Optionally, one or more of these buttons can be directly displayed in the first window, such as in the title bar. For example, in Figure 12, the strategy guide button can be directly displayed in the title bar. After the first device receives an action on the strategy guide button, it displays the game strategy guide; this process does not require the user to perform a selection action.

[0164] Please compare the application window of the screen mirroring application in this embodiment with area A in Figure 3 above. Area A in Figure 3 is used to display the display area sent by the host computer, but this area is not in the form of an application window. For example, it does not have a title bar, work area, scroll bar, etc., and this area does not support movement or scaling. In this embodiment, the first device displays the display information of the host computer through the application window of the local screen mirroring application. The application window of the screen mirroring application includes a title bar, work area, scroll bar, etc., and supports movement, scaling, maximization, minimization, etc. From the user's perspective, during the operation of the first device, there is no significant difference in the operating experience between the screen mirroring application and other local applications of the first device. In other words, the display information of the host computer is well integrated into the lower device, improving the screen mirroring experience. In addition, the screen mirroring application also has AI functions, which can perform AI processing on the display information of the host computer, further improving the user experience.

[0165] Please refer to Figure 13, which is another structural schematic diagram of the first device (i.e., the lower-level machine) provided in an embodiment of this application. Figure 13 can be understood as the software structure of the first device. As shown in Figure 13, the software structure of the first device can be a layered structure. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. The layered architecture can be, for example, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. This application embodiment uses a layered architecture. Taking the system as an example, the software structure of the lower-level machine is illustrated. In some embodiments, the lower-level machine will be... The system is divided into five layers, from top to bottom: application layer, application framework (FWK) layer, hardware abstraction layer (HAL) layer, kernel layer, and hardware layer.

[0166] The application layer can include a range of applications. These include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and browser. It can also include screen mirroring applications and AI applications. AI applications include, for example, object recognition applications and translation applications.

[0167] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications within the application layer. The application framework layer includes some predefined functions. As shown in Figure 13, the application framework layer includes a display framework and an audio framework. The display framework is used to create application windows and display information within them. The audio framework is used to receive calls from the upper application layer and output audio. Figure 13 shows an example where the display framework is located within the FWK layer; optionally, the display framework can also be located within the HAL layer, or partly within the FWK layer and partly within the HAL layer, without limitation. Similarly, Figure 13 shows an example where the audio framework is located within the FWK layer; optionally, the audio framework can also be located within the HAL layer, or partly within the FWK layer and partly within the HAL layer, without limitation. It is understood that, although not shown in Figure 13, the application framework layer may also include content providers, a view system, a phone manager, a resource manager, a notification manager, etc. Content providers are used to store and retrieve data, making this data accessible to applications. The data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying images. The phone manager is used to provide communication functions for electronic devices. For example, managing call status (including connection, hang-up, etc.). The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc. The notification manager allows applications to display notification information in the status bar, which can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify of download completion, message reminders, etc. The notification manager can also be a notification that appears in the system's top status bar as a chart or scrollbar text, such as notifications from background applications, or a notification that appears on the screen as a dialog window. For example, displaying text messages in the status bar, emitting alert sounds, vibrating the electronic device, flashing indicator lights, etc.

[0168] The HAL layer acts as a bridge between the kernel layer and the application framework layer. It converts data provided by the lower kernel layer into data adaptable to the application framework layer, which then provides the data to the uppermost application layer. For example, the HAL layer includes a display information adaptation path and an audio information adaptation path. The display information adaptation path converts the first display information from the host computer transmitted from the kernel layer into information recognizable by the upper layer, such as decompressing the first display information and converting its encoding format to a format supported by the upper layer. The audio information adaptation path converts the first audio information sent from the host computer transmitted from the kernel layer into information recognizable by the upper layer, such as decompressing the audio information and converting its encoding format to a format supported by the upper layer. It should be noted that Figure 13 shows an example where the display information adaptation path is located within the HAL layer. Optionally, the display information adaptation path can also be located in the FWK layer, or partly in the FWK layer and partly in the HAL layer; this is not limited. Similarly, in Figure 13, taking the audio information adaptation path located within the HAL layer as an example, optionally, the audio information adaptation path can also be located in the FWK layer, or partly in the FWK layer and partly in the HAL layer; there are no limitations. The HAL layer can also include a display enhancement module and an audio enhancement module. The display enhancement module is used to optimize the display information, such as enhancing clarity and proportional scaling. The audio enhancement module is used to optimize the audio information, such as noise reduction and voice enhancement.

[0169] The kernel layer includes various hardware drivers used to obtain information from the hardware layer and then send that information to the upper layer, the HAL layer. For example, the kernel layer may include camera drivers, microphone drivers, speaker drivers, display drivers, and screen mirroring assist drivers. These drivers power the screen mirroring assist chip; please refer to the previous description for more information on screen mirroring assist chips.

[0170] The hardware layer includes various hardware components, such as displays, cameras, and speakers, and may also include screen projection auxiliary chips.

[0171] It is understood that the structure shown in Figure 13 does not constitute a specific limitation on the software structure of the lower-level machine. The software structure of the lower-level machine in the embodiments of this application may include more or fewer modules than shown in Figure 13, for example, more or fewer layers than shown in Figure 13, or a certain layer may include more or fewer modules than shown in Figure 13. In addition, the combination / connection relationship between the modules in Figure 13 can also be adjusted and modified.

[0172] It should be noted that the current screen mirroring process is as shown by the dotted line in Figure 13. After receiving the first display information sent by the host computer, the screen mirroring auxiliary chip calls the display driver to drive the display screen to show the first display information. If the host computer also sends the first audio information, the speaker driver is called to drive the speaker to play the first audio information. In other words, in the current screen mirroring process, after the screen mirroring auxiliary chip receives the information sent by the host computer, the information passes through the driver layer and the hardware layer, but does not pass through higher layers, such as the HAL layer, the application framework layer, and the application layer.

[0173] Next, based on Figure 13, the screen projection process provided in the embodiments of this application will be described.

[0174] After receiving the projection information from the host computer, the projection assistance chip in the hardware layer determines whether the projection information includes first display information and first audio information. If so, it separates the first display information and first audio information. The projection assistance chip then sends the separated first display information and first audio information to the projection assistance chip driver in the driver layer. This driver then sends the first display information and first audio information to the upper layer, namely the HAL layer. It should be noted that the projection assistance chip driver may or may not be present, so it is represented by a dashed line in Figure 13. If this driver is not present, other drivers in the kernel layer can be reused, such as the camera driver and / or microphone driver. For example, the projection assistance chip calls the camera driver in the kernel layer to send the first display information to the HAL layer, and calls the microphone driver in the kernel layer to send the first audio information to the HAL layer.

[0175] After receiving the first display information, the HAL layer processes it through a display information adaptation path, such as decompressing and converting the encoding format, to convert it into an encoding format supported by the upper layer (e.g., the application layer). The display information adaptation path can be a new module within the HAL layer, or it can reuse an existing module. For example, as shown in Figure 13, the HAL layer also includes a camera frame, and the display information adaptation path can reuse the camera frame.

[0176] After receiving the first audio information, the HAL layer processes it through an audio information adaptation path, such as decompressing and converting the encoding format, to convert it into an encoding format supported by the upper layer (e.g., the application layer). The audio information adaptation path can be a new module within the HAL layer, or it can reuse an existing module. For example, as shown in Figure 13, the HAL layer also includes an audio framework, and the audio information adaptation path can reuse this audio framework.

[0177] The display information adaptation path sends the first display information to the display framework in the FWK layer. One possible approach is for the display framework to write the first display information into the application package of the casting application. Another possible approach is for the display framework to call the display effect enhancement module to process the first display information (e.g., sharpness enhancement, scaling, etc.) to obtain the second display information, and then write the second display information into the application package of the casting application. Taking the display framework writing the first display information into the application package of the casting application as an example, the casting application can window the first display information for display. For example, as shown in Figure 13, the casting application calls the display framework, creates the first window of the casting application through the display framework, and sends the first display information to the display framework. The display framework is used to call the display effect enhancement module in the HAL layer to perform display optimization processing on the first display information (e.g., sharpness enhancement, scaling, etc.) to obtain the second display information. The display effect enhancement module is also used to call the display driver to drive the display screen to display the second display information.

[0178] The audio information adaptation path sends the first audio information to the audio framework in the FWK layer. One possible approach is for the audio framework to write the first audio information into the application package of the casting application. Another possible approach is for the audio framework to call the audio effects enhancement module to process the first audio information (e.g., noise reduction, voice enhancement) to obtain the second audio information, and then write the second audio information into the application package of the casting application. Taking the audio framework writing the first audio information into the application package of the casting application as an example, the casting application can output the first audio information. For example, as shown in Figure 13, the casting application calls the audio framework, which in turn calls the audio effects enhancement module in the HAL layer. The audio effects enhancement module is used to optimize the first audio information (e.g., noise reduction, voice enhancement) to obtain the second audio information, and it also calls the speaker driver to drive the speaker to play the second audio information.

[0179] As shown in Figure 13, the screen mirroring application also includes settings management and interaction modules. Taking settings management as an example, the application window displays an audio mode button (see Figures 10, 11, or 12 above). This audio mode button allows users to set whether the host computer sends audio information to the slave device, or whether the slave device plays audio information sent by the host computer. Taking the interaction module as an example, the application window displays an AI button. After receiving an operation on the AI ​​button, the user can select a target area within the window to perform AI processing on the content within that area. For example, the screen mirroring application can call the AI ​​framework in the FWK layer. The AI ​​framework is used to obtain the displayed content within the target area of ​​the application window. After obtaining the displayed content, the AI ​​application performs AI processing on the displayed content. Exemplary AI applications may include Huawei Smart Object Recognition, translation applications, etc.

[0180] As shown in Figure 13, the audio framework includes a scene control module, which can be used to control whether the audio information of the screen mirroring application plays based on the current scene. For example, if the second device is currently making a phone call, the scene control module will control the audio information of the screen mirroring application to pause playback.

[0181] Please refer to Figure 14, which is a schematic flowchart of another screen projection method provided in an embodiment of this application. This flowchart can be applied to the structure shown in Figure 14. As shown in Figure 14, the flowchart includes:

[0182] Step 0: Enable the DP in function in the settings application of the first device. The DP in function allows the first device to act as a slave device to receive screen projections from other devices.

[0183] For example, enabling the DP in function in the settings application may include: the settings application displays the window shown in Figure 6(b); after receiving an operation to enable "display mode", the settings application calls the role setting module in the FWK layer, and the role setting module powers on the control chip in the hardware layer. After the control chip powers on, the first device is set as a slave device, for example, switching the MUX to DP in state. Optionally, the control chip can also feed back the role setting result to the settings application, such as whether the switching of the MUX to DP in state was successful. Optionally, step 0 may or may not be executed. For example, if the first device is a slave device by default and does not need to enable the DP in function, then step 0 does not need to be executed. Therefore, step 0 in Figure 14 is represented by a dashed line. Optionally, the role setting module in Figure 14 may or may not be present (for example, if the first device is a slave device by default, then role setting is not required), so it is represented by a dashed line. Optionally, the control chip in the hardware layer may or may not be present. For example, if the first device only supports being a slave device and does not support being a master device, then a control chip does not need to be deployed, so the control chip in Figure 14 is represented by a dashed line.

[0184] Step 1: The settings application on the first device prompts you to connect to the second device.

[0185] For example, the setup application of the first device outputs a prompt message to indicate the connection of the second device. After seeing the prompt message, the user connects the first device to the second device. Optionally, the setup application of the first device can detect the insertion status of the second device. If the second device is successfully inserted, it can determine the projection resolution of the second device. For example, as shown in Figure 14, the setup application calls the camera frame in the FWK layer, calls the camera HAL in the HAL layer through the camera frame, and then calls the camera driver in the kernel layer to power on the projection auxiliary chip through the camera driver. The projection auxiliary chip can be used to determine the projection resolution of the second device. The process of determining the projection resolution has been described above and will not be repeated. Optionally, after the projection auxiliary chip determines the projection resolution of the second device, it reports the projection resolution layer by layer to the setup application. The setup application can send the projection resolution of the second device to the projection application so that the projection application can create a window based on the projection resolution, thereby making the window size of the projection application compatible with the projection resolution of the second device. It should be noted that the process of the projection auxiliary chip determining the projection resolution of the second device can be executed or not, so it is represented by a dashed line in Figure 14.

[0186] It should be noted that step 1 may or may not be executed, hence the dashed line in Figure 14. Taking the case where step 1 is not executed as an example, one possible scenario is that neither step 0 nor step 1 is executed. For instance, the user directly connects the first device to the second device, and the first device defaults to acting as the lower-level device. In this case, the settings application of the first device detects the connection of the second device and can power on the projection assist chip. For example, the settings application calls the camera framework in the FWK layer, which in turn calls the camera HAL in the HAL layer, and then calls the camera driver in the kernel layer to power on the projection assist chip via the camera driver.

[0187] Step 2: The settings application in the first device initializes the adapter path in the first device.

[0188] The adaptation path in the first device may include a display information adaptation path, and optionally, an audio information adaptation path (indicated by dashed lines in the figure). Please refer to Figure 13 above for details on the display information adaptation path and the audio information adaptation path. Therefore, initializing the adaptation path may include: initializing the display information adaptation path, and optionally, initializing the audio information adaptation path. For example, initializing the display information adaptation path may include: initializing the encoding format adjustment module in the display information adaptation path (e.g., powering on the encoding format adjustment module), so that the encoding format adjustment can adjust the first display information of the host computer to an encoding format supported by the upper layer (e.g., the application layer). For example, initializing the audio information adaptation path may include: initializing the encoding format adjustment module in the audio information adaptation path (e.g., powering on the encoding format adjustment module), so that the encoding format adjustment can adjust the first audio information of the host computer to an encoding format supported by the upper layer (e.g., the application layer).

[0189] Taking a display information adaptation path located in the FWK layer and an audio information adaptation path located in the HAL layer as an example, as shown in Figure 14, the settings application in the first device can initialize the display information adaptation path in the FWK layer and the audio information adaptation path in the HAL layer. It should be noted that step 2 can be executed or not, so it is represented by a dashed line in Figure 14.

[0190] Step 3: The settings application in the first device notifies the second device to start data streaming, that is, notifies the second device to start screen mirroring.

[0191] For example, as shown in Figure 14, the application sends a start streaming command to the display information adaptation path. Optionally, it can also send a start streaming command to the audio information adaptation path. The display information adaptation path (or audio information adaptation path) calls the camera driver to send a start streaming command to the projection assist chip. The projection assist chip can send a notification message to the second device to notify it to start projection. After receiving the projection information, the projection assist chip separates the projection information, sending the separated first display information to the projection application through the display information adaptation path, and sending the separated first audio information to the projection application through the audio information adaptation path.

[0192] It should be noted that there is a possibility that the first device may start screen mirroring immediately after connecting to the second device, meaning that the first device does not need to notify the second device to start screen mirroring. Therefore, the process of starting the streaming and the process of the screen mirroring auxiliary chip sending a notification message to the second device in Figure 14 are represented by dashed lines.

[0193] Step 4: The screen mirroring application in the first device is displayed.

[0194] For example, as shown in Figure 14, the screen mirroring application invokes the display framework in the FWK layer to create the first window of the screen mirroring application. The display framework calls the display enhancement module in the HAL layer to optimize the first display information and obtain the second display information. The display effect enhancement module calls the display driver to drive the display screen to display the second display information. As shown in Figure 14, the screen mirroring application invokes the audio framework in the FWK layer. The audio framework calls the sound effect enhancement module in the HAL layer to optimize the first audio information and obtain the second audio information. The sound effect enhancement module calls the speaker driver to drive the speaker to play the second audio information. As mentioned earlier, the second device may or may not send the first audio information to the first device, so the process of the screen mirroring application outputting audio information in Figure 14 is represented by dashed lines.

[0195] The above embodiments use a single host computer as an example. In other embodiments, there can be multiple host computers. For example, the first device supports screen projection from multiple devices to the first device. For example, the first device is connected to a second device and a third device respectively, wherein the first device acts as a slave device, and the second and third devices both act as host computers, that is, the second and third devices both project their screens onto the first device.

[0196] Please refer to Figure 15, which is another structural diagram of the first device, i.e., the lower-level machine. As shown in Figure 15, the first device includes: a first interface, a signal amplifier 1, a control chip 1, a MUX 1, and a projection auxiliary chip 1; and a third interface, a signal amplifier 2, a control chip 2, a MUX 2, and a projection auxiliary chip 2; it also includes a main chip, a display screen, and a speaker. The main chip includes a video interface 1, an audio interface 1, a video interface 2, and an audio receiver 2. Optionally, signal amplifier 1 and signal amplifier 2 can be the same signal amplifier or two different signal amplifiers. Control chip 1 and control chip 2 can be the same control chip or two different control chips. MUX 1 and MUX 2 can be the same MUX or two different MUXs. Projection auxiliary chip 1 and projection auxiliary chip 2 can be the same chip or two different chips. Video interface 1 and video interface 2 on the main chip can be the same video interface or two different video interfaces, and audio interface 1 and audio interface 2 can be the same audio interface or two different audio interfaces.

[0197] The first device connects to the second device through the first interface. The first display information sent by the second device passes through signal amplifier 1, control chip 1, MUX 1, and projection auxiliary chip 1 to reach the main chip. The main chip processes the first display information into second display information for the projection application and displays the second display information through the first window of the projection application. This process has been described in detail above and will not be repeated.

[0198] The first device connects to the third device via the third interface. The third display information sent by the third device passes through signal amplifier 2, control chip 2, MUX 2, and projection auxiliary chip 2 to reach the main chip. The main chip can process the third display information into a fourth display information for the projection application and display the fourth display information through the second window of the projection application. The principle of processing the third display information into the fourth display information for the projection application is the same as the principle of processing the first display information into the second display information for the projection application, and will not be repeated. In addition, the second window operates on the same principle as the first window. For example, the second window also supports moving, scaling, maximizing, and minimizing, and it also has an AI button for AI processing of the displayed information within the second window. AI processing includes at least one of copying, sharing, translating, searching, object recognition, and strategy querying.

[0199] Therefore, the screen mirroring application creates two different windows, each displaying information from a different host computer.

[0200] Figure 16 is a schematic diagram of the structure of the electronic device 1600 provided in an embodiment of this application. The electronic device 1600 can be either the first device (lower-level device) or the second device (upper-level device) mentioned above. As shown in Figure 16, the electronic device 1600 may include: one or more processors 1601; one or more memories 1602; a communication interface 1603; and one or more computer programs 1604. The above-mentioned devices can be connected through one or more communication buses 1605. The one or more computer programs 1604 are stored in the memory 1602 and configured to be executed by the one or more processors 1601. The one or more computer programs 1604 include instructions. For example, when the electronic device 1600 is the first device (lower-level device) mentioned above, the instructions can be used to execute the relevant steps of the first device (lower-level device) as described in the corresponding embodiments above, such as executing the relevant steps of the first device (lower-level device) in Figures 1 to 15. When electronic device 1600 is the second device, i.e., the host computer, as mentioned above, this instruction can be used to execute the relevant steps of the second device, i.e., the host computer, as described in the corresponding embodiments above, such as executing the relevant steps of the second device, i.e., the host computer, as shown in Figures 1 to 15. Communication interface 1603 is used to enable communication between electronic device 1600 and other devices; for example, the communication interface can be a transceiver.

[0201] Based on the above, this application also provides a communication system, which includes a first device and a second device, and can be used to execute the screen projection method provided in any of the above method embodiments.

[0202] Based on the above, this application also provides a computer-readable storage medium storing instructions that, when executed, cause the method provided in any of the above-described method embodiments to be implemented. The computer-readable storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory, random access memory, magnetic disk, or optical disk.

[0203] Based on the above, this application also provides a computer program product, which includes: a computer program (also referred to as code or instructions), which, when run on a computer, causes the computer to perform the method provided in any of the above-described method embodiments. Optionally, the computer may be the first device or the second device described above.

[0204] In all the above implementation schemes, unless otherwise specified or there is a logical conflict, the terminology and / or descriptions of different implementation schemes are consistent and can be referenced by each other. The technical features of different implementation schemes can be combined to form new implementation schemes according to their inherent logical relationships.

[0205] The embodiments of this application involve at least one, including one or more; where "multiple" means two or more. Furthermore, it should be understood that in the description of this specification, terms such as "first," "second," and "third" are used only for descriptive purposes and should not be construed as indicating relative importance or order. For example, "first device" and "second device" do not represent the degree of importance of the two or their order, but are merely for descriptive distinction. In the embodiments of this application, "and / or" merely describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0206] The directional terms mentioned in the embodiments of this application, such as "up", "down", "left", "right", "inner", and "outer", are only for reference to the directions in the accompanying drawings. Therefore, the directional terms used are for better and clearer explanation and understanding of the embodiments of this application, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0207] References to "one embodiment," "in some examples," or "some embodiments" as described in the embodiments of this application mean that one or more embodiments of this specification include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in some examples," "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0208] The methods provided in the embodiments of this application above are described from the perspective of an electronic device (e.g., a mobile phone, a tablet computer) as the executing entity. To implement the functions of the methods provided in the embodiments of this application above, the electronic device may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0209] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)). Where there is no conflict, the solutions in the above embodiments can be combined.

[0210] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0211] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0212] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0213] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0214] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope and intent of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and variations.

Claims

1. A screen projection method, characterized in that, Applied to a first device, the first device including a projection assist chip, a main chip, and a display screen, the projection assist chip being connected to the main chip, and the main chip being connected to the display screen, the method includes: The projection assistance chip receives the first display information sent by the second device; The screen projection auxiliary chip sends the first display information to the main chip; The main chip processes the first display information into second display information generated by the first device itself, and displays the second display information through the display screen.

2. The method according to claim 1, characterized in that, The main chip processes the first display information into second display information generated by the first device itself, and displays the second display information through the display screen, including: The main chip processes the first display information into second display information for a first application of the first device. The first application is used to create a first window and display the second display information in the first window. The first application is a local application of the first device.

3. The method according to claim 2, characterized in that, The main chip processes the first display information into second display information for the first application of the first device, including: The main chip generates second display information based on the first display information. The second display information is located in the application package of the first application. The first application is used to display the second display information in the first window based on the application package.

4. The method according to claim 3, characterized in that, The method further includes: After the main chip determines that the screen mirroring of the second device has ended, it deletes the second display information from the application package.

5. The method according to any one of claims 2-4, characterized in that, The first window includes a first artificial intelligence (AI) button, and the method further includes: Receive the operation on the first AI button and determine the target area within the first window; The displayed content within the target area is processed by AI, and the AI ​​processing includes at least one of translation, search, object recognition, copying, and sharing.

6. The method according to claim 5, characterized in that, The target area is a user-specified area.

7. The method according to any one of claims 2-6, characterized in that, The first window includes a second AI button, and the second displayed information is a game screen. The method further includes: Upon receiving an operation on the second AI button, the game guide corresponding to the game screen is displayed.

8. The method according to any one of claims 2-7, characterized in that, The size of the first window matches the display size of the second device, or the size of the first window is the default size.

9. The method according to any one of claims 2-8, characterized in that, The method further includes: The system receives a first operation on the first window and performs at least one of the following operations on the first window: moving, scaling, maximizing, minimizing, or closing.

10. The method according to any one of claims 1-9, characterized in that, Before the projection assist chip receives the first display information sent by the second device, the method further includes: The main chip outputs a first prompt message, which is used to prompt whether to use the first device as a screen projection receiver to receive the screen projection from the second device. After receiving confirmation from the user, the main chip sends a notification message to the second device, which is used to notify the second device to project its screen onto the first device.

11. The method according to any one of claims 1-10, characterized in that, The main chip includes a video interface that supports a first protocol. Before the projection auxiliary chip sends the first display information to the main chip, the method further includes: The projection assistance chip performs protocol conversion on the first display information to convert the protocol of the first display information into the first protocol. The screen projection auxiliary chip sends the first display information, after protocol conversion, to the main chip through the video interface.

12. The method according to claim 11, characterized in that, The video interface is the MIPI (Mobile Industry Processor Interface).

13. The method according to any one of claims 1-12, characterized in that, The first device further includes a control chip and a MUX, the control chip being connected to the MUX, and the MUX being connected to both the projection assistance chip and the main chip. Before the projection assistance chip receives the first display information sent by the second device, the method further includes: If the control chip determines that the device is the screen projection receiver, it switches the MUX to a first state. In the first state, the MUX connects the screen projection auxiliary chip to the second device, so that the screen projection auxiliary chip receives the first display information sent by the second device.

14. The method according to claim 13, characterized in that, The method further includes: If the control chip determines that the device is the screen projection sending end, it switches the MUX to the second state. In the second state, the MUX connects the main chip and the second device, and the main chip is configured to project the screen to the second device.

15. The method according to any one of claims 1-14, characterized in that, The method further includes: The screen projection assist chip receives the first audio information sent by the second device; The screen projection auxiliary chip sends the first audio information to the main chip; The main chip processes the first audio information into second audio information generated by the first device itself, and outputs the second audio information.

16. The method according to claim 15, characterized in that, The main chip processes the first audio information into second audio information generated by the first device itself, and outputs the second audio information, including: The main chip processes the first audio information into second audio information for a first application of the first device. The first application is used to output the second audio information and is a local application of the first device.

17. The method according to claim 16, characterized in that, The main chip processes the first audio information into second audio information for the first application of the first device, including: The main chip generates second audio information based on the first audio information. The second audio information is located in the application package of the first application. The first application is used to output the second audio information based on the application package.

18. The method according to claim 17, characterized in that, The method further includes: After the main chip determines that the screen mirroring of the second device has ended, it deletes the second audio information from the application package.

19. The method according to any one of claims 15-18, characterized in that, The main chip includes an audio interface that supports a second protocol. Before the screen projection auxiliary chip sends the first audio information to the main chip, the method further includes: The projection assist chip performs protocol conversion on the first audio information to convert the protocol of the first audio information into the second protocol. The screen projection auxiliary chip sends the first audio information, after protocol conversion, to the main chip through the audio interface.

20. The method according to claim 19, characterized in that, The audio interface is a built-in audio I2S interface of the integrated circuit.

21. The method according to any one of claims 2-20, characterized in that, The method further includes: The projection assist chip receives third display information sent by a third device; The screen projection auxiliary chip sends the third display information to the main chip; The main chip processes the third display information into the fourth display information of the first application, and the first application creates a second window to display the fourth display information in the second window.

22. A screen projection method, characterized in that, The method is applied to a communication system, which includes a first device and a second device. The first device includes a projection assist chip, a main chip, and a display screen. The projection assist chip is connected to the main chip, and the main chip is connected to the display screen. The second device sends the first display information to the first device; The screen projection auxiliary chip in the first device receives the first display information sent by the second device and sends the first display information to the main chip. The main chip processes the first display information into second display information generated by the first device itself and displays the second display information through the display screen.

23. The method according to claim 22, characterized in that, The main chip processes the first display information into second display information generated by the first device itself, and displays the second display information through the display screen, including: The main chip processes the first display information into second display information for a first application of the first device. The first application is used to create a first window and display the second display information in the first window. The first application is a local application of the first device.

24. The method according to claim 23, characterized in that, The main chip processes the first display information into second display information for the first application of the first device, including: The main chip generates second display information based on the first display information. The second display information is located in the application package of the first application. The first application is used to display the second display information in the first window based on the application package.

25. The method according to claim 23 or 24, characterized in that, The first window includes a first artificial intelligence (AI) button, and the method further includes: Receive the operation on the first AI button and determine the target area within the first window; The displayed content within the target area is processed by AI, and the AI ​​processing includes at least one of translation, search, object recognition, copying, and sharing.

26. The method according to any one of claims 23-25, characterized in that, The method further includes: The system receives a first operation on the first window and performs at least one of the following operations on the first window: moving, scaling, maximizing, minimizing, or closing.

27. An electronic device, characterized in that, The electronic device is used to perform the method as described in any one of claims 1-21.

28. A communication system, characterized in that, include: First equipment and second equipment; The first device is used to perform the steps of the first device in the method as described in any one of claims 22-26; The second device is used to perform the steps of the second device in the method as described in any one of claims 22-26.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 26.

30. A computer program product, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 26.

Images