Display device and multi-window display method

Abstract

Provided in the present application are a display device and a multi-window display method. The display device is configured to: pre-register a monitoring service that is used for monitoring a task stack; when entering a first mode, create a first task stack for a first application; and when the creation of the first task stack is detected by the monitoring service, trigger the modification of a visible area defined by a first window container. That is, the application is bypassed and a window attribute is directly adjusted at the underlying logic level, so as to achieve the rendering of a corresponding window. The display device is not required to pre-configure a system parameter to turn on a switch of a first mode, and the display device is not limited to the support of an application to the first mode.

Description

A display device and a multi-window display method

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese patent applications filed on March 28, 2025, application number 202510388341.9 and March 31, 2025, application number 202510397927.1, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of display device technology, and in particular to a display device and a multi-window display method. Background Technology

[0004] A display device refers to a terminal device capable of outputting a specific display image, such as a smart TV, mobile terminal, smart advertising screen, projector, etc. Display devices can support multi-window mode, which can be enabled by configuring system parameters. Furthermore, only applications that have explicitly declared support for multi-window mode can be used in multi-window mode; that is, only applications that inherently support multi-window mode can be used. If the display device has not pre-configured system parameters to enable multi-window mode, users will not be able to use it. Additionally, if the application itself does not support multi-window mode, the display device will also be unable to use these applications in multi-window mode. Summary of the Invention

[0005] According to some embodiments of this application, a display device is provided, including: a display configured to: display a first window and display a first page within the first window, the first window being in full-screen mode; a memory configured to store a computer program; and at least one processor connected to the display and the memory, configured to execute the computer program to cause the display device to:, in response to a user instruction to launch a first application, display a second window on top of the first window and display a second page of the first application within the second window, wherein the second window is in full-screen mode; and, in response to a user instruction to enter a multi-window mode, if the multi-window mode is a first mode, create a first task stack corresponding to the first application, and through... A pre-registered listening service monitors a first task stack, which includes application pages associated with a first application, and each application page includes a second page. When the listening service detects that a first task stack has been created, it generates window information for adjusting the second window. Based on the window information, it adjusts the visible area defined by the first window container associated with the first task stack from the entire visible area of ​​the screen to a first visible area. The first window container includes the application window corresponding to each application page, and each application window includes a second window. The first window container is used to define the visible area of ​​the application window on the screen. The display is controlled to show the second window on top of the first window, which is in full-screen mode, according to the first visible area.

[0006] According to some embodiments of this application, a multi-window display method is provided, applied to a display device. The display device displays a first window and a first page within the first window, the first window being in full-screen mode. The method includes: responding to a user instruction to launch a first application, displaying a second window on top of the first window and displaying a second page of the first application within the second window, wherein the second window is in full-screen mode; responding to a user instruction to enter a multi-window mode, if the multi-window mode is a first mode, creating a first task stack corresponding to the first application, and monitoring the first task stack through a pre-registered monitoring service, wherein the first task stack includes application pages associated with the first application, and the application pages include a second page; when the monitoring service detects that the first task stack has been created, generating window information for adjusting the second window; according to the window information, adjusting the visible area defined by the first window container associated with the first task stack from the entire visible area of ​​the screen to a first visible area, the first window container including application windows corresponding to each application page, the application windows including a second window, wherein the first window container is used to define the visible area of ​​the application windows on the screen; and controlling the display to display the second window on top of the first window in full-screen mode, according to the first visible area. Attached Figure Description

[0007] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device according to some embodiments of this application;

[0008] Figure 2 is a hardware configuration block diagram of a display device provided according to some embodiments of this application;

[0009] Figure 3 is an operating system configuration diagram of a display device provided according to some embodiments of this application;

[0010] Figure 4 is a flowchart of a display device according to some embodiments of the present application displaying multiple applications in a first mode;

[0011] Figure 5 is an interactive schematic diagram of a display device according to some embodiments of the present application displaying multiple applications in a first mode;

[0012] Figure 6 is a flowchart of a display device launching a first application according to some embodiments of this application;

[0013] Figure 7 is a timing diagram of a display device according to some embodiments of the present application displaying multiple applications in a first mode;

[0014] Figure 8 is a timing diagram of a display device according to some embodiments of the present application displaying multiple applications in a second mode;

[0015] Figure 9 is an interactive schematic diagram of a display device according to some embodiments of the present application displaying multiple applications in a second mode;

[0016] Figure 10 is a schematic diagram of an interface displaying multiple applications in a third mode according to some embodiments of this application;

[0017] Figure 11 is a schematic diagram of the application runtime architecture provided according to some embodiments of this application;

[0018] Figure 12 is a flowchart of a display device launching a first application according to some embodiments of this application;

[0019] Figure 13 is a timing diagram of a display device applying a multi-window mode according to some embodiments of this application;

[0020] Figure 14 is an interactive schematic diagram of a display device applying a multi-window mode according to some embodiments of this application;

[0021] Figure 15 is a flowchart of a display device entering multi-window mode according to some embodiments of this application. Detailed Implementation

[0022] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims. It should be noted that the brief descriptions of terminology in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0023] In this application embodiment, "display device" refers to a device with screen display and data processing capabilities. For example, display devices include, but are not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, and augmented reality devices.

[0024] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device according to some embodiments of this application. As shown in Figure 1, a user can operate the display device 200 via touch operation, voice, mobile terminal 300, and control device 100. For example, the control device 100 can be a remote control, stylus, gamepad, etc. The display device 200 also communicates with the server 400 via various communication methods, such as a local area network (LAN), wireless local area network (WLAN), and other networks. The display device 200 can provide broadcast television reception functionality and may also be additionally equipped with intelligent network television functionality that provides computer support, including but not limited to network television, smart television, and Interactive Personality TV (IPTV).

[0025] Figure 2 is a hardware configuration block diagram of a display device according to some embodiments of this application. In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, at least one processor 250, a display 260, an audio output device 270, a user input interface 280, a memory, and a power supply.

[0026] Communication device 220 is a component used to communicate with external devices or server 400 according to various communication protocol types. Display device 200 is equipped with multiple communication devices 220 depending on the supported communication methods. Communication devices 220 enable display device 200 to communicate with external devices or server 400 via wireless or wired connections. Detector 230 is used to collect signals from the external environment or to interact with external systems. Device interface 240 is used to connect to external devices. At least one processor 250 is used to control the overall operation of display device 200. The processor includes at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), and a power processor, and provides input / output interfaces from the first to the nth interface. At least one processor 250 controls the operation of display device 200 and responds to user operations through various software control programs / computer programs stored in memory. At least one processor 250 and tuner 210 are located in separate devices; that is, tuner 210 is located in an external device of the main device containing at least one processor 250, such as an external set-top box. Display 260 is used to receive and display image signals output from at least one processor 250. Display 260 includes display function components for presenting the image and driving components for driving the image display. Users can input user commands through a graphical user interface (GUI) displayed on display 260, and user input interface 280 can receive user commands through the GUI. Audio output device 270 can be a built-in speaker of display device 200 or an external audio output device connected to display device 200. User input interface 280 is used to receive instructions input by the user.

[0027] To enable user interaction, in some embodiments, the display device 200 may run an operating system. An operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200. The operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized for a specific operating platform, or a standalone operating system specifically developed for the display device 200.

[0028] Operating systems can be divided into different modules or layers based on their implemented functions. For example, as shown in Figure 3A, in some embodiments, the system can be divided into four layers, from top to bottom: the Applications layer (hereinafter referred to as the "Application Layer"), the Application Framework layer (hereinafter referred to as the "Framework Layer"), the System Runtime Library layer, and the Kernel layer. The Application Layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with users based on the applications. For example, the Application Layer may include a Voice Assistant, which provides voice interaction functionality, allowing users to interact with the display device via voice. The Framework Layer provides Application Programming Interfaces (APIs) and programming frameworks for applications. The Application Framework Layer includes some predefined functions. The Application Framework Layer acts as a processing center, determining the actions taken by applications in the Application Layer. Applications can access system resources and obtain system services during execution through the API interface. In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction libraries contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer. In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. It should be noted that the above examples are merely simple divisions of operating system functions and do not limit the specific form of the operating system of the display device 200 in this embodiment. Depending on the functions of the display device, the type of operating system, and other factors, the number and specific types of layers contained in the operating system can take other forms.

[0029] The display device 200 can operate in a multi-window mode. In this mode, the device can display multiple windows simultaneously, with each window displaying a corresponding application, thus achieving the effect of displaying multiple applications at the same time. In some embodiments, the multi-window mode may include a first mode, which may also be referred to as a small-window mode, a picture-in-picture (PIP) mode, etc. In the first mode, the device can display different windows on upper and lower layers, respectively. The upper-layer window is in a non-full-screen state (i.e., a small-window state), while the lower-layer window is in full-screen state. The upper and lower layers correspond to window hierarchies. In one example, the window hierarchy of the upper-layer window is higher than that of the lower-layer window; that is, the window with the higher hierarchy is displayed over the window with the lower hierarchy.

[0030] In some embodiments, the multi-window mode may include a second mode, which may also be called a split-screen mode, a freeform mode, etc. In the second mode, the display device can display two windows on the same layer, where the two windows correspond to the same window level and there is no overlap between the two windows. As shown in Figure 3B of the operating system, the application layer of the operating system is configured with a multi-window application (Multiview APP), and the display device can implement the multi-window mode through the multi-window application. The application layer of the operating system is also configured with a second application, which can be a native application, a third-party application, etc., such as a live TV application (LiveTv APP). The application of the operating system can also be configured with a screen casting application, such as a content sharing application (Content Sharing APP). The content sharing application is a long-lived service based on the Content Sharing protocol and can listen for push requests of the content sharing protocol.

[0031] Whether a display device can support multi-window mode is subject to several limitations. Firstly, the display device itself is limited. It needs to configure system parameters to enable the native multi-window mode switch to support it. For example, a display device can only support the first mode (such as PiP mode) if the system feature of the first mode is enabled; it can only support the second mode (such as split-screen mode) if the multi-window configuration is set to true; and it can only support the second mode (such as free-form window mode) if the developer options enable free-form window mode. Secondly, the display device is limited by the application. Only applications that explicitly support multi-window mode can be used in multi-window mode on the display device. For example, the application only supports multi-window mode if the (resizeableActivity) corresponding to the application's associated activity is set to true. Furthermore, the application must declare support for multi-window mode in its manifest file (e.g., supportsPicture-In-Picturemode) for the display device to use the application in multi-window mode. If the display device is not pre-configured with a system parameter to enable multi-window mode, users will not be able to use multi-window mode. Furthermore, if the application itself does not support multi-window mode, the display device will also be unable to use these applications in multi-window mode.

[0032] Accordingly, some embodiments of this application provide a display device and a multi-window display method. The method is applied to a display device, which includes: a display configured to display a first window and a first page within the first window, the first window being in full-screen mode; a memory configured to store a computer program; and at least one processor connected to the display and the memory, configured to execute the computer program to cause the display device to perform subsequent processes, so that the display device can support a multi-window mode, regardless of whether it has a native switch for enabling multi-window mode or whether the application itself supports multi-window mode.

[0033] In some embodiments, as shown in Figure 3A, the operating system includes a controller application at its application layer. The display device 200 can then implement an unrestricted multi-window mode based on this controller application. The controller application can be a native application of the display device, starting up along with the display device 200 upon boot. The operating system's framework layer includes system services (SystemServer) and a system interface (SystemUI). The system services include Activity Manager (AM) and Windows Manager (WM) services; the system interface includes the Windows Manager Shell (wmshell) functionality. The system services and system interface provide corresponding interfaces for the controller application, supporting its implementation of an unrestricted multi-window mode.

[0034] In this embodiment, the display device displays a first window before entering multi-window mode, and displays a first page within the first window. The first window is in full-screen mode; that is, the display device displays the first page in full-screen mode. In some embodiments, the first page can be the display device's system launcher, the application page of the display device's native application, or the application page of a third-party application. The following embodiments will specifically illustrate the process of the display device displaying multiple applications in multi-window mode.

[0035] Example 1, Figure 4 is a flowchart of a display device according to some embodiments of this application displaying multiple applications in a first mode. As shown in Figure 4, the specific steps are as follows:

[0036] S401, in response to a user instruction to launch a first application, a second window is displayed above the first window, and a second page of the first application is displayed within the second window.

[0037] The first application is the application the user wants to open. This can be a native application of the display device or any third-party application. The second window displays the content of a second page, which is the application page associated with the first application. The display device configures the second window's window hierarchy to be higher than the first window's. Therefore, the display device can display the second window on top of the first window, allowing the user to view the first application's page. When the display device launches an application in normal mode (i.e., single-window mode), it defaults to displaying the single window in full-screen mode. Thus, the second window is in full-screen mode, meaning the first application is displayed in full-screen mode. Full-screen mode indicates that the entire visible area of ​​the screen can be used to display the window.

[0038] Figure 5 is an interactive diagram illustrating the display device displaying multiple applications in the first mode. As shown in Figure 5A, the display device displays the first page within the first window 51, which is in full-screen mode. Responding to a user's instruction to launch the first application, as shown in Figure 5B, the display device displays a second window 52 on top of the first window 51, and displays the second page of the first application within the second window 52, ​​which is also in full-screen mode. At this time, the second window 52 covers the first window 51, so the user can only see the second page and cannot see the first page.

[0039] Figure 6 is a flowchart illustrating the startup of a first application using a display device according to some embodiments of this application. As shown in Figure 6, the specific steps are as follows:

[0040] Step S601: In response to the user's instruction to start the first application, create the first task stack corresponding to the first application, and listen to the first task stack corresponding to the first application through the listening service.

[0041] When a display device launches an application, it creates a corresponding task stack for that application. This task stack includes the application's associated activities and manages these activities to ensure the application runs correctly. Based on this, in response to the command to launch a first application, the display device creates a corresponding first task stack for that application and manages the application pages associated with it. The application pages associated with the first application include the second page that will be displayed in the second window.

[0042] In some embodiments, the display device registers a listening service upon startup. This listening service monitors the status of the task stack. The task stack status includes an "Appear" status and a "Vanished" status. If the listening service detects an "Appear" task stack, it can determine that the corresponding application has started; if the listening service detects a "Vanished" task stack, it can determine that the corresponding application has exited. In one example, the ShellTaskOrganizer service registers a listening service (ItaskOrganizer) with the TaskOrganizerController service in the system services.

[0043] Step S602: When the listening service detects that the first task stack has been created, create the first window container associated with the first task stack.

[0044] The display device detects the creation of the first task stack through a listening service, indicating that the first application has been launched and its application page can be displayed. When displaying the second window corresponding to the first application, the visible area of ​​the second window on the screen and its window hierarchy need to be defined. This allows a second window of a specified size to be displayed at a specified location on the screen, and enables the user to see / hide the second window. A window container corresponds to an application and is used to contain application windows associated with that application. Each application window corresponds to an application page associated with that application. The window container defines the window attributes of each application window it contains, including the visible area of ​​the window on the screen and its window hierarchy. In other words, the area of ​​the application window displayed on the screen within the window container cannot exceed the visible area defined by the window container, and the window hierarchy of the application window within the window container must be consistent with the window hierarchy defined by the window container. Therefore, the window container can define the size, position, and window hierarchy of application windows from a low-level logical perspective.

[0045] Based on this, after the first application is launched, a first window container associated with the first task stack is created, so as to define the window properties of each application window associated with the first application through the first window container and constrain the display effect of each application window.

[0046] The first window container associated with the first task stack refers to the container used to hold the application windows associated with each application page in the first task stack. Since the application pages include second pages, the first window container also holds the second windows associated with the second pages. The containment relationship between the window container and each application window can be achieved through mounting. By mounting each application window under the first window container, a hierarchical relationship is formed between the window container and the application windows. The mounted entity (window container) is the superior, and the mounting entity (application window) is the inferior. The inferior is constrained by the superior; for example, the window attributes of an application window cannot exceed the window attributes defined by the window container.

[0047] A surface control is used to manage the container parameters of a given window container. Each window container corresponds to one surface control, and different window containers correspond to different surface controls. The display device can modify the container parameters of a given window container through the surface control. In some embodiments, container parameters may include the visible area, window hierarchy, etc., defined by the window container.

[0048] Accordingly, after creating the window container corresponding to the task stack, the display device generates a surface controller for that window container and further establishes a mapping relationship between the task stack and the surface controller. In this way, when displaying an application, the surface controller with the mapping relationship to the task stack corresponding to that application can be quickly obtained. Then, the window attributes defined by the corresponding window container can be managed through this surface controller. Since the window container contains the application windows associated with that application, the effect of adjusting the window attributes of each application window can be achieved.

[0049] The display device can manage the mapping relationships between task stacks and surface controllers of each launched application in the form of a mapping relationship list. That is, the mapping relationship list includes the mapping relationships between task stacks and surface controllers. Specifically, when the display device detects the creation of a task stack through a listening service, it creates a mapping relationship between that task stack and the corresponding surface controller and writes this mapping relationship into the mapping relationship list. When the display device detects the unregistering of a task stack through the listening service, it removes the mapping relationship corresponding to that task stack from the mapping relationship list.

[0050] In some embodiments, the display device stores a task stack list, which includes the package name and task ID of the application that has created the task stack. The task ID is used to uniquely identify the corresponding task stack. After the display device detects a created task stack through a listening service, it obtains the package name of the corresponding application and generates a task ID to identify the task stack. The package name and task ID of the application corresponding to the task stack are then written into the task stack list. After the display device detects a deregistered task stack through the listening service, it removes the package name and task ID of the application corresponding to the task stack from the task stack list.

[0051] Step S603: Set the visible area defined by the first window container to the entire visible area of ​​the screen, and set the window layer defined by the first window container to the top level.

[0052] The visible area defined by the first window container is set to the entire visible area of ​​the screen. Therefore, the maximum visible area of ​​each application window contained within the first container window is the entire visible area of ​​the screen; that is, the application window containing the first application can be displayed in full-screen mode. The window hierarchy defined by the first window container is set to the top level. Therefore, the window hierarchy of each application window contained within the first container window is the top level; that is, the application window containing the first application is displayed at the top level.

[0053] Step S604: Determine the second page to be displayed in the first task stack.

[0054] In one example, the second page could be the application page that is displayed by default when the first application is launched. In another example, the second page could be the application page that was displayed when the first application was last exited.

[0055] Step S605: Display the second window on the top layer according to the entire visible area of ​​the screen, and display the second page within the second window.

[0056] After determining the second page to be displayed, the second window associated with the second page is displayed according to the window properties (the entire visible area of ​​the screen and the top layer) defined by the first window container, and the second page is displayed within the second window. Thus, when the display device launches the first application, it can display the second window in full screen by setting the visible area defined by the first container window to the entire visible area of ​​the screen, and it can also set the window layer defined by the first container window to the top layer, so that the second window is displayed on top of the first window.

[0057] Step S402: In response to the user's instruction to enter multi-window mode, if the multi-window mode is the first mode, create the first task stack corresponding to the first application, and listen to the first task stack through the pre-registered listening service.

[0058] In some embodiments, the control application may provide an entry point to enter multi-window mode, through which the user can input instructions to enter multi-window mode. For example, this entry point may be configured in a shortcut menu, system settings menu, sidebar, etc.

[0059] In some embodiments, the control device may be equipped with a designated key, the key value of which is mapped to entering multi-window mode. Users can input instructions to enter multi-window mode based on this designated key.

[0060] In some embodiments, the display device may display selection information in response to a user instruction to enter a multi-window mode, allowing the user to indicate the desired multi-window mode (first mode or second mode).

[0061] In Example 1, the user's instruction to enter the first mode is used as an example for specific explanation. In this embodiment, the display device determines the application currently displayed on the top layer as the application displayed in a small window. That is, it is necessary to adjust the window where the application currently displayed on the top layer is located; in this embodiment, it is also necessary to adjust the second window. When the display device enters the multi-window mode, it needs to recreate the first task stack corresponding to the first application to trigger the listening service to detect the appearance of the first task stack, thereby triggering the onTaskAppeared callback to obtain the relevant objects and information used to implement the first mode. In one example, the display device can recreate the first task stack when restarting the first application. The process of the display device listening to the first task stack through the pre-registered listening service can be referred to step S601, which will not be repeated here.

[0062] Figure 7 is a timing diagram of a display device according to some embodiments of this application displaying multiple applications in a first mode. Referring to Figure 7, when the display device boots up, it starts the Controller APP and the System UI (S701). After the System UI starts, it registers a listening service with the system service through the System UI (S702).

[0063] Step S403: When the listening service detects that the first task stack has been created, window information for adjusting the second window is generated.

[0064] Adjusting the second window involves reducing its size and adjusting its position on the screen. In the first mode, the smaller window needs to be displayed on top, therefore, there is no need to adjust the window hierarchy of the second window. The specific steps of the process for determining the window information for adjusting the second window according to some embodiments of this application are as follows:

[0065] Step S801: When the listening service detects that the first task stack has been created, obtain the task stack list.

[0066] The process of displaying the device management task stack list can be referred to in step S602, and will not be elaborated here. It is understood that the obtained task stack list includes the package name of the first application and the task stack identifier corresponding to the first task stack.

[0067] As shown in Figure 7, the display device, in response to a user instruction to enter the first mode (S703), launches the first application (S704) and creates the first task stack corresponding to the first application. After the display device 200 detects the creation of the first task stack through the listening service in the system service (S705), it can trigger the onTaskAppeared callback. The system service writes the package name of the first application and the task stack identifier corresponding to the first task stack into the task stack list to update the task stack list (S706), and sends the updated task stack list back to the system interface through a specified interface of the AM service (such as the getRunningTaskInfo interface) (S707). The system interface then passes the updated task stack list to the control application (S708).

[0068] Step S802: Determine the task stack identifier corresponding to the first task stack from the task stack list based on the package name of the first application.

[0069] The task stack identifier corresponding to the first task stack is determined so that the first task stack can be identified based on the task stack identifier, and the first window container and the first surface controller associated with the first task stack can be further identified. As shown in Figure 7, in response to a user instruction to enter the first mode, the control application can identify the first application entering the first mode and further determine the package name of the first application. After obtaining the task stack list, the task stack identifier corresponding to the first task stack can be determined based on the package name of the first application.

[0070] Step S803: Generate window information based on the task stack identifier corresponding to the first task stack and the default window attributes.

[0071] Upon entering the first mode, the display device can generate window information according to the window attributes configured for the first mode (default window attributes). In the example, default window attributes may include the default window size, default window ratio, default position on the screen, and the window being at the top level. Based on the default window attributes and the area information of the entire visible area of ​​the screen, adjustment information for the second window can be generated. This includes, for example, the window's scaling ratio, scaling direction, movement direction, movement distance, adjusted window size, and adjusted position on the screen. In the example, window information may include: the task stack identifier corresponding to the first task stack, the window's horizontal scaling ratio, the window's vertical scaling ratio, and the window's horizontal and vertical offsets (x, y).

[0072] As shown in Figure 7, after the control application generates window information (S709), it transmits the window information (corresponding to the command to start the first mode) to the system interface (S713). In some embodiments, the control application can encapsulate the task stack identifier and adjustment information corresponding to the first task stack into a Parcel object and transmit it to the system interface together. The control application can also transmit the task stack identifier and adjustment information corresponding to the first task stack to the system interface through IPC communication mechanisms such as Messenger, broadcast, or AIDL. Thus, after the display device detects that the first task stack has been recreated, it can generate corresponding window information for the first task stack in a targeted manner, and then adjust the second window in a targeted manner.

[0073] Step S404: Based on the window information, adjust the visible area defined by the first window container associated with the first task stack from the entire visible area of ​​the screen to the first visible area.

[0074] Since the visible area defined by the first window container is used to limit the visible area of ​​each application window associated with the first application on the screen, adjusting the visible area defined by the first window container can achieve the effect of adjusting the size and position of each application window on the screen. In some embodiments, the first visible area is a portion of the visible area of ​​the entire visible area of ​​the screen. The specific steps of adjusting the first window container according to some embodiments of this application are as follows:

[0075] Step S901: Obtain the pre-stored list of mapping relationships.

[0076] The process of the display device managing the mapping relationship list through the system interface can be referred to in step S603, which will not be elaborated here.

[0077] As shown in Figure 7, after the display device detects the creation of the first task stack (S710) through the listening service in the system service, it can trigger the onTaskAppeared callback. After the system service creates the first window container and generates the first surface controller corresponding to the first window container (S710), it calls back the first task stack and the first surface controller to the system interface (S711). The system interface creates a mapping relationship between the first task stack and the first surface controller and writes the mapping relationship into the mapping relationship list. This updated mapping relationship list is the pre-stored mapping relationship list to be obtained (S712).

[0078] Step S902: Obtain the first surface controller corresponding to the first task stack according to the mapping relationship list.

[0079] In one example, based on step S803, the display device can determine the first task stack based on the task stack identifier corresponding to the first task stack included in the window information. Then, it can determine the mapping relationship corresponding to the first task stack from the mapping relationship list, and obtain the first surface controller corresponding to the first task stack based on this mapping relationship. As shown in Figure 7, the system interface determines the first task stack based on the task stack identifier corresponding to the first task stack in the window information, and further obtains the first surface controller corresponding to the first task stack based on the mapping relationship list (S714).

[0080] Therefore, the display device manages the mapping relationship between the task stack and the surface controller corresponding to each launched application through a mapping relationship list. After determining the window information used to adjust the second window, the display device can determine the first surface controller used to manage the first window container based on this mapping relationship list, thereby adjusting the visible area defined by the first window container through the first surface controller.

[0081] Step S903: Based on the window information, adjust the visible area defined by the first window container from the entire visible area of ​​the screen to the first visible area through the first surface controller.

[0082] The display device can shrink the visible area defined by the first window container from the entire visible area of ​​the screen to the first visible area based on the size information in the window information. Furthermore, based on the position information in the window information, the position of the first visible area is defined at a first position on the screen. Therefore, by shrinking the visible area defined by the first window container and modifying the position of the visible area defined by the first window container on the screen, the display device can correspondingly shrink the second window within the first window container and display the second window at the adjusted position.

[0083] Step S405: Display the second window above the first window in full-screen mode, according to the first visible area.

[0084] In some embodiments, the display device may construct an image composition object (such as SurfaceControl.Transaction) or a window container composition object (such as WindowContainer.Transaction) corresponding to the first visible area based on the first surface controller to control and adjust the window properties of the second window.

[0085] As shown in Figure 7, the system interface constructs an image composition object (based on window information) (S715), and adjusts the second window using the image composition object constructed by the application, entering the first mode and displaying the adjusted second window (S716). As shown in C of Figure 5, in response to the command to enter the first mode, the display device can shrink the second window 52 and display the shrunken second window 52 in the upper right corner of the screen. Therefore, the display device can, by pre-registering a listening service for listening to the task stack, create the first task stack of the first application when entering the first mode. When the listening service detects the creation of the first task stack, it can trigger the modification of the visible area defined by the first window container, effectively bypassing the application and directly adjusting window attributes from the underlying logic to achieve rendering of the corresponding window. This eliminates the need for the display device to pre-configure system parameters to enable or disable the first mode and is not limited by the application's support for the first mode.

[0086] The display device can respond to the user's adjustment command for the second window, adjusting the position of the second window on the screen and / or adjusting the size / ratio of the second window. Specifically, the display device modifies the container parameters of the first window container to change the visible area defined by the first window container, thereby achieving the effect of adjusting the second window. This process can be referred to steps S404-S405 and S901-S903, and will not be elaborated here.

[0087] The display device can exit the first mode when the listening service detects that the first task stack has been deregistered. As shown in Figure 7, the display device detects the deregistration of the first task stack through the listening service in the system service (S717), triggers onTaskVanished, and reports the status of the first task stack as deregistered to the system interface (S718). Based on this result, the system interface removes the mapping relationship between the first task stack and the first surface controller from the mapping relationship list (S719) and sends a notification to the control application that it has exited the first application. The control application exits the first mode according to the notification (S720). After exiting the first mode, the display device can restore the second window to full-screen mode. Alternatively, it can close the second window and display the first window in full-screen mode.

[0088] Example 2: The specific steps of the process for displaying multiple applications in a second mode using a display device according to some embodiments of this application are as follows:

[0089] Step S1001: In response to the user's instruction to enter multi-window mode, if the multi-window mode is the second mode, obtain the total number N of the split-screen areas indicated by the user, and the N split-screen applications indicated by the user to use in the second mode, where N is an integer greater than or equal to 2.

[0090] The user's instruction to enter the second mode includes the total number N of split-screen areas. For example, if the user instructs to enter the second mode and further instructs to enter a three-screen mode, the total number N of split-screen areas is 3. The user's instruction to enter the second mode includes the N split-screen applications used in the second mode. These N split-screen applications may or may not include the second application belonging to the first page. The total number of split-screen applications is equal to the total number of split-screen areas. In some embodiments, the user can specify the correspondence between the N split-screen applications and the N split-screen areas in the instruction to enter the second mode.

[0091] Step S1002: Create the task stack corresponding to the split-screen application and call the multi-window interface to create N root task containers by calling the task management controller through the multi-window interface.

[0092] The process of creating a task stack corresponding to a split-screen application on the display device can be referred to the process of creating the first task stack in Example 1, and will not be repeated here. The task stack corresponding to the split-screen application includes the application page associated with the split-screen application. Each root task container corresponds to a split-screen region, and different root task containers correspond to different split-screen regions. The root task container is used to define the visible area of ​​the windows contained within it on the screen, where the visible area defined by the root task container corresponds to the corresponding split-screen region. In other words, the root task container is used to constrain the visible area of ​​the application windows it contains on the screen, so that the application windows it contains match the corresponding split-screen region.

[0093] In this embodiment, the display device adds a multi-window configuration interface and defines this interface for calling the Task Organizer Controller. The Task Organizer Controller is used to create the root task container. Therefore, the display device can directly call the Task Organizer Controller to create the root task container by calling the multi-window interface. The specific steps of the display device creating the root task container according to some embodiments of this application are as follows:

[0094] Step S1101: Calculate the region information of the visible area of ​​each split-screen region on the screen based on the total number N of split-screen regions. In some embodiments, the region information may include the position information of the split-screen region on the screen, the size of the split-screen region, etc.

[0095] In one example, the display device can calculate the visible area information of each split-screen region on the screen by dividing the entire visible area of ​​the screen equally, based on the total number of split-screen regions N. In another example, the display device can calculate the visible area information of each split-screen region on the screen according to a split-screen strategy corresponding to the total number of split-screen regions N. For example, the split-screen strategy for a four-screen display is to display in two rows, with the upper row forming three split-screen regions in an equal division manner, and the lower row corresponding to the remaining split-screen region. In yet another example, the display device can determine the corresponding area information based on user-defined split-screen regions. In some embodiments, after determining the area information corresponding to each split-screen region, the display device can add an identifier to the area information to represent the corresponding split-screen region.

[0096] Step S1102: Call the multi-window interface to call the task management controller to create the corresponding root task container according to the region information.

[0097] Specifically, the size of the visible area defined by the corresponding root task container is set according to the size information in the region information, and the position of the visible area defined by the corresponding root task container on the screen is set according to the position information in the region information. In some embodiments, the task management controller can create the root task container through a task container constructor (such as Task.Builder). Furthermore, the size and position of the visible area defined by the root task container can be set according to the region information by calling the resize method.

[0098] Figure 8 is a timing diagram of a display device according to some embodiments of this application displaying multiple applications in a second mode. As shown in Figure 8, the display device, in response to a user instruction to enter the second mode (S1201), controls the application to launch N split-screen applications (S1202) and sends an instruction to the system interface to create root task containers (S1203). In response to this instruction, the system interface calculates the area information of each split-screen area based on the total number N of split-screen areas and adds corresponding markers (S1204). The system interface adds a multi-window interface, through which the task management controller in the system service can be called, allowing the task management controller to construct N root task containers through the task container constructor (not shown in Figure 8) (S1205). After the task container constructor constructs N root task containers, the task management controller calls the resize method to set the visible area defined by each root task container according to the area information, thus completing the creation of N root task containers, i.e., constructing N root task containers based on the area information (S1206). After creating N root task containers, the system service sends a notification to the system interface (S1207), and the system interface then transmits the notification to the control application (S1208). Thus, the display device, through the task management controller, creates root task containers according to the area information of the corresponding split-screen region, effectively ensuring that the application windows contained in the root task containers are displayed within the corresponding split-screen region, thereby guaranteeing the split-screen effect between multiple split-screen applications.

[0099] Step S1003: Mount the application windows associated with each application page in the task stack to the corresponding root task container.

[0100] By adjusting the mounting relationship between the application window and its parent, the application window associated with the split-screen application is mounted to the corresponding root task container. After adjusting the mounting relationship, the visible area of ​​the split-screen application window on the screen will be adjusted to be constrained by the visible area defined by the root task container. That is, the application window of the split-screen application will match the corresponding split-screen area. In Figure 8, after receiving the notification that the root task container has been created, the control application obtains the task stack identifier corresponding to each split-screen application (refer to the process of obtaining the task stack identifier in Embodiment 1, which will not be repeated here) (S1209), and sends an instruction to enter the second mode to the system interface (S1210). The system interface calls the AM interface of the system service (S1211) to mount the application window associated with the task stack corresponding to the task stack identifier (i.e., the application window associated with the split-screen application) to the corresponding root task container (S1212).

[0101] Step S1004: Control the display to adjust the application window corresponding to the split-screen application according to the visible area defined by the root task container, and display the adjusted application window in the corresponding split-screen area, and display the application page associated with the split-screen application in the adjusted application window.

[0102] Based on the visible area defined by the root task container, the visible area of ​​the contained application windows on the screen can be effectively constrained, thereby achieving the effect of split-screen.

[0103] Figure 9 is an interactive schematic diagram of a display device according to some embodiments of this application displaying multiple applications in a second mode. As shown in Figure 9A, the display device displays a first window 131 in full screen and displays a first page within the first window 131. The display device responds to user instructions to enter the second mode, such as an instruction to enter a four-screen mode, and indicates four split-screen applications, including the second application to which the first page belongs, as well as application A, application B, and application C. Based on the total number of split-screen areas N being 4, and taking the display device dividing the entire visible area of ​​the screen equally as an example, the display device calculates the area information corresponding to each split-screen area, and creates four root task containers (such as root task container a, root task container b, root task container c, and root task container d) by calling the task management controller through the multi-window interface, and sets the visible area defined by each root task container based on the calculated area information. After creating the four root task containers, the application windows associated with the second application, application A, application B, and application C are respectively mounted under the corresponding root task containers. Taking the example of mounting the application window associated with the second application to the root task container a, the application window associated with application A to the root task container b, the application window associated with application B to the root task container c, and the application window associated with application C to the root task container d, if the visible area defined by root task container a is the upper left corner of the screen, the visible area defined by root task container b is the upper right corner of the screen, the visible area defined by root task container c is the lower left corner of the screen, and the visible area defined by root task container d is the lower right corner of the screen, as shown in B of Figure 9, the second application is displayed in the upper left corner of the screen, application A is displayed in the upper right corner of the screen, application B is displayed in the lower left corner of the screen, and application C is displayed in the lower right corner of the screen.

[0104] Therefore, when the display device enters the second mode, it can directly call the task management controller used to create root task containers through the multi-window interface. This creates root task containers corresponding to multiple split-screen areas. Then, by mounting the application windows of multiple split-screen applications to different root task containers, and utilizing the root task containers' restriction on the visible area of ​​their internal windows on the screen, the effect of displaying multiple split-screen application windows on the screen can be achieved. It is evident that there is no need for the display device to pre-configure system parameters to enable the second mode, and it is not limited by the application's support for the second mode.

[0105] Example 3, based on Example 1, allows users to split the first window located at the lower layer into a third mode, where multiple split-screen applications are displayed at the lower layer, and a second window in a non-full-screen state is displayed at the upper layer. The specific steps of the process for displaying multiple applications in the third mode using a display device according to some embodiments of this application are as follows:

[0106] Step S1401: In response to the user's instruction to enter the multi-window mode, if the multi-window mode is the third mode, obtain the total number M of the split-screen areas indicated by the user, and the M split-screen applications indicated by the user to be used in the third mode, where M is an integer greater than or equal to 2.

[0107] The display device receives focus from the first window in response to the user's control command. With focus on the first window, the display device can enter the third mode in response to the user's instruction to enter the third mode. The instruction to enter the third mode is similar to the instruction to enter the second mode. Step S1401 can be referred to step S1001, and will not be repeated here.

[0108] Step S1402: Create the task stack corresponding to the split-screen application and call the multi-window interface to create M root task containers by calling the task management controller through the multi-window interface.

[0109] The task stack corresponding to the split-screen application includes the application page associated with the split-screen application. Each root task container corresponds to a split-screen area. The root task container is used to define the window hierarchy of the windows contained within it, as well as their visible area on the screen. The visible area defined by the root task container corresponds to the corresponding split-screen area, and the window hierarchy is located below the second window. Step S1402 can be referred to as step S1002, and will not be repeated here. The main difference is that in embodiment 3, the root task container needs to define a window hierarchy lower than the second window to ensure that the second window can maintain the first mode.

[0110] Step S1403: Mount the application windows associated with each application page in the task stack to the corresponding root task container.

[0111] Step S1404: Adjust the application window corresponding to the split-screen application according to the visible area defined by the root task container, and display the adjusted application window in the corresponding split-screen area, and display the application page associated with the split-screen application in the adjusted application window.

[0112] The adjusted application windows are displayed below the second window. Steps S1403 to S1404 can be referred to steps S1003 to S1004, and will not be repeated here.

[0113] Figure 10 is a schematic diagram of an interface displaying multiple applications in a third mode according to some embodiments of this application. Taking a user-indicated split-screen mode, with split-screen applications including application A and application B as an example, the display device creates application window 151 and application window 152 below the second window 52. Application A is displayed in application window 151, and application B is displayed in application window 152. Thus, in the first mode, the display device can continue to perform split-screen operation below the second window to enter the third mode. This split-screen operation does not require the display device to pre-configure system parameters to enable the split-screen mode switch, and is not limited by the application's support for the split-screen mode.

[0114] Example 4: Based on Example 1, the display device can be configured with a multi-window mode switching function to achieve switching between a first mode and a second mode. The specific steps of the display device switching multi-window mode according to some embodiments of this application are as follows:

[0115] Step S1601: In response to the user's input mode switching command, the multi-window interface is called to create two root task containers through the task management controller, each root task container corresponding to a split-screen area.

[0116] The root task container is used to define the visible area of ​​the window contained in the root task container on the screen, wherein the visible area defined by the root task container corresponds to the corresponding split screen area; step S1601 can refer to step S1002, and will not be repeated here.

[0117] In some embodiments, in response to a mode switching command, the display device first identifies whether the first page is a specified page, such as a Launcher page. If it is the specified page, the display device terminates the mode switching process.

[0118] Step S1602: Obtain the first task stack corresponding to the first application and the second task stack to which the first page belongs.

[0119] The first task stack includes application pages associated with the first application, and the second task stack includes application pages associated with the application to which the first page belongs. If both the first application and the second application to which the first page belongs are already running, the task stacks corresponding to both applications can be directly retrieved.

[0120] Step S1603: Mount the application windows associated with the application pages in the first task stack and the application windows associated with the application pages in the second task stack to the corresponding root task containers.

[0121] Step S1604: Adjust the first window and the second window according to the visible area defined by the root task container, and display the adjusted first window and the second window in the corresponding split-screen area, and display the first page in the adjusted first window and the second page in the adjusted second window.

[0122] Steps S1603 to S1604 can be referred to steps S1003 to S1004, and will not be repeated here.

[0123] Therefore, the display device can flexibly switch between the first mode and the second mode without the need for the display device to pre-configure system parameters to enable or disable the first and second modes, and is not limited by the application's support for the first and second modes.

[0124] Figure 11 is a schematic diagram of the application runtime architecture provided according to some embodiments of this application. As shown in Figure 11, the architecture of multi-window applications and related applications is mainly illustrated. Among them, multi-window applications are the main applications for implementing multi-window services. Multi-window applications can calculate the area information of each window in multi-window mode (such as the size of the window, its position on the screen, etc.). The display device can establish a screen projection connection with a projection device (in the following embodiments of this application, the projection device used to provide projection content is referred to as the first device) based on different multi-screen interaction protocols, and transmit projection signals. In one example, the multi-screen interaction protocol may include the Miracast protocol and the Content Sharing protocol. The Miracast protocol mainly supports mirroring the content displayed on other devices (such as the first device) to the display device. The Content Sharing protocol mainly transmits the content displayed on other devices (such as the first device) to the display device in a push manner.

[0125] The following configurations can be used to implement screen mirroring functionality in multi-window mode: In one implementation, the multi-window application integrates the Miracast protocol, which is encapsulated in a Miracast service to support screen mirroring in multi-window mode. In another implementation, the content sharing application integrates the Content Sharing protocol, supporting push-to-screen mirroring. Correspondingly, the multi-window application integrates a Content Sharing Service Binder, and the content sharing application integrates a Content Sharing Service. The multi-window application and the content sharing application can establish a connection based on the Content Sharing Service Binder and Content Sharing Service, respectively. After the connection is established, they can communicate bidirectionally using a communication mechanism (such as Messenger). Thus, the multi-window application can obtain the screen mirroring signal from the content sharing application, enabling push-to-screen mirroring in multi-window mode.

[0126] In some embodiments, the multi-window application is also configured with related interfaces, such as system broadcast interfaces, application settings interfaces, Hsp_avmw, and System actions. In some embodiments, the multi-window application may also integrate services such as game templates, Source+ projection, Connection Tips, and integrated projection UI. In some embodiments, the content sharing application may also integrate services such as Sharing Session and Player. When the display device is displaying a second application, if it needs to enter multi-window mode, it needs to launch the multi-window application and continue to display the second application within the multi-window application. This process involves switching between applications: launching the multi-window application will overlay the window where the second application is located, and the second application will go to the background and stop playing. After launching, the multi-window application will call the relevant interfaces of the second application within the application to continue displaying the second application within the multi-window application. During this process, when the second application goes to the background and is displayed again within the multi-window application, screen flickering, blackouts, and playback interruptions may occur. When the display device exits multi-window mode, it also involves switching between applications: when exiting the multi-window application, the second application will be paused first, and after exiting the multi-window application, the second application will be restarted. During this process, as the display device switches from multi-window mode to full-screen display of the second application, phenomena such as screen flickering, black bars, and playback interruptions may occur. The display device's use of multi-window mode can affect the currently displayed application.

[0127] The display device provided according to some embodiments of this application can effectively ensure the normal display of the current application when starting and exiting multi-window mode. In the embodiments of this application, the display device displays a second application in a first window before entering multi-window mode. The first window is in full-screen mode, that is, the display device displays the second application in full-screen mode. In the embodiments of this application, the process of applying multi-window mode is described using the LiveTv live streaming application as an example.

[0128] Figure 12 is a flowchart illustrating the startup of a second application using a display device according to some embodiments of this application. As shown in Figure 12, the specific steps are as follows:

[0129] In step S1801, in response to the instruction to start the second application, the second application is started, and the background remote service is started.

[0130] As shown in Figure 11, the second application integrates a background remote service (such as TvRemoteService). This background remote service supports communication mechanisms (such as Messenger mechanisms) and can establish a communication link with the multi-window application to establish an association between the multi-window application and the second application, or in other words, to bind the multi-window application and the second application. In one implementation, the multi-window application integrates an application communication service (such as TvServiceBinder). Based on this application communication service, the multi-window application can establish a communication link with the background remote service in the second application for binding. Figure 13 is a timing diagram of a display device applying a multi-window mode according to some embodiments of this application. As shown in Figure 13, in response to the instruction to start the second application, the display device controls the display of the first window by the TV view service in the second application, and displays the content of the second application (such as a live broadcast) in the first window. Among them, the TV view service controls the first window to be in full-screen mode (S1901). After controlling the full-screen display of the first window, the TV view service also sends a start instruction to the background remote service (S1902) to start the background remote service (S1903).

[0131] Step S1802: Register the callback interface in the background remote service.

[0132] A callback interface is used to trigger the sending of region information to the second application when the background remote service receives region information transmitted by the multi-window application. The callback interface is registered in the background remote service; in the example, a callback process can be created and a callback method (such as ServiceCallback) can be set. This callback process is used to trigger a callback when the background remote service receives region information transmitted by the multi-window application, so that the received region information is sent back to the TV view service in the second application through this callback interface. As shown in Figure 13, after starting the background remote service, the TV view service registers a callback interface in the background remote service, such as setting the ServiceCallback callback method (S1904). The region information is information instructing the second application to adjust the first window, as indicated by the multi-window application. Therefore, when the display device starts the second application, it starts the background remote service and registers the callback interface, which lays the foundation for establishing communication with the multi-window application and obtaining the region information indicated by the multi-window application in a timely manner, ensuring the quality of use of the second application when applying multi-window mode. Figure 14 is an interactive schematic diagram of a display device applying multi-window mode according to some embodiments of this application. The display device responds to the instruction to launch the second application, launches the second application, as shown in Figure 14A, displays the first window 71 in full-screen mode, and displays the second application within the first window 71.

[0133] Figure 15 is a flowchart illustrating the process of a display device entering multi-window mode according to some embodiments of this application. As shown in Figure 15, the specific steps are as follows:

[0134] Step S2101: In response to the instruction to start multi-window mode, start the multi-window application and establish a communication mechanism between the second application and the multi-window application.

[0135] In one implementation, the entry point for multi-window mode can be configured in the system settings menu, shortcut menu, sidebar, drop-down menu, drop-up menu, etc. The display device can respond to user control commands by displaying the corresponding menu on the first window, and in response to the user's command to select the multi-window mode entry point (i.e., the command to start multi-window mode), launch the multi-window application and establish a communication mechanism between the second application and the multi-window application. In another implementation, the entry point for multi-window mode can be configured on the control device as a designated key on the control device, the key value of which is mapped to the task of entering multi-window mode. The display device responds to the user's command based on the designated key input (i.e., the command to start multi-window mode), launches the multi-window application, and establishes a communication mechanism between the second application and the multi-window application. The display device, in response to this command, launches the multi-window application to support multi-window mode. The display device, in response to this command, also establishes a communication mechanism between the second application and the launched multi-window application, enabling bidirectional communication between the multi-window application and the second application.

[0136] In one implementation, as shown in Figure 13, the multi-window application responds to the user's instruction to start the multi-window mode (S1905) and starts the multi-window mode (S1906). It can send a binding instruction to the application communication service (such as TvServiceBinder) integrated within it (S1907). The application communication service responds to the binding instruction and establishes a binding relationship with the background remote service in the second application (S1908) to establish a communication mechanism (such as the Messenger mechanism) between the multi-window application and the second application, enabling the two to communicate bidirectionally.

[0137] Step S2102: Create a second window corresponding to the multi-window application on top of the first window. The second window is transparent and in full-screen mode.

[0138] After launching a multi-window application, a second window corresponding to the multi-window application is created on top of the first window. That is, the second window's window hierarchy is higher than the first window's, and its content will overlay the content of the first window. The second window is used to display the content of the multi-window application, that is, the content corresponding to the multiple applications that need to be used in multi-window mode. When launching the multi-window application, the second application is not paused; its normal playback continues. Therefore, to avoid affecting the display of the second application's content, the second window is transparent.

[0139] In this way, even if the second window is displayed over the first window, the second application located in the first window below can still be displayed normally through the second window. Therefore, the process of starting multi-window mode does not affect the normal display of the second application. As shown in Figure 13, after the multi-window application is started, a transparent second window is created over the first window (S1909). Referring to Figure 14B, a transparent second window 72 is created over the first window 71, so that the second application within the first window 71 can be displayed through the second window 72.

[0140] Step S2103: Calculate the area information of the first split-screen area in the second window through a multi-window application, and transmit the area information to the second application through a communication mechanism.

[0141] After creating the second window, that is, after launching the multi-window application's interface, the multi-window application calculates the distribution of each application within that interface. In some embodiments, the split-screen area corresponding to the second application within the multi-window application's interface is designated as the first split-screen area. The multi-window application calculates the area information of this first split-screen area, such as its size and position on the screen. After obtaining the area information of the first split-screen area, the multi-window application transmits this information to the second application through a communication mechanism, allowing the second application to adjust the first window based on this information.

[0142] As shown in Figure 13, the multi-window application calculates the initial size and position of the first split-screen area corresponding to the second application to obtain area information (S1910), and transmits this area information to the background remote service of the second application across processes via the application communication service in the form of a notification (S1911). After receiving the notification of the area information (S1912), the background remote service of the second application triggers a callback to send the area information back to the TV view of the second application (S1913).

[0143] In step S2104, the second application adjusts the size and position of the first window based on the area information.

[0144] The second application identifies the size and position of the first split-screen area from the area information. Since the first window needs to match the first split-screen area so that the second application is displayed within the area corresponding to the first split-screen area, the second application needs to adjust the size of the first window to match the size of the first split-screen area and adjust the position of the first window to match the position of the first split-screen area. As shown in Figure 13, after receiving the area information, the TV view resets the size and position of the first window (S1914).

[0145] Step S2105: Display the adjusted first window.

[0146] The adjusted first window allows the second application to be accurately displayed on the screen in the area corresponding to the first split-screen region. This achieves the effect of displaying the second application in multi-window mode. Referring to Figure 14C, if the first split-screen region is half the screen and located on the left side, the adjusted first window 71 should be half the screen size and positioned on the left side. In this case, the adjusted first window 71 can be displayed through the second window 72, achieving a split-screen effect.

[0147] Therefore, after launching a multi-window application, the display device creates a second window corresponding to the multi-window application on top of the currently displayed first window, and this second window is configured to be transparent. This way, after launching the multi-window application, there's no need to exit the currently displayed second application, and even if the second window is displayed on top of the first window, its transparency allows the second application within the first window to be displayed through it, thus ensuring the normal display of the second application. Furthermore, by establishing a communication mechanism between the multi-window application and the second application, cross-process communication is possible, allowing the user to control the second application through the upper-level multi-window application.

[0148] In some embodiments, the instruction to initiate multi-window mode further specifies the split-screen application to be used in multi-window mode, which may include a second application and a first application. The first application may be a screen-casting application, which may be an application that implements screen-casting functionality based on the Miracast protocol or the Content Sharing protocol. After entering multi-window mode, if a screen-casting signal is detected, the corresponding screen-casting image is displayed in the second split-screen area. This second split-screen area and the first split-screen area can be combined to form the entire area of ​​the second window.

[0149] In some embodiments, after entering multi-window mode, the user can further specify the first application displayed in the second split-screen area. This first application can be a screen-casting application, which can be an application that implements screen-casting functionality based on the Miracast protocol or the Content Sharing protocol. After the multi-window application detects the screen-casting signal, it displays the corresponding screen-casting screen in the second split-screen area.

[0150] According to some embodiments of this application, the following two scenarios for implementing screen projection function in multi-window mode are also provided, which are described in detail below.

[0151] Scenario 1: The specific steps of the display device entering multi-window mode according to some embodiments of this application in Scenario 1 are as follows:

[0152] Step S2201: Receive a first request sent by the first device. The first request is used to establish a screen mirroring connection.

[0153] The first device is a screen mirroring device that is to establish a screen mirroring connection with a display device. In some embodiments, the first device can be a mobile phone, tablet computer, smart wearable device, smart TV, etc. If a user wants to use the first device to provide a screen mirroring signal to the display device, that is, to display the content provided by the first device on the display device, the user can operate the first device to control the first device to send a first request to the display device to request the establishment of a screen mirroring connection with the display device.

[0154] In step S2202, in response to the first request, after the first device passes the verification, a screen mirroring connection is established with the first device through the communication device, and an instruction to start the multi-window mode is generated.

[0155] In response to the first request, the display device can perform device verification. For example, it can verify the device's security. In one example, the display device can display a pop-up notification prompting the user that the first device is requesting screen mirroring.

[0156] The display device can respond to a user's instruction to agree to screen mirroring from the first device, determine that the first device has passed verification, and establish a screen mirroring connection with the first device. Alternatively, the display device can respond to a user's instruction to reject screen mirroring from the first device, determine that the first device has not passed verification, and refuse to establish a screen mirroring connection with the first device. If the first device passes verification, the display device establishes a screen mirroring connection with the first device and also generates an instruction to start a multi-window mode; that is, the display device automatically starts a multi-window mode to simultaneously display the second application and the mirrored screen in multi-window mode.

[0157] Therefore, after receiving the first request from the first device to establish a screen mirroring connection, if the first device is verified, the display device can automatically generate a command to start a multi-window mode and start the multi-window mode based on the command, so as to display the screen mirroring provided by the first device in multi-window mode. In this way, it can be ensured that the screen mirroring is displayed simultaneously by automatically starting the multi-window mode without affecting the currently displayed second application.

[0158] Scenario 2: The specific steps of the display device according to some embodiments of this application entering multi-window mode in Scenario 2 are as follows:

[0159] In step S2301, in response to the instruction to start multi-window mode, a second request is also generated, which is used to request the establishment of a screen mirroring connection.

[0160] In response to the command to start multi-window mode, the display device generates a second request to automatically trigger the screen mirroring function.

[0161] Step S2302: Send a second request to the first device via the communication device.

[0162] The display device actively sends a second request to the first device. The first device is a device scanned by the display device, which can be a device selected by the user, for example, a device selected by the user based on a scan result list provided by the display device; the first device can also be a device that has established a screen mirroring connection with the display device. That is, the split-screen applications that the display device prioritizes in multi-window mode include screen mirroring applications. When the display device starts multi-window mode, it defaults to prioritizing the use of screen mirroring functionality in multi-window mode, thus automatically finding a screen mirroring device and establishing a screen mirroring connection without requiring user instruction. Therefore, when the display device starts multi-window mode, it also starts the screen mirroring application to actively request a screen mirroring connection from the first device and to request the screen mirrored image from the first device. This effectively simplifies the user's operation steps for using the screen mirroring application. The following provides a detailed description of the screen mirroring function used in multi-window mode. The specific steps of the display device displaying the screen mirrored image according to some embodiments of this application are as follows:

[0163] Step S2401: The waiting screen is displayed in the second split-screen area of ​​the second window.

[0164] The second split-screen area, together with the first split-screen area, constitutes the entire area of ​​the second window. Before a multi-window application detects a casting signal (i.e., before displaying the cast image), a waiting screen can be displayed in the second split-screen area. This waiting screen informs the user that a casting connection has been established and is waiting for a casting signal, clarifying why the cast image is not currently displayed. Furthermore, in cases where the casting connection is automatically initiated, the user is informed that the casting function has been automatically activated.

[0165] Step S2402: After listening to the screen projection signal sent by the first device through the communication device, the screen projection screen is displayed in the second split-screen area of ​​the second window according to the screen projection signal.

[0166] After a multi-window application detects a screen mirroring signal—for example, by receiving the signal via the Miracast protocol or through a content sharing communication service—it generates and displays the corresponding screen mirroring image. This allows the display device to show a waiting screen before displaying the screen mirroring image, informing the user of the current status of the screen mirroring application and preventing the split-screen area from being undisplayed, thus improving the user experience.

[0167] The specific steps of the process for displaying the projected screen in full screen using a display device according to some embodiments of this application are as follows:

[0168] In step S2501, in response to the full-screen command, the multi-window application sends a pause command to the second application through a communication mechanism.

[0169] If a user wants to view the projected screen in full screen, they can input a full-screen command into the display device. In response to this command, the display device can send a pause command to the second application via a communication mechanism through multi-window applications, thus preventing the user from missing the content being played in the second application after the projected screen is displayed in full screen.

[0170] In step S2502, the second application pauses playback and adjusts the projected screen to full-screen mode.

[0171] The second application pauses its current playback in response to a pause command. The multi-window application adjusts the projected image to full-screen mode to meet the user's viewing needs. Therefore, the display device can respond to the user's full-screen command and adjust the projected image to full-screen mode when the user wants to view it. Furthermore, it can automatically pause the playback of the second application to prevent the user from missing its content. In some embodiments, when the multi-window application is displaying the projected image in full-screen mode, the display device can respond to the user's instruction to exit full-screen mode, allowing the multi-window application to adjust the projected image according to the second split-screen area, so that the adjusted projected image is displayed in the second split-screen area of ​​the second window. Furthermore, the multi-window application also sends a playback command to the second application via a communication mechanism to resume playback in the second application.

[0172] The specific steps of the process for a display device to exit multi-window mode according to some embodiments of this application are as follows:

[0173] In step S2601, in response to the instruction to exit multi-window mode, the multi-window application sends an adjustment instruction to the second application through a communication mechanism.

[0174] If a user wants to exit multi-window mode, they can input an exit command using the Back button, Home button, or other buttons on the control device. The adjustment command is used to instruct the first window to be adjusted to full-screen mode. As shown in Figure 13, in response to the user's instruction to exit multi-window mode (S1915), the display device sends an adjustment command to the background remote service in the second application via the application communication service (S1916-S1917). After receiving the adjustment command, the background remote service triggers a callback to redirect the adjustment command to the TV view of the second application (S1918).

[0175] Step S2602: Exit the second window, and the second application adjusts the first window to full-screen mode in response to the adjustment command.

[0176] The multi-window application control exits the second window to exit multi-window mode. The second application responds to the adjustment command by adjusting the first window to full-screen mode so that, after exiting multi-window mode, the second application can be displayed in full-screen mode again. As shown in Figure 13, after receiving the adjustment command, TV view adjusts the first window back to full-screen mode (S1919).

[0177] Therefore, in response to the command to exit multi-window mode, the display device can automatically restore the first window to full-screen mode when exiting the second window, ensuring the display quality of the second application after exiting multi-window mode. Furthermore, since the multi-window application and the second application belong to different windows, exiting the multi-window application will not affect the normal playback of the second application. As shown in Figure 14D, in response to the command to exit multi-window mode, the display device exits the second window 72 and readjusts the first window 71 back to full-screen mode, restoring full-screen display of the second application after exiting multi-window mode.

[0178] According to some embodiments of this application, another specific step in the process of exiting multi-window mode for a display device is as follows:

[0179] Step S2701: If no screen mirroring signal is detected, generate an instruction to exit multi-window mode.

[0180] When using screen mirroring in multi-window mode on a display device, if the multi-window application does not detect the mirroring signal, it means there is no screen to be mirrored, or the screen to be mirrored is empty. In other words, there is no need to continue using multi-window mode. Therefore, multi-window mode can automatically generate a command to exit multi-window mode.

[0181] In step S2702, in response to the instruction to exit multi-window mode, the multi-window application sends an adjustment instruction to the second application through a communication mechanism.

[0182] Step S2702 can be referred to step S2601, and will not be repeated here. Therefore, when the display device is displaying the screen projection provided by the first device in multi-window mode, if no screen projection signal is detected, it means that there is currently no screen projection. The device can automatically generate a command to exit multi-window mode, so as to automatically exit the multi-window application and continue to display the second application in full-screen mode, avoiding areas without screen projection, thereby ensuring the user experience.

[0183] In some embodiments, a multi-window display method is provided, applied to a display device, wherein the display device displays a first window and displays a first page within the first window, the first window being in full-screen mode; the method includes: in response to a user instruction to launch a first application, displaying a second window on top of the first window and displaying a second page of the first application within the second window, wherein the second window is in full-screen mode; in response to a user instruction to enter a multi-window mode, if the multi-window mode is a first mode, creating a first task stack corresponding to the first application, and listening to the first task stack through a pre-registered listening service, wherein the first task stack includes application pages associated with the first application, and the application pages include a second page; when the listening service detects that the first task stack has been created, generating window information for adjusting the second window; according to the window information, adjusting the visible area defined by the first window container associated with the first task stack from the entire visible area of ​​the screen to a first visible area, the first window container including application windows corresponding to each application page, the application windows including a second window, wherein the first window container is used to define the visible area of ​​the application window on the screen; and controlling the display to display the second window on top of the first window in full-screen mode, according to the first visible area.

[0184] In some embodiments, another multi-window display method is also provided, which can be applied to a display device, wherein the display device displays a second application within a first window. The method includes: in response to an instruction to initiate a multi-window mode, launching a multi-window application and establishing a communication mechanism between the second application and the multi-window application; creating a second window corresponding to the multi-window application on top of the first window, the second window being transparent and in full-screen mode; calculating region information of a first split-screen area in the second window through the multi-window application and transmitting the region information to the second application through the communication mechanism; adjusting the size and position of the first window according to the region information by the second application; and displaying the adjusted first window.

[0185] The specific implementation process of the above method is detailed in the aforementioned embodiments and will not be repeated here.

[0186] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the embodiments.

Claims

1. A display device, comprising: The monitor is configured to display a first window and display a first page within the first window, wherein the first window is in full-screen mode. Memory, configured to store computer programs; At least one processor, connected to the display and the memory, is configured to execute the computer program to enable the display device to: In response to a user's instruction to launch a first application, a second window is displayed on top of the first window, and a second page of the first application is displayed within the second window, wherein the second window is in full-screen mode; In response to a user instruction to enter multi-window mode, if the multi-window mode is a first mode, a first task stack corresponding to the first application is created, and the first task stack is monitored through a pre-registered listening service. The first task stack includes application pages associated with the first application, and the application pages include the second page. When the listening service detects that the first task stack has been created, it generates window information for adjusting the second window; Based on the window information, the visible area defined by the first window container associated with the first task stack is adjusted from the entire visible area of ​​the screen to the first visible area. The first window container includes the application window corresponding to each application page, and the application window includes the second window. The first window container is used to define the visible area of ​​the application window on the screen. The display is controlled to show the second window above the first window in full-screen mode, according to the first visible area.

2. The display device according to claim 1, wherein when, in response to a user instruction to launch a first application, a second window is displayed on top of the first window, and a second page of the first application is displayed within the second window, the at least one processor is specifically configured to execute the computer program to cause the display device to: In response to the user's instruction to launch the first application, a first task stack corresponding to the first application is created, and the first task stack corresponding to the first application is monitored through the monitoring service; When the listening service detects that the first task stack has been created, it creates the first window container associated with the first task stack. Set the visible area defined by the first window container to the entire visible area of ​​the screen, and set the window layer defined by the first window container to the top level; Determine the second page to be displayed in the first task stack; The display is controlled to show the second window on the top layer according to the entire visible area of ​​the screen, and the second page is displayed within the second window.

3. The display device according to claim 1, wherein when the listening service detects that the first task stack has been created and generates window information for adjusting the second window, the at least one processor is specifically configured to execute the computer program to cause the display device to: When the listening service detects that the first task stack has been created, it obtains a task stack list. The task stack list includes the package name of the application that created the task stack and a task stack identifier. The task stack identifier is used to uniquely identify the corresponding task stack. The application list includes the package name of the first application and the task stack identifier of the first task stack; Based on the package name of the first application, determine the task stack identifier corresponding to the first task stack from the task stack list; The window information is generated based on the task stack identifier corresponding to the first task stack and the default window attributes.

4. The display device according to claim 3, wherein when the visible area defined by the first window container associated with the first task stack is adjusted from the entire visible area of ​​the screen to the first visible area, the at least one processor is specifically configured to execute the computer program to cause the display device to: Retrieve a pre-stored list of mapping relationships, which includes mappings between task stacks and surface controllers. These mappings are stored when the listening service detects a created task stack. The surface controller is used to manage the container parameters of the window container associated with the corresponding task stack, and the container parameters include the visible area defined by the window container; According to the mapping relationship list, obtain the first surface controller corresponding to the first task stack; Based on the window information, the visible area defined by the first window container is adjusted from the entire visible area of ​​the screen to the first visible area by the first surface controller.

5. The display device according to claim 1, wherein when adjusting the visible area defined by the first window container associated with the first task stack from the entire visible area of ​​the screen to a first visible area according to the window information, the at least one processor is specifically configured to execute the computer program to cause the display device to: Based on the size information in the window information, the visible area defined by the first window container is reduced from the entire visible area of ​​the screen to the first visible area; Furthermore, based on the position information in the window information, the position of the first visible area is defined as a first position on the screen.

6. The display device according to claim 1, wherein when the display shows a first window and a first page is displayed within the first window, the at least one processor is further configured to execute the computer program to cause the display device to: In response to a user instruction to enter the multi-window mode, if the multi-window mode is the second mode, obtain the total number N of the split-screen areas indicated by the user, and the N split-screen applications indicated by the user to be used in the second mode, wherein... N is an integer greater than or equal to 2; Create a task stack corresponding to the split-screen application and call a multi-window interface to call the task management controller through the multi-window interface to create N root task containers. The task stack corresponding to the split-screen application includes the application page associated with the split-screen application. Each root task container corresponds to a split-screen area. The root task container is used to define the visible area of ​​the window contained in the root task container on the screen. The visible area defined by the root task container corresponds to the corresponding split-screen area. Mount the application windows associated with each application page in the task stack to the corresponding root task container; The display is controlled to adjust the application window corresponding to the split-screen application according to the visible area defined by the root task container, and the adjusted application window is displayed in the corresponding split-screen area, and the application page associated with the split-screen application is displayed in the adjusted application window.

7. In the display device according to claim 6, when creating the task stack corresponding to the split-screen application and calling the multi-window interface to create N root task containers by calling the task management controller through the multi-window interface, the at least one processor is specifically configured to execute the computer program to enable the display device to: Based on the total number N of the split-screen regions, calculate the region information of the visible area of ​​each split-screen region on the screen; The multi-window interface is invoked to call the task management controller to create the corresponding root task container according to the region information. in, According to the size information in the area information, set the size of the visible area defined by the root task container, and according to the position information in the area information, set the position of the visible area defined by the root task container on the screen.

8. The display device according to any one of claims 1-5, wherein after controlling the display to display the second window in the first visible area above the first window in full-screen mode, the at least one processor is further configured to execute the computer program to cause the display device to: In response to a user instruction to enter multi-window mode, if the multi-window mode is a third mode, the total number M of the split-screen areas indicated by the user, and the M split-screen applications indicated by the user to be used in the third mode, wherein... M is an integer greater than or equal to 2; Create a task stack corresponding to the split-screen application and call a multi-window interface to call the task management controller through the multi-window interface to create M root task containers. The task stack corresponding to the split-screen application includes the application page associated with the split-screen application. Each root task container corresponds to a split-screen area. The root task container is used to define the window hierarchy of the windows contained in the root task container and the visible area on the screen. The visible area defined by the root task container corresponds to the corresponding split-screen area. The window hierarchy is located below the second window. Mount the application windows associated with each application page in the task stack to the corresponding root task container; The display is controlled to adjust the application window corresponding to the split-screen application according to the visible area defined by the root task container, and the adjusted application window is displayed in the corresponding split-screen area, and the application page associated with the split-screen application is displayed in the adjusted application window, wherein each adjusted application window is displayed below the second window.

9. The display device according to any one of claims 1-5, wherein after controlling the display to display the second window in the first visible area above the first window in full-screen mode, the at least one processor is further configured to execute the computer program to cause the display device to: In response to a user-inputted mode switching command, a multi-window interface is invoked to call the task management controller and create two root task containers. Each root task container corresponds to a split-screen region. The root task container defines the visible area of ​​the windows contained within it on the screen. The visible area defined by the root task container corresponds to the corresponding split-screen area; Obtain the first task stack corresponding to the first application and the second task stack to which the first page belongs, wherein the first task stack includes the application page associated with the first application and the second task stack includes the application page associated with the application to which the first page belongs; The application windows associated with the application pages in the first task stack and the application windows associated with the application pages in the second task stack are respectively mounted to the corresponding root task containers; The display is controlled to adjust the first window and the second window according to the visible area defined by the root task container, and the adjusted first window and the second window are displayed in the corresponding split-screen area, and the first page is displayed in the adjusted first window and the second page is displayed in the adjusted second window.

10. The display device according to claim 1, further comprising: communication devices; The display is configured to display a second application within a first window; The at least one processor is further configured to execute the computer program to enable the display device: In response to the command to start multi-window mode, a multi-window application is launched, and a communication mechanism is established between the second application and the multi-window application; A second window corresponding to the multi-window application is created on top of the first window. The second window is transparent and in full-screen mode. The multi-window application calculates the area information of the first split-screen area in the second window, and transmits the area information to the second application through the communication mechanism. The second application adjusts the size and position of the first window based on the region information; Control the display to show the adjusted first window.

11. The display device of claim 10, wherein the at least one processor is further configured to execute the computer program to cause the display device to: In response to an instruction to launch the second application, the second application is launched, and a background remote service is launched to support the communication mechanism; A callback interface is registered in the background remote service. The callback interface is used to trigger the sending of the region information to the second application when the background remote service receives the region information transmitted by the multi-window application.

12. The display device of claim 10, wherein before launching a multi-window application in response to an instruction to launch a multi-window mode, and before establishing a communication mechanism between the second application and the multi-window application, the at least one processor is further configured to execute the computer program to cause the display device to: Receive a first request sent by a first device, the first request being used to establish a screen mirroring connection; In response to the first request, after the first device passes verification, the screen projection connection is established with the first device through the communication device, and the instruction to start the multi-window mode is generated.

13. The display device of claim 10, wherein when the multi-window application is launched in response to an instruction to launch a multi-window mode, and a communication mechanism is established between the second application and the multi-window application, the at least one processor is further specifically configured to execute the computer program to cause the display device to: In response to the instruction to start multi-window mode, a second request is also generated, which is used to request the establishment of a screen mirroring connection; The second request is sent to the first device through the communication device.

14. The display device according to claim 12 or 13, wherein when controlling the display to show the adjusted first window, the at least one processor is further configured to execute the computer program to cause the display device to: The display is controlled to show a waiting screen in the second split-screen area of ​​the second window, and the second split-screen area and the first split-screen area together constitute the entire area of ​​the second window; After the screen projection signal sent by the first device is detected by the communication device, the display is controlled to display the screen projection image in the second split-screen area of ​​the second window according to the screen projection signal.

15. The display device of claim 14, wherein after controlling the display to show the adjusted first window, the at least one processor is further configured to execute the computer program to cause the display device to: In response to a full-screen command, the multi-window application sends a pause command to the second application through the communication mechanism; The second application pauses playback and adjusts the projected screen to full-screen mode.

16. The display device of claim 10, wherein after controlling the display to show the adjusted first window, the at least one processor is further configured to execute the computer program to cause the display device to: In response to an instruction to exit multi-window mode, the multi-window application sends an adjustment instruction to the second application through the communication mechanism; Exit the second window, and the second application adjusts the first window to full-screen mode in response to the adjustment command.

17. The display device according to claim 16, wherein the communication device is configured to establish a projection connection with the first device and receive a projection signal sent by the first device; When the multi-window application sends an adjustment command to the second application via the communication mechanism in response to an instruction to exit multi-window mode, the at least one processor is specifically configured to execute the computer program to cause the display device to: If the screen mirroring signal is not detected, generate the command to exit multi-window mode; In response to the instruction to exit multi-window mode, the multi-window application sends an adjustment instruction to the second application through the communication mechanism.

18. A multi-window display method, applied to a display device, wherein the display device displays a first window and displays a first page within the first window, the first window being in full-screen mode; the method includes: In response to a user's instruction to launch a first application, a second window is displayed on top of the first window, and a second page of the first application is displayed within the second window, wherein the second window is in full-screen mode; In response to a user instruction to enter multi-window mode, if the multi-window mode is a first mode, a first task stack corresponding to the first application is created, and the first task stack is monitored through a pre-registered listening service. The first task stack includes application pages associated with the first application, and the application pages include the second page. When the listening service detects that the first task stack has been created, it generates window information for adjusting the second window; Based on the window information, the visible area defined by the first window container associated with the first task stack is adjusted from the entire visible area of ​​the screen to the first visible area. The first window container includes the application window corresponding to each application page, and the application window includes the second window. The first window container is used to define the visible area of ​​the application window on the screen. The display is controlled to show the second window above the first window in full-screen mode, according to the first visible area.

Images