View frame transmission method and apparatus

Abstract

A view frame transmission method and apparatus. The transmission method applied to a display end comprises: the display end receives at least one data stream, the data stream comprising at least one view frame layer and a layer data block corresponding to the view frame, the layer data block being used for representing a transmission mode of the data stream, and the layer data block comprising any one of the following pieces of content: single-stream multi-layer transmission description information, dual-stream multi-layer transmission description information, and multi-stream multi-layer transmission description information; and the display end displays the view frame in at least two display areas of the display end, the view refresh rates of the at least two display areas being different.

Description

Image data transmission method and device

[0001] This application claims priority to Chinese Patent Application No. 202510061076.3, filed on January 14, 2025, entitled “Image Data Transmission Method and Processing Apparatus”, the contents of which shall be construed as incorporated herein by reference. Technical Field

[0002] This article relates to, but is not limited to, the field of display technology, and in particular to a method and apparatus for transmitting image data (view frame). Background Technology

[0003] Variable Refresh Rate (VRR) is a display technology that allows a monitor to dynamically adjust its refresh rate based on changes in the output frames. This means the monitor's refresh rate can be synchronized with the frame rate generated by the graphics processing unit (GPU), providing a smoother visual experience. For Ultra High Definition displays, the High Definition Multimedia Interface (HDMI) can also use VRR technology, but HDMI's VRR technology only provides a change in the refresh rate of the entire screen. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This disclosure provides an image data transmission method and apparatus.

[0006] On one hand, this disclosure provides an image data transmission method applied to a display device, the method comprising:

[0007] The display terminal receives at least one data stream, the data stream including at least one layer of image data and a layer data block corresponding to the image data. The layer data block is used to represent the transmission mode of the data stream, and the layer data block includes any one of the following: single-stream multi-layer transmission description information, dual-stream multi-layer transmission description information, and multi-stream multi-layer transmission description information. The display terminal displays the image data in at least two display areas of the display terminal, and the image refresh rates of the at least two display areas are different.

[0008] On the other hand, this disclosure provides an image data transmission method applied at a source end, the method comprising:

[0009] The source end sends at least one data stream to the display end. The data stream includes at least one layer of image data and a layer data block corresponding to the image data. The layer data block is used to represent the transmission mode of the image data in the data stream. The layer data block includes any one of the following: single-stream multi-layer transmission description information, dual-stream multi-layer transmission description information, and multi-stream multi-layer transmission description information. The image data is used to provide display information for at least two display areas of the display end, and the image refresh rates of the at least two display areas are different.

[0010] In another aspect, embodiments of this disclosure also provide an apparatus, including a processor and a memory storing a computer program executable on the processor, wherein the processor executes the program to implement the steps of the aforementioned image data transmission method applied to a display end, or the processor executes the program to implement the steps of the aforementioned image data transmission method applied to a source end.

[0011] Other features and advantages of this disclosure will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the disclosure. Other advantages of this disclosure may be realized and obtained by means of the methods described in the description and the accompanying drawings.

[0012] After reading and understanding the accompanying diagrams and detailed descriptions, the other aspects can be understood.

[0013] Overview of the attached figures

[0014] The accompanying drawings are used to provide an understanding of the technical solutions disclosed herein and form part of the specification. They are used together with the embodiments of the disclosed invention to explain the technical solutions of the present invention and do not constitute a limitation on the technical solutions of the present invention.

[0015] Figure 1 is a schematic diagram of HMD data transmission based on single-stream dual-layer;

[0016] Figure 2 is a schematic diagram of the interaction between the source end and the display end provided in this embodiment;

[0017] Figure 3 is a schematic diagram of the video stream transmission process between the source end and the display end in an embodiment of this disclosure;

[0018] Figure 4 is a schematic diagram of the transmission method at the display end according to an embodiment of this disclosure;

[0019] Figure 5 is a schematic diagram of the source-end transmission method according to an embodiment of this disclosure;

[0020] Figure 6 is a schematic diagram of transmission method 1 according to an embodiment of this disclosure;

[0021] Figures 7A and 7B are schematic diagrams of transmission method 2 according to an embodiment of this disclosure;

[0022] Figures 8A and 8B are schematic diagrams of transmission method 3 according to an embodiment of this disclosure;

[0023] Figures 9A and 9B are schematic diagrams of transmission method 4 according to an embodiment of this disclosure;

[0024] Figure 10 is a schematic diagram of another transmission method 5 according to an embodiment of the present disclosure;

[0025] Figures 11A and 11B are schematic diagrams of multi-view transmission according to embodiments of this disclosure.

[0026] Figure 12 is a flowchart of the change of transmission method according to an embodiment of this disclosure;

[0027] Figure 13 is a schematic diagram of the apparatus according to an embodiment of this disclosure.

[0028] Detailed Explanation

[0029] This disclosure describes several embodiments, but these descriptions are exemplary and not limiting, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.

[0030] This disclosure includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this disclosure may also be combined with any conventional features or elements to form unique inventive solutions. Any feature or element of any embodiment may also be combined with features or elements from other inventive solutions to form another unique inventive solution. Therefore, it should be understood that any feature shown and / or discussed in this disclosure may be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.

[0031] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that the method or process does not depend on the specific order of steps described herein. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims relating to the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments disclosed herein.

[0032] In the DisplayPort (DP) protocol, an example of transmitting AR / VR (Augmented Reality / Virtual Reality) video is shown in Figure 1, which illustrates data transmission in a Head-Mounted Display (HMD) based on a single-stream, dual-layer transmission. Single-stream, dual-layer transmission refers to transmitting multiple types of data simultaneously through different layers or channels within the same video stream. In the scheme shown in Figure 1, high-definition and low-definition areas are separated into different layers. Layer 0 (hereinafter referred to as L0) carries low-resolution image data, while Layer 1 (hereinafter referred to as L1) carries high-resolution image data. Resolution refers to the number and density of pixels in an image or video; higher resolution means more pixels and richer details. Although high-resolution video data is transmitted in layered transmission, its display capability depends on the resolution of the HMD. When the HMD device's resolution supports L1 display, the L0 and L1 image data are combined and displayed at the HMD end, as shown in Figure 1. When the HMD device's resolution is insufficient to support L1 display, only the L0 image data from the received bitstream is presented. If the source end has high processing capabilities, the data can also be combined at the source end and transmitted to the HMD end as a single stream via the DP protocol.

[0033] In the aforementioned AR / VR transmission scenarios, L0 serves as the base layer. At the HMD end, L0 image data can be expanded to fit the AR / VR projection area. Therefore, its transmission is low-resolution compared to the actual projection area of ​​ultra-high definition. L1, as the main gaze area in the HMD, is often not expanded at the HMD end and directly projects the original image size.

[0034] However, the above-mentioned traffic splitting design has the following problems: For the overlapping area of ​​the two streams, that is, the part where the L0 image data overlaps with the L1 image data in Figure 1, taking the left view (L in the figure) as an example, the content of the central area of ​​the L0 image data is the same as the content of the central area of ​​the L1 image data. Furthermore, due to the merging of the L0 and L1 image data during display, the content of the central area of ​​the L0 image data is repeatedly sent, resulting in a certain amount of bandwidth waste. In addition, the above traffic splitting design does not change the refresh rate; that is, the screen refresh rate is uniform. The refresh rate refers to the number of times the display redraws the image per second. For example, a 60Hz refresh rate means that the display refreshes 60 times per second. The higher the refresh rate, the better the stability of the displayed image (screen). Moreover, the above traffic splitting design is for AR / VR scenarios and is not suitable for ordinary display scenarios.

[0035] Considering the multi-view information transmission and ultra-high pixel backplane refresh rate of future light field display technology, the display end may support the refresh rate change of specific screen areas. Therefore, this embodiment provides a technical solution that can realize Variable Partition Variable Refresh Rate (VPVRR), which can support the refresh rate change for specific screen areas.

[0036] Figure 2 is a schematic diagram of the interaction between the source and display terminals provided in this embodiment. The source terminal includes a first encoding module for generating image data (View Frame), and the display terminal includes a second encoding module for parsing the image data. The display terminal may also include registers and a driver chip, wherein the registers are used to store the parsed image data, and the driver chip is used to drive the display screen to display the image data. The display terminal can be, for example, various display devices with display screens.

[0037] Figure 3 is a schematic diagram of the image data transmission process between the source and display ends. As shown in the figure, it includes the following steps:

[0038] Step 1: Establish a data transmission channel between the display end and the source end;

[0039] For example, the display end and the source end establish an ultra-high-definition data transmission channel through the DP protocol; in this example, only the DP protocol is used. In other embodiments, other protocols can be used to establish the transmission channel.

[0040] Step 2: The display end and the source end communicate their capabilities.

[0041] By establishing a capability communication mechanism between the display and the source, the source can output data streams that meet its display requirements. For example, the display can inform the source that it supports VPVRR and related VPVRR capabilities, so that the source can generate matching image data based on the display's capabilities.

[0042] Optionally, the display can also determine whether the bandwidth is supported and configure the transmission link. For example, if the current available bandwidth of the DP line is 50Gbps, and the estimated video stream usage is 47.5Gbps, which is less than the available bandwidth, it means that the video stream can be transmitted at present. If the display indicates that it supports VPVRR and the switch is on, then VPVRR link configuration will be performed.

[0043] The display terminal can notify the source terminal of its capabilities through status data. For example, the source terminal sends a status data read message to the display terminal, and after receiving the message, the display terminal sends status data to the source terminal, including but not limited to any one or more of the following: supported single-stream format types, supported dual-stream format types, and supported multi-stream format types.

[0044] In this embodiment, single-stream format types include: single-stream single-layer, single-stream dual-layer, and single-stream multi-layer; dual-stream format types include: dual-stream dual-layer and dual-stream multi-layer; multi-stream format types include: multi-stream multi-layer. The term "multi-layer" includes two or more layers, and the term "multi-stream" includes three or more streams.

[0045] Step 3: The source end transmits image data to the display end according to the capabilities supported by the display end;

[0046] If the display terminal has the VPVRR function enabled, the source terminal will directly process the data according to the VPVRR type supported by the display terminal and transmit the image data to the display terminal.

[0047] Step 4: The display terminal displays the received image data, and the image refresh rates of at least two display areas of the display terminal are different.

[0048] This embodiment enables the source end to transmit corresponding data streams to the display end that has enabled the VPVRR function through capability interaction between the source end and the display end. Different transmission methods are selected according to the capabilities of the display end, and the display end uses the corresponding display method to display, thereby achieving the effect of at least two display partitions with different refresh rates, which can provide users with a better display experience.

[0049] The above is the overall end-to-end communication process. The following sections will explain the process from the display end and the information source end.

[0050] For the display end, the processing procedure is shown in Figure 4, including the following steps:

[0051] Step 11, the display terminal receives at least one data stream, the data stream including at least one layer of image data;

[0052] Step 12: The display terminal displays the image data in at least two display areas of the display terminal, and the image refresh rates of the at least two display areas are different.

[0053] The display screen of the display terminal includes at least two display areas (or display partitions).

[0054] Optionally, during display, the display terminal displays one layer of image data from a data stream in a display area, or the display terminal displays a group of layer image data from one layer of image data in a display area.

[0055] By receiving and processing image data at the display end, the refresh rate of at least two display areas on the screen is different, thus improving the user's viewing experience.

[0056] For example, the data stream further includes layer data blocks corresponding to the image data. These layer data blocks represent the transmission mode of the current data stream and include any one of the following: single-stream multi-layer transmission description information, dual-stream multi-layer transmission description information, or multi-stream multi-layer transmission description information. Through these layer data blocks, the display device can determine the transmission mode used and process the data accordingly.

[0057] For the source end, the processing procedure is shown in Figure 5, including the following steps:

[0058] Step 21: The source end sends at least one data stream to the display end. The data stream includes at least one layer of image data. The image data is used to provide display information for at least two display areas of the display end. The image refresh rates of the at least two display areas are different.

[0059] For example, the source sends a data stream to the display, the data stream including at least one layer of image data; or, the source sends two data streams to the display, each data stream including at least one layer of image data; or, the source sends at least three data streams to the display, each data stream including at least one layer of image data.

[0060] In order to achieve different refresh rates for at least two display areas on the display end, the sizes of at least two layers of image data transmitted at the source end can be different.

[0061] The source end can decide whether to process the image data based on the capabilities of the display end. By sending the data stream from the source end to the display end, the image refresh rate of at least two display areas on the display screen is different, thereby improving the user's viewing experience.

[0062] In an exemplary embodiment, the source sends image data to the display via data blocks, each containing basic information about the transmission. For single-stream, single-layer transmission, the data block contains the information shown in Table 1. For single-stream, dual-layer transmission, the data block may contain the information shown in Table 2. For dual-stream, dual-layer transmission, the data block may contain the information shown in Table 3. For single-stream, multi-layer transmission, the data block may contain the information shown in Table 4. For dual-stream, multi-layer transmission, the data block may contain the information shown in Table 5. For multi-stream, multi-layer transmission, the data block may contain the information shown in Table 6. Furthermore, for dual-stream, multi-layer transmission, when the number of layers in the two data streams is different, the data block contains the format shown in Table 7.

[0063] The table includes one or more of the following information: Product ID, Display Parameters, Type x Detailed Timing, Display Interface, VPVRR Control Data Block, and VPVRR Layer Data Block. The VPVRR Control Data Block represents the negotiation information for the transmission mode, indicating the data stream and data layer attributes preferred by the display. The VPVRR Layer Data Block represents the transmission mode of the data stream, indicating the data stream and data layer attributes of the receiving data block. Taking a dual-stream, dual-layer transmission mode as an example, the data block format used is Table 3. Two data streams are transmitted: the first data stream (or first stream) includes first-layer image data, and the second data stream (or second stream) includes second-layer image data. When transmitting the first layer (L0) data of the first data stream, the data block carries the content from the left half of Table 3; when transmitting the second layer image data of the second data stream, the data block carries the content from the right half of Table 3.

[0064] Table 1 Single Flow Single Layer

[0065] Table 2 Single-flow double-layer

[0066] Table 3 Dual-flow double-layer

[0067] Table 4 Single-flow multilayer

[0068] Table 5 Dual-flow multilayer

[0069] Table 6 Multi-flow Multi-layer

[0070] Table 7 Dual-flow multilayer (asymmetric):

[0071] In an exemplary embodiment, the source end may also carry layer data blocks corresponding to the image data in the data stream, namely, the VPVRR layer data blocks mentioned above. These layer data blocks represent the transmission mode of the current data stream and include any one of the following: single-stream multi-layer transmission description information, dual-stream multi-layer transmission description information, and multi-stream multi-layer transmission description information. Through these layer data blocks, the display end can be informed of the transmission mode used, so that the display end can process the data using the corresponding processing method.

[0072] The above-mentioned single-stream multi-layer transmission description information is used to indicate that this is a single-stream transmission, that is, there is only one data stream, or it can also be called single-stream transmission description information. The single-stream multi-layer transmission description information includes one of the following information: single-stream single-layer description information, single-stream double-layer description information, single-stream multi-layer description information, for example, represented by a two-bit binary field. An exemplary way is: 00 represents single-stream single-layer, 01 represents single-stream double-layer, 10 represents single-stream triple-layer, and 11 represents single-stream quadruple-layer.

[0073] The aforementioned dual-stream multi-layer transmission description information is used to indicate that this is a dual-stream transmission, meaning there are two data streams. It can also be referred to as dual-stream transmission description information. This dual-stream multi-layer transmission description information includes one of the following: first-stream description information and second-stream description information. In each first-stream description information, a two-bit binary field can be used to represent the number of layers supported by the current stream. An example is as follows: in the first-stream description information, 00 indicates that this stream supports single-stream single-layer, meaning there is only one layer of image data in the first stream; 01 indicates that this stream supports single-stream dual-layer, meaning there are two layers of image data in the first stream. The second-stream description information is the same and will not be described further here.

[0074] The aforementioned multi-stream, multi-layer transmission description information is used to indicate that this is a multi-stream transmission, meaning there are three or more data streams. Alternatively, it can be referred to as multi-stream transmission description information, which includes description information for each stream. In the description information for each stream, a two-bit binary field can be used to represent the number of layers supported by the current stream. An example is as follows: in the first stream description information, 00 indicates that this stream supports a single stream, single layer, meaning there is only one layer of image data in the first stream; 01 indicates that this stream supports a single stream, dual layers, meaning there are two layers of image data in the first stream.

[0075] For example, the layer data block may also include data stream number information to indicate which stream is being transmitted this time. It may also include layer number information to indicate which layer is being transmitted this time. The layer data block can specify the content definition of the data stream, such as the total number of streams and layers, and which stream and layer of data is being transmitted currently.

[0076] Layer data blocks carry information sent from the source to the display. These layer data blocks may include, but are not limited to, one or more of the following: block revision, number of payload bytes, total number of layers, current layer, number of streams, multiple layer single stream transport description, multiple layer dual stream transport description, and multiple layer multiple stream transport description.

[0077] See Table 8 below for layer data blocks. The offsets and bit values ​​in the table are for illustrative purposes only; the offsets and number of bits may vary depending on adjustments to reserved values. In the table, 'h' represents hexadecimal and 'b' represents binary. The field values ​​in the table are for illustrative purposes only and can be adjusted as needed. For example, the total number of bits in a field can be increased or decreased, such as changing content defined or described using 2 bits to 1 bit.

[0078] Table 8 Data Blocks

[0079] In an exemplary embodiment, the display terminal can request the source terminal to change the transmission mode based on its own situation, such as changing from multi-stream multi-layer to dual-stream multi-layer, or from dual-stream dual-layer to single-stream single-layer. For example, the display terminal can send a control data block to the source terminal, requesting the source terminal to change the transmission mode, for example, specifying the desired transmission mode. This control data block can reflect the current capabilities of the display terminal. After receiving the control data block sent by the display terminal, the source terminal sends a data stream to the display terminal according to the transmission mode indicated in the control data block. When necessary, the display source terminal can regenerate the image data. The display terminal receives the data stream sent by the source terminal according to the changed transmission mode.

[0080] For example, when the display device has visual tracking functionality, it can change the transmission mode of data blocks by controlling them when it detects a change in the viewer's gaze area; or, if the display device's processing power is limited, resulting in a limited refresh rate for local areas (e.g., when some layers of image data are discarded), the display device can change the transmission mode of data blocks; or, when a bandwidth bottleneck occurs (e.g., bandwidth is below a preset bandwidth threshold or data block transmission times out), the display device can change the transmission mode of data blocks; or, when a rendering bottleneck occurs (determining whether a rendering bottleneck occurs can be achieved using existing technologies), the display device can change the transmission mode of data blocks. The change in transmission mode can be, for example, enabling the VPVRR function to allow the source end to initiate VPVRR mode video stream transmission, or changing the VPVRR transmission mode.

[0081] The control data block may include, but is not limited to, one or more of the following: Block Revision, Number of Payload Bytes, Multiple Layer Single Stream Transport Coordinate, Multiple Layer Dual Stream Transport Coordinate, and Multiple Layer Multiple Stream Transport Coordinate. See Table 9 below. The offsets and bit values ​​in the table are for illustrative purposes only; the offsets and bit numbers may vary depending on adjustments to reserved values. The field values ​​in the table are for illustrative purposes only and may be adjusted as needed.

[0082] Table 9 Control Data Block

[0083] In an exemplary embodiment, the positions of different display areas can be preset. Taking the setting of two display areas as an example, the center area of ​​the screen can be defined as the second display area, and the remaining area of ​​the display screen can be defined as the first display area. Furthermore, the positions of the display areas can be adjusted. For example, the display area position information can be defined through layer data blocks.

[0084] The following uses several VPVRR transmission methods as examples to illustrate the transmission methods between the source and display ends. The source end can choose any of these methods based on the capabilities of the display end, or when the display end requests a change in transmission method, the source end will transmit image data according to the transmission method required by the display end.

[0085] In the following embodiments, in order to avoid repeating the previous descriptions, the descriptions or explanations in any embodiment may be applied to other embodiments without conflict.

[0086] Figure 6 is a schematic diagram of transmission mode 1 of the present disclosure. This mode is a dual-stream dual-layer transmission mode, and the transmission process includes steps 1.1-1.2.

[0087] Step 1.1, for the source end: the source end sends two data streams to the display end—a first data stream and a second data stream, wherein the first data stream includes first layer image data (referred to as L0 data or L0 image data), which is used to provide display information for the first display area; the second data stream includes second layer image data (referred to as L1 data or L1 image data), which is used to provide display information for the second display area;

[0088] In this example, the source also includes layer data blocks corresponding to the image data in the data stream. These layer data blocks contain dual-stream, multi-layer transmission description information, specifying that this transmission is a dual-stream, dual-layer transmission. The source indicates the specific stream and layer of the dual-stream, dual-layer transmission during each data stream transmission.

[0089] From the display perspective, the final display result is that the refresh rate of the first display area is different from that of the second display area. Therefore, it is necessary to send the display data of the first display area and the display data of the second display area separately. Considering that image stitching places high demands on the processing capabilities of the display, in this embodiment, the image transmitted for the display area with a lower refresh rate is the image corresponding to the entire screen size or an image that can be displayed on the entire screen (e.g., an image that can be displayed on the entire screen after being adjusted by a scaling factor). The image transmitted for the display area with a higher refresh rate is the image corresponding to that local display area. That is, the L0 image data size corresponds to the size of the display screen, and the L1 image data size corresponds to the size of the second display area. For the display, the display data of the first display area comes from the first layer image data of the first data stream, and the display data of the second display area comes from the second layer image data of the second data stream.

[0090] Figure 6 illustrates an example where the central area of ​​the display screen is designated as the second display area, and the remaining areas as the first display area. In the first data stream, the image size of the first layer of image data corresponds to the entire screen. In the second data stream, the image size of the second layer of image data is smaller than that of the first layer of image data.

[0091] In this example, the source end does not need to merge the images. Therefore, the L1 image data transmission rate is higher than the L0 image data transmission rate per unit time, so that the image refresh rate in the second display area is higher than the image refresh rate in the first display area. The first layer of image data includes multiple frames of image data, each frame of image data corresponds to a playback timestamp, and the difference between the timestamps of two adjacent first layer image data frames is the first time difference. The second layer of image data also includes multiple frames of image data, each frame of image data corresponds to a playback timestamp, and the difference between the timestamps of two adjacent second layer image data frames is the second time difference. The first time difference is greater than the second time difference.

[0092] Each data stream consists of multiple (at least two) data blocks, the format and content of which are described above.

[0093] The resolution of the second layer image data can be the same as or higher than that of the first layer image data; this article does not impose any restrictions on this.

[0094] Because the refresh rate of the first display area is lower than that of the second display area during playback, each frame of L0 image data corresponds to multiple frames of L1 image data during transmission. As shown in Figure 6, the second stream includes L1 image data at every moment, while the first stream only includes L0 image data at a portion of the moments. If the refresh rate of the first display area is half that of the second display area, then one frame of L0 image data corresponds to two frames of L1 image data; that is, during display, one frame of L0 image data needs to be merged with two frames of L1 image data. If the refresh rate of the first display area is one-third that of the second display area, then one frame of L0 image data corresponds to three frames of L1 image data.

[0095] For example, the data stream may also carry metadata, which contains additional information related to the image data content to help users understand and manage the image data. Metadata may include one or more of the following: total duration, resolution, frame rate, total size, creation time, timestamp, etc.

[0096] Step 1.2, for the display end: The display end receives a first data stream and a second data stream. The first data stream includes first layer image data, and the second data stream includes second layer image data. The first layer image data is used to provide display information for the corresponding first display area, and the second layer image data is used to provide display information for the corresponding second display area. The display end parses L0 data and L1 data from the first stream and the second stream respectively, and displays the first layer image data and the second layer image data according to the playback time of the image data. The image refresh rate of the second display area is higher than that of the first display area.

[0097] For example, when the display terminal displays the image data according to the playback time, at least two frames of second-layer image data are merged with one frame of first-layer image data to form the image data to be displayed. Specifically, when merging a frame of second-layer image data with the same playback time with a frame of first-layer image data, the second-layer image data is used to replace (or cover) a part of the first-layer image data to form the image data to be displayed. When merging the remaining second-layer image data, the second-layer image data is used to replace a part of the image data displayed in the previous frame to form the image data to be displayed.

[0098] Taking a refresh rate of 1 / 2 that of the second display area as an example, one frame of L0 image data is merged with two frames of L1 image data before being displayed. For example, as shown in Figure 6, at time t0, the L0 image data at time t0 is merged with the L1 image data at time t0 before being displayed. During merging, a portion of the L0 image data is replaced with L1 image data. At time t1, the L0 image data at time t0 is merged with the L1 image data at time t1 before being displayed. At time t2, the L0 image data at time t2 is merged with the L1 image data at time t2 before being displayed. At time t3, the L0 image data at time t2 is merged with the L1 image data at time t3 before being displayed, and so on.

[0099] For example, if the refresh rate of the first display area is 1 / 3 of the refresh rate of the second display area, then one frame of L0 image data is merged with three frames of L1 image data and displayed. At time t0, the L0 image data at time t0 is merged with the L1 image data at time t0 and displayed. At time t1, the L0 image data at time t0 is merged with the L1 image data at time t1 and displayed. At time t2, the L0 image data at time t0 is merged with the L1 image data at time t2 and displayed, and so on.

[0100] The advantage of using layered transmission is that when the display end's processing capability is insufficient (for example, when the display end's resolution does not support the resolution of the second display area), only the received L0 image data can be displayed, resulting in strong compatibility.

[0101] In this embodiment, L0 is the base refresh layer, and L1 is the local area refresh layer, with the refresh rate of the local area being higher than that of the overall area. Image composition is performed at the display end, reducing the processing load on the source end. As shown in Figure 6, the time sequence of each stream is very regular.

[0102] Figures 7A and 7B are schematic diagrams of transmission mode 2 of the present disclosure. This mode is a dual-stream dual-layer transmission mode, and the transmission process includes steps 2.1-2.2.

[0103] Step 2.1, for the source end: the source end sends a first data stream and a second data stream to the display end, wherein the first data stream includes first layer image data (hereinafter referred to as L0 data or L0 image data), which is used to provide display information for the first display area; the second data stream includes second layer image data (hereinafter referred to as L1 data or L1 image data), which is used to provide display information for the second display area.

[0104] In this example, the source also includes layer data blocks corresponding to the image data in the data stream. These layer data blocks contain dual-stream, multi-layer transmission description information, specifying that this transmission is a dual-stream, dual-layer transmission. The source indicates the specific stream and layer of the dual-stream, dual-layer transmission during each data stream transmission.

[0105] The difference from the embodiment in Figure 6 is that, in this embodiment, the source end merges the first layer image data and the second layer image data with the same playback time into the first layer image data in the first data stream, and the second layer image data is used to replace a part of the first layer image data during merging.

[0106] Assuming the original data consists of multiple frames of L0 and L1 image data, the source end merges some of the L1 image data with the L0 image data at its local end. The merged L0 image data and L1 image data have the same playback timestamp. The merged image data is transmitted through the first data stream, while the unmerged L1 image data is transmitted through the second data stream.

[0107] As shown in Figure 7A, the refresh rate of the second display area is twice that of the first display area. In the first data stream, the first frame of image data is the merged image data of L0 at time t0 and L1 at time t0, and the second frame of image data is the merged image data of L0 at time t2 and L1 at time t2. The L1 image data at time t1 and L1 at time t3 are transmitted in the second data stream. As shown in Figure 7B, the refresh rate of the second display area is three times that of the first display area. In the first data stream, the first frame of image data is the merged image data of L0 at time t0 and L1 at time t0, and the second frame of image data is the merged image data of L0 at time t3 and L1 at time t3. The L1 image data at time t1, L1 at time t2, L1 at time t4, and L1 at time t5 are transmitted in the second data stream.

[0108] In this example, for ease of understanding, the image data in the first data stream is described as the data resulting from the merging of L0 and L1 image data. However, those skilled in the art will understand that the image data in the first data stream can be described as normal image data, while the image data in the second data stream is local area data from the image data in the first data stream.

[0109] Step 2.2, for the display end: The display end receives a first data stream and a second data stream. The first data stream includes first layer image data, and the second data stream includes second layer image data. The first layer image data is used to provide display information for the corresponding first display area, and the second layer image data is used to provide display information for the corresponding second display area. The display end parses L0 data and L1 data from the first stream and the second stream respectively, and displays the first layer image data and the second layer image data according to the playback time of the image data. The image refresh rate of the second display area is higher than that of the first display area.

[0110] When the display terminal displays the image data according to the playback time, it displays the first layer of image data and the second layer of image data according to the playback time interval of the image data. The display terminal displays one frame of the first layer of image data at the first playback time. The first layer of image data includes the content of the second layer of image data. When displaying the second layer of image data, a part of the image data displayed in the previous frame is replaced with the second layer of image data to be displayed. The replaced part is the second display area, and the remaining area is the first display area.

[0111] Taking a refresh rate of half that of the second display area as an example, since the image data in the first data stream is already merged, the display only needs to merge the L0 image data in the first data stream with the L1 image data in the second data stream when displaying the image data in the second data stream. As shown in Figure 7A, at time t0, the image data at time t0 in the first data stream is displayed; at time t1, the image data at time t0 in the first data stream is merged with the L1 image data at time t1 in the second data stream, and during merging, a portion of the L0 image data is replaced (covered) with the L1 image data; at time t2, the image data at time t2 in the first data stream is displayed; at time t3, the image data at time t2 in the first data stream is merged with the L1 image data at time t3 in the second data stream, so that the L1 image data at time t3 covers a portion of the image data at time t2, and so on.

[0112] Taking a refresh rate of 1 / 3 that of the second display area as an example, as shown in Figure 7B, at time t0, the image data at time t0 in the first data stream is displayed; at time t1, the image data at time t0 in the first data stream is merged with the L1 image data at time t1 in the second data stream, so that the L1 image data at time t1 partially covers the image data at time t0; at time t2, the image data at time t0 in the first data stream is merged with the L1 image data at time t2 in the second data stream, so that the L1 image data at time t2 partially covers the image data at time t0, or so that the L1 image data at time t2 partially covers the image data at time t0, or so that the L1 image data at time t2 partially covers the image data at time t0. The first data stream displays image data at time t1; at time t3, the image data at time t3 in the first data stream is merged with the L1 image data at time t4 in the second data stream, so that the L1 image data at time t4 covers a portion of the image data at time t3; at time t5, the image data at time t3 in the first data stream is merged with the L1 image data at time t5 in the second data stream, so that the L1 image data at time t5 covers a portion of the image data at time t3, or the L1 image data at time t5 covers a portion of the image data at time t4, and so on.

[0113] The advantage of layered transmission of image data for different display areas is that when the display is unable to support high refresh rate display due to limited processing capabilities, the display can receive only the image data corresponding to the low refresh rate area, making the display processing more flexible.

[0114] In this embodiment, the synthesis of a portion of the data frames is completed by the source end, while the synthesis of another portion of the data frames is completed by the display end. By adopting the method of this embodiment, the source end and the display end can share the processing pressure, thereby reducing the processing pressure of both the source end and the display end.

[0115] Furthermore, in this method, since the first data stream transmits merged image data, the covered local image data does not need to be transmitted, reducing the transmission of redundant images and saving bandwidth, thus requiring less bandwidth.

[0116] Figures 8A and 8B are schematic diagrams of transmission mode 3 in the embodiments of this disclosure. This mode is a single-stream single-layer transmission mode, and the transmission process includes steps 3.1-3.2.

[0117] Step 3.1, for the source end: the source end sends a first data stream to the display end, the first data stream including the first layer of image data;

[0118] In this example, the source also includes layer data blocks corresponding to the image data in the data stream. The layer data blocks include single-stream multi-layer transmission description information, and the description information indicates that this transmission is a single-stream single-layer transmission.

[0119] One optional single-stream, single-layer implementation is shown in Figure 8A. The source end generates L0 image data and sends it to the display end. In this implementation, the source end generates the image data, and the display end does not need to perform composite processing; it only needs to parse and display the data. This method can be used as a method compatible with VPVRR and existing technologies. For example, when the display end does not support VPVRR, this embodiment can be used for display.

[0120] Another optional single-stream single-layer implementation is shown in Figure 8B. The image data generated by the source end is the merged L0 image data and L1 image data. L1 is shown here for ease of understanding. This does not mean that this method is a two-layer transmission method. It is only used to help understand that the refresh resolution of the L1 area (second display area) and the L0 area (first display area) is different. In essence, this method is still a single-stream single-layer transmission method.

[0121] In this method, the source end merges L1 image data and L0 image data, and the merged image data is transmitted through the first data stream. Taking the refresh rate of the first display area as half that of the second display area as an example, one frame of L0 image data is merged with two frames of L1 image data respectively. During merging, a portion of the L0 image data is replaced with L1 image data. For example, as shown in Figure 8B, the L0 image data at time t0 is merged with the L1 image data at time t0 to form the new L0 image data at time t0, which is the final displayed image data. The L0 image data at time t0 is merged with the L1 image data at time t1 to form the new L0 image data at time t1. The L0 image data at time t2 is merged with the L1 image data at time t2 to form the new L0 image data at time t2. The L0 image data at time t2 is merged with the L1 image data at time t3 to form the new L0 image data at time t3, and so on. For example, if the refresh rate of the first display area is 1 / 3 of the refresh rate of the second display area, then one frame of L0 image data is merged with three frames of L1 image data respectively, which will not be elaborated here.

[0122] Step 3.2, for the display end: The display end receives the first data stream, parses the first layer image data from the data stream, and displays it.

[0123] For the single-stream, single-layer transmission method shown in Figure 8A above, the image data has a single refresh rate.

[0124] In the single-stream, single-layer transmission method shown in Figure 8B above, the image refresh rate of the second display area is greater than that of the first display area.

[0125] In this embodiment, all image generation is completed by the source end, which reduces the processing load on the display end. However, since all frames are complete frames, the bandwidth requirements are relatively high.

[0126] Figures 9A and 9B are schematic diagrams of transmission method 4 according to an embodiment of this disclosure. This method is a single-stream, single-layer transmission method, and the transmission process includes steps 4.1-4.2:

[0127] Step 4.1, for the source end: the source end sends a first data stream to the display end. The first data stream includes first layer image data. The first layer image data includes at least two sets of image data. Each set of image data is used to provide display information for a corresponding display area. The at least two sets of image data have different sizes and are arranged according to the playback time interval.

[0128] In this example, the source also includes layer data blocks corresponding to the image data in the data stream. The layer data blocks include single-stream multi-layer transmission description information, and the description information indicates that this transmission is a single-stream single-layer transmission.

[0129] For example, the first layer of image data includes two sets of image data. The first set of image data is used to provide display information for the corresponding first display area, and the second set of image data is used to provide display information for the corresponding second display area. The first set of image data and the second set of image data have different sizes. A frame of the first set of image data and at least one frame of the second set of image data are arranged according to the playback time interval.

[0130] Assume that the first size L0 image data is the first set of image data, and its image size is adapted to the entire display area. The second size L1 image data is the second set of image data, and its image size is adapted to the second display area. The entire display area excluding the second display area is the first display area. The resolution of the first display area is the first resolution, and the resolution of the second display area is the second resolution. For example, the second resolution is twice the first resolution. Then, after transmitting one frame of the first size L0 image data, one frame of the second size L0 image data is transmitted, as shown in Figure 9A. The first data stream includes: the first size L0 image data at time t0, the second size L0 image data at time t1, the first size L0 image data at time t2, the second size L0 image data at time t3, and so on. If the second resolution is three times the first resolution, then after transmitting one frame of first-size L0 image data, two frames of second-size L0 image data are transmitted, as shown in Figure 9B. The first data stream includes: first-size L0 image data at time t0, second-size L0 image data at time t1, second-size L0 image data at time t2, first-size L0 image data at time t3, second-size L0 image data at time t4, second-size L0 image data at time t5, and so on.

[0131] This method is similar to transmission method 2, except that all image data are transmitted through the same data stream and the same layer of image data.

[0132] Step 4.2, for the display end: The display end receives a first data stream, the first data stream includes a first layer of image data, the first layer of image data includes at least two sets of image data, each set of image data is used to provide display information for a corresponding display area, the at least two sets of image data are arranged according to the playback time interval; the display end displays the at least two image data in the corresponding display area according to the playback time of the image data.

[0133] For example, the first layer of image data includes two sets of image data. The first set of image data is used to provide display information for the corresponding first display area, and the second set of image data is used to provide display information for the corresponding second display area. One frame of the first set of image data and at least one frame of the second set of image data are arranged according to the playback time interval. The display terminal displays each frame of image data according to the arrangement order of the first set of image data and the second set of image data. When displaying the second set of image data, a portion of the most recently displayed frame of the first set of image data is replaced with the currently displayed frame of the second set of image data. The replaced portion is the second display area, and the remaining area is the first display area. The image refresh rate of the second display area is higher than that of the first display area.

[0134] Taking the example that the refresh rate of the first display area is half the refresh rate of the second display area, as shown in Figure 9A, at time t0, the first size L0 image data at time t0 is displayed; at time t1, the first size L0 image data at time t0 and the second size L0 image data at time t1 are merged, so that the second size L0 image data at time t1 covers a local area (i.e., the second display area) of the first size L0 image data at time t0; at time t2, the first size L0 image data at time t2 is displayed; at time t3, the first size L0 image data at time t2 and the second size L0 image data at time t3 are merged, so that the second size L0 image data at time t3 covers a local area of ​​the first size L0 image data at time t2, and so on.

[0135] Taking the refresh rate of the first display area as 1 / 3 of the refresh rate of the second display area as an example, as shown in Figure 9B, at time t0, the first size L0 image data at time t0 is displayed; at time t1, the first size L0 image data at time t0 is merged with the second size L0 image data at time t1, so that the second size L0 image data at time t1 covers a local area of ​​the first size L0 image data at time t0; at time t2, the first size L0 image data at time t0 is merged with the second size L0 image data at time t2, so that the second size L0 image data at time t2 covers the first size L0 image data at time t0. A local area in a size L0 image data; at time t3, display the first size L0 image data at time t3; at time t4, merge the first size L0 image data at time t3 with the second size L0 image data at time t4, so that the second size L0 image data at time t4 covers the local area in the first size L0 image data at time t3; at time t5, merge the first size L0 image data at time t3 with the second size L0 image data at time t5, so that the second size L0 image data at time t5 covers the local area in the first size L0 image data at time t3, and so on.

[0136] In this embodiment, the synthesis of intermediate frames is performed by the display end, and the image data at time tn needs to be synthesized using the image data at time tn-1 or tn-2. Using this method, the transmission of redundant images can be reduced, bandwidth can be saved, and bandwidth requirements are lower.

[0137] Figure 10 is a schematic diagram of transmission method 5 of the present disclosure. This method is a single-stream dual-layer transmission method, and the transmission process includes steps 5.1-5.2.

[0138] Step 5.1, for the source end: the source end sends a first data stream to the display end. The first data stream includes a first layer image data (L0) and a second layer image data (L1). The first layer image data is used to provide display information for the corresponding first display area, and the second layer image data is used to provide display information for the corresponding second display area. The first layer image data and the second layer image data have different sizes. The first layer image data and the second layer image data are arranged according to the playback time interval.

[0139] In this example, the source also includes layer data blocks corresponding to the image data in the data stream. These layer data blocks contain single-stream multi-layer transmission description information, specifying that this transmission is a single-stream dual-layer transmission. The source indicates which stream and layer of data is being transmitted in each data stream transmission.

[0140] As shown in Figure 10, the first data stream includes: first-size L0 image data at time t0, second-size L1 image data at time t1, first-size L0 image data at time t2, second-size L1 image data at time t3, and so on. If the second resolution is three times the first resolution, then after transmitting one frame of first-size L0 image data, two frames of second-size L1 image data are transmitted. The first data stream includes: first-size L0 image data at time t0, second-size L1 image data at time t1, second-size L1 image data at time t2, first-size L0 image data at time t3, second-size L1 image data at time t4, second-size L1 image data at time t5, and so on. The first-size L0 image data can be image data merged at the source end. First-layer image data and second-layer image data with the same playback timestamp are merged to form new first-layer image data. As shown in Figure 10, the L0 image data at time t0 and the L1 image data at time t0 are merged to form new L0 image data at time t0. During merging, parts of the L0 image data are replaced with L1 image data.

[0141] Step 5.2, for the display end: The display end receives a first data stream, which includes a first layer of image data and a second layer of image data. The first layer of image data is used to provide display information for a corresponding first display area, and the second layer of image data is used to provide display information for a corresponding second display area. The first layer of image data and the second layer of image data are arranged according to a playback time interval. The display end displays the first layer of image data and the second layer of image data according to the playback time interval of the image data. When displaying the second layer of image data, a portion of the most recently displayed first layer of image data is replaced with the currently displayed second layer of image data. The replaced portion is the second display area, and the remaining portion is the first display area. The image refresh rate of the second display area is higher than that of the first display area.

[0142] Taking the refresh rate of the first display area as half that of the second display area as an example, as shown in Figure 10, at time t0, the L0 image data at time t0 is displayed; at time t1, the L0 image data at time t0 and the L1 image data at time t1 are merged so that the L1 image data at time t1 covers a part of the L0 image data at time t0; at time t2, the L0 image data at time t2 is displayed; at time t3, the L0 image data at time t2 and the L1 image data at time t3 are merged so that the L1 image data at time t3 covers a part of the L0 image data at time t2, and so on.

[0143] To facilitate understanding of the temporal relationship between L0 and L1 image data, L1 image data at time t0 is shown within the L0 image data at time t0. This L0 image data at time t0 can also be understood as a complete image.

[0144] In this embodiment, the synthesis of some data frames is completed by the source end, while the synthesis of other data frames is completed by the display end. Using this method, the source end and the display end can share the processing load. Furthermore, the display end can choose to display L0 image data and L1 image data, or only L0 image data, based on its own needs, making processing more flexible. In addition, this transmission method can reduce the transmission of redundant images, saving bandwidth.

[0145] This embodiment also provides a transmission method 6, which is a multi-stream, multi-layer transmission method. The transmission process includes steps 6.1-6.2.

[0146] Step 6.1, for the source end: the source end sends at least two data streams to the display end, each data stream including a layer of image data, the image data in one of the data streams is used to provide display information for the first display area of ​​the display end, and the image data in the other data streams is used to provide display information for at least one second display area of ​​the display end, and the size of at least one image data used to provide display information for the second display area is different from that of the image data used to provide display information for the first display area.

[0147] In this example, the source also includes layer data blocks corresponding to the image data in the data stream. These layer data blocks contain multi-stream, multi-layer transmission description information, specifying the number of streams and layers in this transmission. The source indicates the current stream and layer number of the data during each data stream transmission.

[0148] Step 6.2, for the display end: The display end receives at least two data streams, each data stream including one layer of image data, the image data in one data stream is used to provide display information for the first display area of ​​the display end, and the image data in the other data streams is used to provide display information for at least one second display area of ​​the display end; the display end displays each layer of image data according to the playback time, and the image refresh rate of the at least one second display area is higher than the image refresh rate of the first display area.

[0149] Specifically, during display: for each refresh of the image data in the first display area, the image data in the second display area is refreshed at least twice. For example: after each refresh of the image data in the first display area, when refreshing the image data in the second display area, a portion of the previous frame's image data is replaced with the image data that provides display information for the second display area.

[0150] By using a multi-stream, multi-layer transmission method, at least two second display areas can have different refresh rates than the first display area. For example, in addition to the first layer of image data corresponding to the entire screen display area, there can be two or more layers of image data. Each layer of image data corresponds to a second display area (local area). The positions of different second display areas can be different. The refresh rates of any two second display areas can be the same or different. At least one second display area has a different refresh rate than the first display area.

[0151] For example, in the above multi-stream multi-layer mode, each data stream may include multi-layer image data, and the transmission method may refer to the single-stream multi-layer transmission mode.

[0152] Based on the above-mentioned various transmission methods, single-stream, dual-stream, or multi-stream transmission methods can be used for multi-view display. Figures 11A and 11B are schematic diagrams of the multi-view display method according to an embodiment of this disclosure. Each stream transmits image data for one viewpoint, which is equivalent to a stream multiplexing transmission method. The transmission process includes steps 7.1-7.2.

[0153] Step 7.1, for the source end: the source end sends at least two data streams to the display end, wherein each data stream is used to provide display information for a user's perspective;

[0154] Step 7.2, for the display end: The display end receives at least two data streams, wherein each data stream is used to provide display information for a user's viewpoint, and the image refresh rates of at least two user viewpoints are different.

[0155] When a single data stream is used to transmit data from a single viewpoint, the transmission methods of at least two data streams can be different, or at least two data streams can be the same. That is, the transmission methods for the two viewpoints can be the same or different. Each viewpoint can employ any of the aforementioned transmission methods. For example, image data from a single viewpoint can be transmitted using a single stream, or image data from a single viewpoint can be transmitted using at least two streams, such as a viewpoint employing the aforementioned dual-stream or multi-stream transmission methods.

[0156] Taking two perspectives as an example, as shown in Figure 11A, the first perspective (view0) uses the above transmission method 3 to transmit L0 image data in a single stream, and the second perspective (view1) still uses the above transmission method 3 to transmit L1 image data in a single stream. However, the image refresh rates of different perspectives can be different. For example, the refresh rate of the display area of ​​the first perspective is twice that of the display area of ​​the second perspective.

[0157] As shown in Figure 11B, the first viewpoint uses the above-mentioned transmission method 4 to transmit L0 image data in a single stream, and the second viewpoint uses the above-mentioned transmission method 3 to transmit L1 image data in a single stream. The image refresh rate of the second viewpoint is the same as the image refresh rate of the first display area (the display area of ​​the first size L0 image data) in the first viewpoint. The refresh rate of the second display area (the display area of ​​the second size L0 image data, the second size is smaller than the first size) in the first viewpoint is higher than the refresh rate of the first display area.

[0158] In other embodiments, there may be other combinations, such as using transmission method 4 for the first viewpoint and transmission method 2 for the second viewpoint, etc., which will not be listed here.

[0159] The transmission process can be found in the previous description and will not be repeated here.

[0160] Although the above descriptions primarily use two display areas as examples, those skilled in the art can implement variable refresh rates for at least three areas based on the above embodiments. For instance, L0 image data provides display data for display area A, L1 image data provides display data for display area B, and L2 image data provides display data for display area C. The image refresh rates of display area B and display area C can be the same or different, and the image refresh rates of display area B and display area C can both be greater than the image refresh rate of display area A.

[0161] In any of the above embodiments, in order to achieve high-definition display of a local area (e.g., the viewing area), the resolution of the image data in different display areas can be different. For example, L0 image data can be low-resolution data, and L1 image data can be high-resolution data.

[0162] After the display end and the source end, under the condition that the bandwidth allows, have comprehensively considered and selected any of the above transmission methods for video stream transmission, the display end displays the image data using the variable partition and variable refresh rate method of this embodiment. The display end can also adjust the display method according to its own situation. As shown in Figure 12, when the display end encounters a problem that affects the display, such as the display end being unable to perform frame alignment due to chip overheating, i.e., the refresh capability is limited, a control data block (as mentioned in Table 9 above) can be sent to request the source end to change the transmission method. For example, the desired transmission method can be identified in the control data block. For example, the Layer Description of bit 03h of the data block can be modified to 0h, indicating that the display end expects to perform single-layer transmission. After parsing the information of data block 03h, the source end changes the original two-layer transmission to single-layer transmission, and sets the value of the Layer Description of bit 03h of the layer data block (as mentioned in Table 8 above) to 0h when transmitting data, indicating that the change is successful.

[0163] In an exemplary embodiment, during conventional single-stream transmission, due to bandwidth limitations and the demand for high frame rates, the remaining bandwidth can be calculated to determine whether a change in transmission mode is necessary. For example, in conventional data stream transmission, DisplayPort Alternative Mode allows for a 50 / 50 bandwidth split for data and video, where DP Alternative Mode is an operating mode that transmits DP signals via a USB Type-C connector. Some USB versions, such as USB4, support USB 3.2 tunnels (for USB data transmission), DisplayPort tunnels (for video and / or audio signals), and PCI-Express tunnels (for computer hardware-related data transmission), enabling intelligent bandwidth management to fully utilize multi-purpose connections. For instance, if a USB4 connection is used to output to a 4K display, but the same connection is also used for data transmission, the host system can allocate the maximum available bandwidth for data transmission without hindering the 4K video stream. When the data transmission volume suddenly increases, some video bandwidth needs to be reduced to make data transmission more efficient. In this case, the transmission mode can be changed to a less bandwidth-intensive mode within the DisplayPort tunnel. For example, if the original display and source terminals used method 1 to transmit image data, where L0 image data was transmitted at 60Hz and L1 image data at 120Hz, changing from transmission method 1 to transmission method 2 would reduce the size of the second stream by about half, correspondingly reducing the hardware resources required for image compositing at the display. In some special cases, such as when the refresh rate of L2 image data increases, the size of the second stream may not change significantly; in this case, although the transmission bandwidth remains the same, the refresh rate has increased.

[0164] The above examples illustrate chip overheating and bandwidth limitation. In other examples, a change in transmission mode may be triggered if any of the following scenarios occur:

[0165] The source detected an action scene, and some of the footage needed to be transmitted at a higher frequency (e.g., 144Hz).

[0166] The display end sends a frame rate increase instruction via control data blocks;

[0167] The display end detects rapid changes and movements by recognizing the user's head movements, and the second stream needs to transmit more information.

[0168] The display board is experiencing a temperature alarm, and resource usage needs to be reduced.

[0169] While the above lists several situations that may trigger a change in transmission mode, it is not exhaustive. This example only demonstrates the method for changing the transmission mode; the reasons for changing the transmission mode are not limited in this article.

[0170] In an exemplary embodiment, in a normal single-person transmission scenario, if it becomes a multi-person scenario, i.e., a second perspective is added, and the first user's first perspective is different from the second user's second perspective, i.e., the gaze range is different, then the transmission method can be changed to increase the stream to support the viewing experience of more people.

[0171] For example, if L0 image data is transmitted at 60Hz and L1 image data is transmitted at 120Hz, and the transmission method is transmission method 1 as described above, then if a new viewer is added, one possible solution is to extend the original stream, i.e., add a new stream for the new viewpoint. In this case, bandwidth can be saved by adjusting the transmission method to reduce the transmission of redundant images. For example, changing the original transmission method 1 to transmission method 2 as described above will reduce the data volume by half due to the elimination of redundancy.

[0172] This disclosure embodiment achieves variable partitioning and variable refresh rate by designing a data transmission method for the display end and the source end. During collaboration, a designated area of ​​the screen can be displayed at a high refresh rate, while the remaining area can be displayed at a low refresh rate. Furthermore, the corresponding indicator area and refresh rate are variable. This disclosure embodiment provides multiple transmission methods, adaptable not only to high-performance source and display ends but also backward compatible with lower-performance source and display ends, demonstrating strong compatibility. This disclosure embodiment also allows for changes in transmission methods. In complex situations, the composition of the stream content can be altered to adapt to different usage scenarios, offering convenience and flexibility. This disclosure embodiment, through low-refresh-rate and high-refresh-rate split transmission, can better allocate transmission traffic, improve transmission efficiency, and reduce transmission lane idle rates. Moreover, due to split transmission, the stream can be carried on different signal lanes, resulting in better anti-interference capabilities and improved transmission performance.

[0173] Furthermore, the variable partitioning and variable refresh rate technology provided in this embodiment greatly helps save bandwidth for high refresh rates. For example, using the VPVRR technology of this embodiment to achieve a dual-lens 4K (4096×2160) display with an 8-bit color depth, the first display area has a refresh rate of 60Hz, and the second display area has a refresh rate of 120Hz. Each of the first and second display areas occupies 50% of the screen, requiring a bandwidth of 71.2 Gbps. Without using DSC (Display Stream Compression) technology, only one DP2.1 cable (maximum effective bandwidth of 77.4Gbps) is needed. However, without VPVRR technology, a dual-lens 4K display with an 8-bit color depth and a 120Hz refresh rate requires 94.9Gbps ​​of bandwidth, and without DSC technology, two DP2.1 cables are needed. Therefore, the solution of this embodiment can save on the use of cables and corresponding interfaces.

[0174] In this embodiment, the information source is, for example, a computer device or a server, and the display end is, for example, various display devices.

[0175] This disclosure also provides an apparatus. The apparatus may include a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the operations performed by the source terminal or the display terminal in this disclosure.

[0176] As shown in Figure 13, in one example, device 100 may include: processor 110, memory 120, bus system 130 and transceiver 140, wherein the processor 110, the memory 120 and the transceiver 140 are connected through the bus system 130, the memory 120 is used to store instructions, and the processor 110 is used to execute the instructions stored in the memory 120 to control the transceiver 140 to send signals.

[0177] It should be understood that processor 110 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0178] Memory 120 may include read-only memory and random access memory, and provides instructions and data to processor 110. A portion of memory 120 may also include non-volatile random access memory. For example, memory 120 may also store device type information.

[0179] In addition to the data bus, bus system 130 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 130 in the figure.

[0180] In implementation, the processing performed by the terminal device can be accomplished through integrated logic circuits in the hardware of the processor 110 or through software instructions. That is, the steps of the method disclosed in this embodiment can be executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other storage media. This storage medium is located in memory 120, and the processor 110 reads information from memory 120 and, in conjunction with its hardware, completes the steps of the aforementioned method. To avoid repetition, further details are omitted here.

[0181] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term "computer storage medium" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0182] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include at least one of those features.

[0183] In the description of this disclosure, "multiple" means at least two, such as two, three, etc., unless otherwise expressly defined.

[0184] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0185] Although embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An image data transmission method, applied to a display end, the method comprising: The display terminal receives at least one data stream, the data stream including at least one layer of image data and a layer data block corresponding to the image data. The layer data block is used to represent the transmission mode of the data stream, and the layer data block includes any one of the following: single-stream multi-layer transmission description information, dual-stream multi-layer transmission description information, and multi-stream multi-layer transmission description information. The display terminal displays the image data in at least two display areas of the display terminal, and the image refresh rates of the at least two display areas are different.

2. The method according to claim 1, wherein, The display terminal displays the image data in at least two display areas on the display terminal, including: the display terminal displays one layer of image data from a data stream in one display area, or the display terminal displays a group of layer image data from one layer of image data in one display area.

3. The method according to claim 1, wherein, The display terminal receives at least one data stream, including: the display terminal receives a first data stream, the first data stream includes a first layer of image data, the first layer of image data includes at least two sets of image data, each set of image data is used to provide display information for a display area, and the at least two sets of image data are arranged according to a playback time interval; The display terminal displays the image data in at least two display areas on the display terminal, including: the display terminal displays the at least two image data in the corresponding display areas according to the playback time of the image data.

4. The method according to claim 1, wherein, The display terminal receives at least one data stream, including: the display terminal receives a first data stream, the first data stream includes a first layer of image data, the first layer of image data includes two sets of image data, the first set of image data is used to provide display information for a first display area, the second set of image data is used to provide display information for a second display area, and a frame of the first set of image data and at least one frame of the second set of image data are arranged according to a playback time interval. The display terminal displays the image data in at least two display areas of the display terminal, including: the display terminal displays each frame of image data in the arrangement order of the first group of image data and the second group of image data; when displaying the second group of image data, a portion of the most recently displayed first group of image data is replaced with the second group of image data to be displayed; the replaced portion is the second display area, and the remaining area is the first display area; the image refresh rate of the second display area is higher than the image refresh rate of the first display area.

5. The method according to claim 1, wherein, The display terminal receives at least one data stream, including: the display terminal receives a first data stream and a second data stream, the first data stream including a first layer of image data, the second data stream including a second layer of image data, wherein the first layer of image data is used to provide display information for a first display area, and the second layer of image data is used to provide display information for a second display area; The display terminal displays the image data in at least two display areas on the display terminal, including: the display terminal displays the first layer of image data and the second layer of image data according to the playback time of the image data, and the image refresh rate of the second display area is higher than the image refresh rate of the first display area.

6. The method according to claim 5, wherein, The display terminal displays the first layer of image data and the second layer of image data according to the playback time of the image data, including: The display terminal merges at least two frames of second-layer image data with one frame of first-layer image data to form the image data to be displayed. Specifically, when merging a frame of second-layer image data with the same playback time with a frame of first-layer image data, the second-layer image data replaces a portion of the first-layer image data during merging to form the image data to be displayed. When merging other frames of second-layer image data, the second-layer image data replaces a portion of the previously displayed image data to form the image data to be displayed; or... The display terminal displays the first layer of image data and the second layer of image data according to the playback time interval of the image data. The display terminal displays a frame of the first layer of image data at the first playback time. The first layer of image data includes the content of the second layer of image data. When displaying the second layer of image data, a part of the image data displayed in the previous frame is replaced with the second layer of image data to be displayed. The replaced part is the second display area, and the remaining area is the first display area.

7. The method according to claim 1, wherein, The display terminal receives at least one data stream, including: the display terminal receives at least two data streams, each data stream including a layer of image data, wherein the image data in one data stream is used to provide display information for a first display area of ​​the display terminal, and the image data in the remaining data streams is used to provide display information for at least one second display area of ​​the display terminal; The display terminal displays the layer image data in at least two display areas of the display terminal, including: the display terminal displays each layer image data according to the playback time, and the image refresh rate of the at least one second display area is higher than the image refresh rate of the first display area.

8. The method according to claim 7, wherein, The display terminal displays each layer of image data according to the playback time, including: refreshing the image data of the first display area once, and refreshing the image data of the second display area at least twice. Among them, after refreshing the image data of the first display area once, when refreshing the image data of the second display area, the image data that provides display information for the second display area is used to replace a part of the previous frame of image data.

9. The method according to any one of claims 1-8, wherein, The display terminal receives at least one data stream, including: the display terminal receives at least two data streams, wherein each data stream is used to provide display information for a user's viewpoint, the image refresh rates of the at least two user viewpoints are different, and the transmission methods of the at least two data streams are the same or different.

10. The method according to claim 1, further comprising: the display end sending a control data block to the source end for requesting the source end to change the transmission mode, and receiving a data stream sent by the source end according to the changed transmission mode, wherein the control data block includes one or more of the following: single-stream multi-layer transmission negotiation information, dual-stream multi-layer transmission negotiation information, and multi-stream multi-layer transmission negotiation information.

11. An image data transmission method, applied at a source end, the method comprising: The source end sends at least one data stream to the display end. The data stream includes at least one layer of image data and a layer data block corresponding to the image data. The layer data block is used to represent the transmission mode of the image data in the data stream. The layer data block includes any one of the following: single-stream multi-layer transmission description information, dual-stream multi-layer transmission description information, and multi-stream multi-layer transmission description information. The image data is used to provide display information for at least two display areas of the display end, and the image refresh rates of the at least two display areas are different.

12. The method according to claim 11, wherein, The source end sends a data stream to the display end, the data stream including at least one layer of image data; or The source end sends two data streams to the display end, each data stream including at least one layer of image data; or The source end sends at least three data streams to the display end, and each data stream includes at least one layer of image data.

13. The method according to claim 11, wherein, The source end sends at least one data stream to the display end, including: the source end sends a first data stream to the display end, the first data stream includes a first layer of image data, the first layer of image data includes at least two sets of image data, each set of image data is used to provide display information for a display area, the at least two sets of image data have different sizes, and the at least two sets of image data are arranged according to the playback time interval.

14. The method according to claim 11, wherein, The source end sends at least one data stream to the display end, including: the source end sends a first data stream to the display end, the first data stream includes a first layer of image data, the first layer of image data includes two sets of image data, the first set of image data is used to provide display information for a first display area, the second set of image data is used to provide display information for a second display area, the first set of image data and the second set of image data have different sizes, and one frame of the first set of image data and at least one frame of the second set of image data are arranged according to the playback time interval.

15. The method according to claim 11, wherein, The source end sends at least one data stream to the display end, including: the source end sends a first data stream and a second data stream to the display end, wherein the first data stream includes a first layer of image data, and the second data stream includes a second layer of image data, wherein the first layer of image data is used to provide display information for a first display area, and the second layer of image data is used to provide display information for a second display area, and the first layer of image data and the second layer of image data have different sizes.

16. The method according to claim 15, further comprising: The source end merges the first-layer image data and the second-layer image data with the same playback time into the first-layer image data in the first data stream. During the merging, the second-layer image data is used to replace a part of the first-layer image data.

17. The method according to claim 11, wherein, The source end sends at least one data stream to the display end, including: the source end sends at least two data streams to the display end, each data stream including a layer of image data, wherein the image data in one data stream is used to provide display information for a first display area of ​​the display end, and the image data in the remaining data streams is used to provide display information for at least one second display area of ​​the display end, wherein at least one image data used to provide display information for the second display area has a different size than the image data used to provide display information for the first display area.

18. The method according to any one of claims 11-17, wherein, The source end sends at least one data stream to the display end, including: the source end sends at least two data streams to the display end, wherein each data stream is used to provide display information for a user's perspective, and the transmission methods of the at least two data streams are the same or different.

19. The method according to claim 11, further comprising: a source receiving a control data block sent by a display, the control data block including a transmission mode requested by the display; the source sending a data stream according to the transmission mode indicated in the control data block; the control data block including one or more of the following: single-stream multi-layer transmission negotiation information, dual-stream multi-layer transmission negotiation information, and multi-stream multi-layer transmission negotiation information.

20. An apparatus comprising a processor and a memory storing a computer program executable on the processor, wherein, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-10 or 11-19.

Images