Latency detection method and apparatus, device, storage medium, and program product

Abstract

The present application provides a latency detection method and apparatus, a device, a storage medium, and a program product. The latency detection method comprises: sending an interaction event carrying a timestamp of a current time point to a master device, such that the master device adds the timestamp carried in the interaction event to frame data of a current frame; receiving the frame data sent by the master device, and calculating a time difference between the timestamp carried in the frame data and a time point at which the frame data is received; and on the basis of a time difference of double-buffered frame data and a time difference of single-buffered frame data, determining a display latency on a display subsystem side of the master device, wherein the single-buffered frame data and the double-buffered frame data are respectively frame data obtained by the master device in a single-buffer mode and a double-buffer mode. The time differences in the single-buffer mode and the double-buffer mode are used to detect display latency with high accuracy, thereby improving the accuracy of MTP compensation and overcoming the problem of dizziness caused by excessively large MTP latency.

Description

Delay detection methods, apparatus, equipment, storage media and program products

[0001] This application claims priority to Chinese Patent Application No. 202411854014.X, filed on December 16, 2024, entitled “Delay Detection Method, Apparatus, Device, Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of multi-device interaction technology, and in particular to a delay detection method, apparatus, device, storage medium, and program product. Background Technology

[0003] MTP (Motion To Photons) latency refers to the time from motion to the display of the corresponding image. Excessive MTP latency can easily cause dizziness. Taking head-mounted displays as an example, MTP latency refers to the time from when the user's head moves to when the image displayed on the screen switches to the corresponding image. To overcome dizziness caused by MTP, it is necessary to detect MTP latency.

[0004] MTP latency consists of multiple delays, one of which is caused by the operating system's display subsystem. In the operating system's display pipeline, after the application layer's image is rendered, it needs to be routed through the display subsystem before being displayed on the screen via the display driver. The latency caused by this routed route varies across different operating systems and even within the same operating system under different settings. Effectively detecting this latency is a problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a delay detection method, apparatus, device, storage medium, and program product, which realizes the calculation of display delay on the display subsystem side by utilizing the time difference between sending interactive events and receiving screen data in single-buffering and double-buffering modes. The detection accuracy is high, the accuracy of MTP compensation is improved, and the problem of dizziness caused by excessive MTP is overcome.

[0006] In a first aspect, this application provides a delay detection method applied to a slave device, the method comprising:

[0007] Send an interactive event carrying a timestamp of the current time to the master device, so that the master device adds the timestamp carried in the interactive event to the image data of the current frame;

[0008] Receive the screen data sent by the main device, and calculate the time difference between the timestamp carried in the screen data and the moment the screen data was received;

[0009] The display latency of the main device display subsystem is determined based on the time difference between double-buffered screen data and single-buffered screen data; wherein, the single-buffered screen data and the double-buffered screen data are the screen data obtained by the main device in single-buffered mode and double-buffered mode, respectively.

[0010] Optionally, the display latency on the main device display subsystem side is determined based on the time difference between double-buffered and single-buffered screen data, including:

[0011] Calculate the average time difference of multiple double-buffered frame data and calculate the average time difference of multiple single-buffered frame data;

[0012] The display delay is determined to be the difference between the average time difference of the double-buffered screen data and the average time difference of the single-buffered screen data.

[0013] Optionally, receiving the screen data sent by the main device and calculating the time difference between the timestamp carried in the screen data and the moment the screen data was received includes:

[0014] Receive multiple screen data sent by the main device;

[0015] For each received frame data, after the magic number verification of the frame data passes, the timestamp in the frame data is read, and the time difference between the read timestamp and the current time is calculated to obtain the time difference of the frame data.

[0016] Optionally, the method further includes:

[0017] When the magic number verification of the screen data fails, the time difference of the obtained screen data is output to calculate the display delay.

[0018] Optionally, the method further includes:

[0019] The periodic fluctuation delay is determined based on the calculated time difference between multiple single-buffered image data and / or the time difference between multiple double-buffered image data.

[0020] Optionally, the method further includes:

[0021] Based on the display delay and the periodic fluctuation delay, delay compensation is performed on the algorithm running by the slave device;

[0022] The algorithm run by the slave device includes at least one of an intermediate frame generation algorithm and a pose prediction algorithm.

[0023] Secondly, this application provides another delay detection method applied to a main device, the method comprising:

[0024] Retrieve interactive events sent from the device;

[0025] Read the timestamp from the interaction event and update the saved timestamp to the read timestamp;

[0026] The currently saved timestamp is added to the current frame data in single-buffered mode and double-buffered mode respectively to obtain single-buffered frame data and double-buffered frame data.

[0027] The single-buffered screen data and the double-buffered screen data are sent to the slave device so that the slave device can determine the display latency of the master device's display subsystem based on the timestamps carried in the single-buffered screen data and the double-buffered screen data.

[0028] Optionally, the currently saved timestamp is added to the current frame's image data in single-buffered mode and double-buffered mode, respectively, to obtain single-buffered image data and double-buffered image data, including:

[0029] In single-buffered mode, multiple consecutive frames are displayed, and the timestamps and magic numbers saved when each frame is displayed are added to the frame data of each frame to obtain multiple single-buffered frame data.

[0030] Switch the mode to double buffering mode. In double buffering mode, display multiple consecutive frames and add the timestamp and magic number saved when each frame is displayed to the frame data of each frame to obtain multiple double buffered frame data.

[0031] The magic number added to the last double-buffered frame data is different from the magic number added to the frame data of other frames.

[0032] Thirdly, this application provides a delay detection device applied to a slave device, the device comprising:

[0033] An interactive event sending module is used to send an interactive event carrying a timestamp of the current time to the main device, so that the main device adds the timestamp carried in the interactive event to the image data of the current frame.

[0034] The time difference calculation module is used to receive the screen data sent by the main device and calculate the time difference between the timestamp carried in the screen data and the moment when the screen data is received.

[0035] The delay determination module is used to determine the display delay of the main device display subsystem based on the time difference between the double-buffered screen data and the time difference between the single-buffered screen data; wherein the single-buffered screen data and the double-buffered screen data are the screen data obtained by the main device in single-buffered mode and double-buffered mode, respectively.

[0036] Fourthly, this application provides another delay detection device for use in a main device, the device comprising:

[0037] The interaction event acquisition module is used to acquire interaction events sent from the device;

[0038] The timestamp update module is used to read the timestamp in the interaction event and update the saved timestamp to the read timestamp.

[0039] The timestamp addition module is used to add the currently saved timestamp to the current frame data generated in single-buffered mode and double-buffered mode respectively, to obtain single-buffered frame data and double-buffered frame data;

[0040] The screen data transmission module is used to send the single-buffered screen data and the double-buffered screen data to the slave device, so that the slave device can determine the display delay on the display subsystem side of the master device based on the timestamps carried in the single-buffered screen data and the double-buffered screen data.

[0041] Fifthly, this application provides an electronic device, including: a processor and a memory, wherein code is stored in the memory, and the processor executes the code stored in the memory to perform the method provided in the first aspect of this application.

[0042] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method provided in the first aspect of this application.

[0043] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in the first aspect of this application.

[0044] The latency detection method, apparatus, device, storage medium, and program product provided in this application realize the detection of display latency using a slave device that interacts with a master device. The slave device continuously sends interactive events carrying timestamps to the master device. After obtaining the image data of the current frame, the master device adds the timestamp of the latest interactive event to the image data and returns the image data carrying timestamps to the slave device. The slave device uses the time difference between the timestamp carried in the image data and the time of receiving the image data in single-buffered mode and double-buffered mode to realize the latency detection on the display subsystem side. The detection accuracy is high. Through latency compensation, the dizziness problem caused by excessive MTP latency is overcome, and the accuracy of the algorithm running in the slave device is improved. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0047] Figure 2 is a flowchart illustrating a delay detection method provided in an embodiment of this application;

[0048] Figure 3 is a data structure diagram of screen data provided in an embodiment of this application;

[0049] Figure 4A is a schematic diagram of the data processing process in single-buffered mode;

[0050] Figure 4B is a schematic diagram of the data processing process in double buffering mode;

[0051] Figure 5 is a flowchart illustrating another delay detection method provided in an embodiment of this application;

[0052] Figure 6 is a schematic diagram of the distribution of the two sets of time differences provided in the embodiments of this application;

[0053] Figure 7 is a flowchart illustrating another delay detection method provided in an embodiment of this application;

[0054] Figure 8 is a schematic diagram of the interaction process between the master device and the slave device provided in an embodiment of this application;

[0055] Figure 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0056] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0058] In multi-device interaction scenarios, the device responsible for the core or complex algorithms can be called the master device, while the device responsible for simple algorithms is called the slave device. The master device is usually a device with an operating system, such as a mobile phone, computer, or wearable device. Wearable devices can be head-mounted displays (HMDs), smart glasses, etc. Slave devices are simpler in structure, and their core components can be chips, such as HID (Human Interface Devices), such as keyboards, mice, game controllers, touchpads, and graphics tablets.

[0059] In scenarios where the display of a master device is controlled through interactive operations on a slave device, display latency measurement and compensation are required to overcome the dizziness caused by MTP.

[0060] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application. Figure 1 uses an HMD as the main device and a controller that interacts with the HMD as the slave device. As shown in Figure 1, after wearing the HMD, the user operates it by holding the controller. Through the control commands of the controller, the user adjusts the display screen of the HMD, such as switching from screen 1 to screen 2. The HMD is used to display virtual reality content or to synthesize virtual reality content with collected real-world images, videos, etc.

[0061] The scenario shown in Figure 1 is only an example. The delay detection method provided in this application can also be applied to scenarios where the master device is a computer and the slave device is a game controller, or where the master device is smart glasses and the slave device is a chip installed on the smart glasses.

[0062] During the interaction between the HMD and the controller, there is a certain time difference between the user's action and the HMD displaying the corresponding screen, also known as MTP latency. When this time difference is large, such as greater than 20ms, it can cause motion sickness and affect the user experience.

[0063] MTP latency typically comprises several components, one of which is due to the buffering mode of the host device's operating system. After the host device's operating system completes image rendering at the application layer, there is an unpredictable latency between the rendering and the display being shown on the screen via the display driver. This latency is caused by the rotation operations of the display subsystem. Different operating systems correspond to different latency levels, and even the same operating system will exhibit different latency levels under different settings. Therefore, a technical solution is needed to measure the latency on the host device's display subsystem side.

[0064] Based on this, this application provides a latency detection method for measuring the latency of the display subsystem of the main device, i.e., the time difference between the end of image rendering and the display (denoted as display latency). Specifically, the method uses the timestamp in the interactive event sent by the device to add a timestamp to the image data of the main device. The display latency is detected by comparing the time of the image data received from the device in single-buffered mode and double-buffered mode with the time difference of the timestamp therein. The method utilizes the different image data generation mechanisms in different buffering modes to achieve latency detection with high accuracy. Through latency compensation, the method effectively reduces MTP latency, overcomes the dizziness problem caused by excessive MTP, and provides a reliable data foundation for the algorithm's operation, thereby improving the accuracy of the algorithm's operation.

[0065] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0066] Figure 2 is a flowchart illustrating a delay detection method provided in an embodiment of this application. This delay detection method is applied to a slave device, which can be an HID device or a chip, such as an external chip of a master device. As shown in Figure 2, the delay detection method provided in this embodiment includes the following steps:

[0067] Step S201: Send an interactive event carrying a timestamp of the current time to the master device, so that the master device adds the timestamp carried in the interactive event to the image data of the current frame.

[0068] The current frame's image data is the image data output by the main device using either single buffer or double buffer mode, and this image data can be displayed on the screen through the display driver.

[0069] The main device is a display device with an operating system, which can be wearable devices, computers, mobile phones, smart TVs, etc.

[0070] The slave device can be an external chip connected to the master device, such as FPGA (Field-Programmable Gate Array), SOC (System on Chip), ASIC (Application-Specific Integrated Circuit), etc.

[0071] During the latency detection phase, the content of the interaction event may only be the timestamp of the current moment in the device clock domain. In the practical application phase, the content of the interaction event is the specific interaction operation.

[0072] For example, the image data can be data corresponding to a pure white or pure black image.

[0073] The slave device can periodically send interactive events carrying the current time's timestamp to the master device. The master device can use a polling method to query the interactive events sent by the slave device. After a new interactive event is found, the timestamp is read and saved so that after the current frame's rendering is completed and the output screen data is obtained, the timestamp is added to the screen data.

[0074] The main device can continuously update the displayed image according to the screen refresh cycle, realizing the rendering and display of multiple consecutive frames. When displaying each frame, the latest timestamp is added to the image data using single-buffering or double-buffering mode to obtain single-buffered image data or double-buffered image data.

[0075] For example, the timestamp can be written into 8 bytes of the frame header of the image data.

[0076] In addition to writing a timestamp, the frame header of the image data can also contain information such as the frame number and magic number.

[0077] For example, Figure 3 is a data structure diagram of screen data provided in an embodiment of this application. As shown in Figure 3, the first 4 bytes (bytes 0 to 3) of the screen data sent to the slave device are used to store a magic number, such as 0x89504E57; the following 8 bytes are used to store the timestamp read from the interaction event; the frame number is written to the 4 bytes after the timestamp. The remaining bytes are used to write the screen data. For example, the screen of odd-numbered frames can be pure white (0xffffff) and the screen of even-numbered frames can be pure black (0x000000).

[0078] The slave device can send interactive events to the master device via USB (Universal Serial Bus). The master device, in turn, sends screen data with timestamps to the slave device via protocols such as DP (DisplayPort) and MIPI (Mobile Industry Processor Interface).

[0079] Specifically, the slave device can be equipped with the function of sending interactive events by configuring the USB Gadget driver from the slave device. By creating a program, the current time stamp can be sent periodically, and the interactive events carrying the timestamp can be transmitted to the master device via USB.

[0080] The master device sends the frame data of each displayed frame back to the slave device. Each frame data carries the slave device's timestamp to indicate when the interactive event was sent.

[0081] Step S202: Receive the screen data sent by the main device, and calculate the time difference between the timestamp carried in the screen data and the time when the screen data is received.

[0082] The device receives frame data from the master device, including frame data in single-buffered mode and double-buffered mode. It reads the timestamps from each received frame data and calculates the time difference between the read timestamp and the moment the frame data was received.

[0083] The master device sequentially sends the frame data, each carrying a timestamp from the slave device, to the slave device according to the display order. Upon receiving the frame data for each frame, the slave device calculates the time difference between the moment it receives the data and the timestamp read from the frame data. This time difference represents the time between the slave device sending the interaction event and the slave device receiving the frame data.

[0084] The video data is divided into single-buffered video data and double-buffered video data according to the different buffering modes. Single-buffered video data is the video data obtained by the master device in single-buffered mode, while double-buffered video data is the video data obtained by the master device in double-buffered mode. The aforementioned time difference needs to be calculated for each type of video data.

[0085] Step S203: Based on the time difference of double-buffered screen data and the time difference of single-buffered screen data, determine the display delay on the main device display subsystem side.

[0086] For ease of description, the time difference calculated using single-buffered screen data is denoted as the single-buffered time difference, and the time difference calculated using double-buffered screen data is denoted as the double-buffered time difference. The difference between the single-buffered time difference and the double-buffered time difference can be used as the display latency on the main device display subsystem side.

[0087] In different buffering modes, the time taken from the master device sending screen data to the slave device receiving that data is the same. Subtracting the time difference calculated in single-buffering mode from the time difference calculated in double-buffering mode removes the latency caused by transmission, while retaining the difference caused by the buffering mode, which is the display latency on the master device's display subsystem side. The display latency on the display subsystem side is the time difference from the end of rendering to the display on the screen.

[0088] Figure 4A illustrates the data processing in single-buffered mode. As shown in Figure 4A, in single-buffered mode, the input signal needs to pass through the rendering engine, compositing engine, and display driver of the data processing unit (DPU) sequentially to achieve the display of one frame and obtain the image data. Figure 4B illustrates the data processing in double-buffered mode. Referring to Figures 4A and 4B, compared to single-buffered mode, double-buffered mode adds a step of display subsystem rotation before display driver, that is, the data buffered in the two memory buffers needs to be exchanged in the display subsystem. Therefore, the time difference obtained by subtracting the single-buffered time difference from the double-buffered time difference is the time difference caused by the rotation of the display subsystem, which is the aforementioned display delay.

[0089] Optionally, the display latency on the main device display subsystem side is determined based on the time difference between double-buffered and single-buffered screen data, including:

[0090] Calculate the average time difference of multiple double-buffered screen data and the average time difference of multiple single-buffered screen data; determine the display delay as the difference between the average time difference of the double-buffered screen data and the average time difference of the single-buffered screen data.

[0091] After obtaining the display latency, it can be combined with other latencies, such as transmission latency, processing latency, and rendering latency, to obtain the MTP latency.

[0092] The latency detection method provided in this embodiment enables the detection of display latency using a slave device that interacts with the master device. The slave device continuously sends interactive events carrying timestamps to the master device. After obtaining the image data of the current frame, the master device adds the timestamp from the latest interactive event to the image data and returns the image data carrying timestamps to the slave device. The slave device then uses the time difference between the timestamp carried in the image data and the time of receiving the image data in single-buffering and double-buffering modes to detect display latency. The detection accuracy is high. Through latency compensation, it overcomes the dizziness problem caused by excessive MTP latency and improves the accuracy of the algorithm running within the slave device.

[0093] Figure 5 is a flowchart of another delay detection method provided in the embodiment of this application. This embodiment further refines steps S202 and S203 based on the embodiment shown in Figure 2, and adds a step of periodic fluctuation delay detection after step S202.

[0094] As shown in Figure 5, the delay detection method provided in this embodiment may specifically include the following steps:

[0095] Step S501: Periodically send an interactive event carrying a timestamp of the current time to the master device, so that the master device adds the timestamp carried in the interactive event to the image data of the current frame.

[0096] Step S502: Receive multiple screen data sent by the master device.

[0097] During the latency test, the master device displays multiple consecutive frames and sequentially sends the frame data of each frame, with timestamps added, to the slave device.

[0098] The master device continuously reads interactive events and timestamps, saving the latest timestamp. The latest timestamp is then added to the screen data of the currently displayed screen.

[0099] At time t1, the master device receives the interaction event, reads the timestamp TS1, and updates the saved timestamp of the slave device to TS1; at time t2 (t2>t1), it obtains the image data F of the i-th frame. i Add timestamp TS1 to the image data F i In the middle; then, at time t3 (t3>t2), the master device receives the interaction event, reads the timestamp TS2, and updates the saved timestamp of the slave device to TS2; at time t4 (t4>t3), the screen data F of the (i+1)th frame is obtained. i+1 Then add the timestamp TS2 to the screen data F. i+1 In the middle, and so on.

[0100] The frequency of interactive events is greater than or equal to the frequency of the main device screen refresh, thus adding different timestamps to the image data of different frames.

[0101] When the device receives screen data, it can trigger an interrupt after the hardware has collected a certain amount of screen data, such as 1 / 3. Then, the software can read the timestamp, magic number, frame number, and other parameters from the screen data collected by the hardware, without having to wait for all screen data to be transmitted, thus improving efficiency.

[0102] Step S503: For each received frame data, after the magic number verification of the frame data passes, read the timestamp in the frame data, and calculate the time difference between the read timestamp and the current time to obtain the time difference of the frame data.

[0103] After receiving a frame data, the magic number and timestamp are read. If the magic number passes the verification and is a default fixed value, the time difference between the timestamp and the current time is calculated to obtain the time difference of the frame data. Similarly, the time difference is calculated for each frame data in multiple single-buffered frame data and multiple double-buffered frame data in the same way, resulting in two sets of time differences.

[0104] If the magic number verification passes, it indicates that the image data is image data that needs to be delayed. Read the timestamp in it, calculate the difference between the current timestamp and the read timestamp, and obtain the time difference of the image data.

[0105] The time difference between each frame of data is saved for subsequent calculations of display latency.

[0106] Furthermore, the image data can also contain frame numbers, and the frame numbers and time differences can be associated and stored.

[0107] Step S504: When the magic number verification of the screen data fails, output the time difference of the obtained screen data to calculate the display delay.

[0108] If the magic number verification of the currently received screen data fails (e.g., the magic number is not equal to the default fixed value), it indicates that screen data acquisition has ended, and there is no need to continue acquiring screen data. Delay detection can be performed based on the existing data. The time difference of the obtained screen data is output locally for calculating display delay.

[0109] The master device sends multiple frame data, including multiple single-buffered frame data and multiple double-buffered frame data. The magic number added to the last frame data is different from the magic numbers in the other frame data to indicate the end of frame data acquisition by the slave device.

[0110] For example, the multiple frame data sent by the master device may include N single-buffered frame data and N+1 double-buffered frame data. The magic number added to the last double-buffered frame data is different from the magic numbers in the other frame data. That is, the last double-buffered frame data will not pass the magic number verification.

[0111] Step S505: Calculate the average time difference of the multiple double-buffered frame data output and the average time difference of the multiple single-buffered frame data output.

[0112] Step S506: Determine that the display delay is the difference between the average time difference of the double-buffered screen data and the average time difference of the single-buffered screen data.

[0113] The output time difference is divided into two sets: one set is the time difference of double-buffered screen data, and the other set is the time difference of single-buffered screen data. The average value of each set of time differences is calculated, and the difference between the average values ​​of the two sets of time differences is determined as the display delay.

[0114] Figure 6 is a schematic diagram of the distribution of the two sets of time differences provided in the embodiment of this application. As shown in Figure 6, the master device first displays N frames (frames 0 to N-1) in single-buffered mode, and sequentially returns the image data of these N frames to the slave device, where N is a positive integer. Then, it switches to double-buffered mode and displays N+1 frames (frames N to 2N), and sequentially returns the image data of these N+1 frames to the slave device. The magic number carried in the image data of the 2Nth frame is invalid, i.e., the verification fails. 2N time differences are calculated from these 2N frames of image data, representing the time differences from frames 0 to 2N-1. The time differences from frames 0 to N-1 form one group, the single-buffered time difference; the time differences from frames N to 2N-1 form another group, the double-buffered time difference. The distribution of the two sets of time differences is plotted with the frame number as the horizontal axis and the time difference as the vertical axis, as shown in Figure 6. Calculate the average time difference between frames 0 and N-1 to obtain the average single-buffer time difference, and calculate the average time difference between frames N and 2N-1 to obtain the average double-buffer time difference. The difference between the average double-buffer time difference and the average single-buffer time difference is the display delay caused by the rotation of the main device's display subsystem.

[0115] Step S507: Determine the periodic fluctuation delay based on the time difference of the multiple single-buffered image data output and / or the time difference of the multiple double-buffered image data output.

[0116] Among them, periodic fluctuation delay is a delay that changes periodically with time or frame number, and is caused by different clock signals.

[0117] Specifically, the periodic fluctuation delay can be determined by using the minimum value among multiple single / double buffered frame data and its corresponding frame number, the difference between the remaining time differences and the minimum value, and the frame numbers corresponding to the remaining time differences.

[0118] Periodic latency fluctuations can be detected for both single-buffered and double-buffered frame data, yielding the periodic latency fluctuations under both buffering modes. Typically, the only difference between the two modes is the minimum value of the periodic latency fluctuations; the overall trend of the fluctuations is the same.

[0119] In addition to detecting display latency caused by the rotation of the display subsystem, the different clock signals used by the slave and master devices cause fluctuations in MTP latency, meaning it varies over time. Therefore, it is necessary to detect latency fluctuations caused by these constant differences. This latency fluctuation exhibits periodicity and can be measured by the time difference between any of the two sets of screen data mentioned above.

[0120] Referring again to Figure 6, the correspondence between the time difference and frame number shown in Figure 6 reveals that the difference in clock signals between the master and slave devices causes fluctuations in the MTP latency (the delay between sending interactive events to and receiving lake surface data) across different frames. Furthermore, due to the periodicity of the clock signal, this fluctuation is also periodic; Figure 6 uses a period of N frames as an example. By measuring this periodic latency fluctuation, appropriate delay compensation can be selected for different frames during the application phase, further improving the accuracy of the compensation.

[0121] Optionally, the method further includes:

[0122] Based on the display latency and the periodic fluctuation latency, latency compensation is performed on the algorithm running on the slave device; the algorithm running on the slave device includes at least one of an intermediate frame generation algorithm and a pose prediction algorithm.

[0123] Attitude prediction algorithms are used to estimate a user's attitude, and attitude interpolation algorithms can be used. Delay compensation can improve the accuracy of the motion data upon which attitude prediction relies, thereby improving the overall accuracy of attitude prediction.

[0124] The pose prediction algorithm requires user pose information P0 at the rendering time, display pose information P1 predicted at the rendering time, and pose information P2 at the screen display time. The prediction of P2 can be compensated by the display delay or periodic fluctuation delay, thereby realizing algorithms such as ATW algorithm and ASW algorithm to reduce MTP latency through P2.

[0125] For example, intermediate frame generation algorithms can be ATW (Asynchronous Time Warp) or ASW (Asynchronous Space Warp). Delay compensation can improve the accuracy of predicting user actions in ATW and ASW algorithms, thereby improving the quality of the generated intermediate frames and ensuring the continuity of the footage.

[0126] In this embodiment, a magic number added to the screen data is used to identify the screen data type and indicate the end of screen data acquisition, simplifying the control logic and achieving fully automatic latency detection. The difference between the average time difference of screen data in multiple double-buffered modes and the average time difference of screen data in multiple single-buffered modes is used to detect latency on the main device display subsystem side, overcoming the impact of latency fluctuations caused by different clock signals and improving the accuracy of latency detection. Furthermore, detection of periodic fluctuation latency is provided, achieving latency compensation for fluctuations, overcoming latency fluctuations caused by clock signals, and improving the accuracy of latency compensation.

[0127] Figure 7 is a flowchart illustrating another delay detection method provided in an embodiment of this application. This delay detection method is applied to a main device, which is a display device with an operating system. As shown in Figure 7, the delay detection method includes the following steps:

[0128] Step S701: Obtain the interaction event sent from the device.

[0129] The master device can use a polling method to query the interaction events of the slave devices.

[0130] Step S702: Read the timestamp in the interaction event and update the saved timestamp to the read timestamp.

[0131] When the master device detects a new interaction event, it reads the timestamp carried in the interaction event and updates the timestamp of the slave device that has been saved to the newly read timestamp.

[0132] Step S703: Add the currently saved timestamp to the current frame's image data in single-buffered mode and double-buffered mode respectively to obtain single-buffered image data and double-buffered image data.

[0133] Specifically, in single-buffered mode, the rendering program in single-buffered mode adds the saved timestamp to the image data of the current frame in single-buffered mode; in double-buffered mode, the rendering program in double-buffered mode adds the saved timestamp to the image data of the current frame in double-buffered mode.

[0134] Optionally, the currently saved timestamp is added to the current frame data generated in single-buffered mode and double-buffered mode, respectively, to obtain single-buffered frame data and double-buffered frame data, including:

[0135] In single-buffered mode, multiple consecutive frames are displayed, and the timestamps and magic numbers saved when each frame is displayed are added to the frame data of each frame, resulting in multiple single-buffered frame data. Switching to double-buffered mode, multiple consecutive frames are displayed, and the timestamps and magic numbers saved when each frame is displayed are added to the frame data of each frame, resulting in multiple double-buffered frame data. The magic number added to the last double-buffered frame data is different from the magic numbers added to the other frame data.

[0136] The main device can also first run in double-buffered mode to obtain multiple double-buffered frame data, and then switch the buffering mode to single-buffered mode to obtain multiple single-buffered frame data. In this case, the last frame data is single-buffered frame data, in which the added magic number is different from the magic number added to the other frame data.

[0137] The magic number added to the last frame data is a magic number that fails the verification, while the magic numbers added to other frame data are magic numbers that can be verified, such as 0x89504E57.

[0138] Step S704: The single-buffered screen data and the double-buffered screen data are sent to the slave device so that the slave device can determine the display delay on the display subsystem side of the master device based on the timestamps carried in the single-buffered screen data and the double-buffered screen data.

[0139] After receiving single-buffered or double-buffered screen data, the data is sent to the slave device. The slave device then calculates the time difference based on the timestamp carried in the single-buffered or double-buffered screen data and the time it was received. Based on the difference between the calculated time difference of the single-buffered and double-buffered screen data, the display latency on the master device's display subsystem side is determined.

[0140] Figure 8 is a schematic diagram of the interaction process between the master device and the slave device provided in an embodiment of this application. As shown in Figure 8, the master device and the slave device are connected via USB and DP routes, respectively. Before latency detection, link stability can be tested, including HID thread testing and image reception testing. The slave device sends continuously increasing values, such as 0, 1, 2, 3, via USB. The master device receives the values ​​via the HID thread and prints them. By observing whether the values ​​are continuous, the stability of the HID thread is tested. The master device sends image data of multiple consecutive frames carrying frame numbers and magic numbers via DP. The slave device verifies the magic number in the received image data. If it passes, it reads the frame number. The continuity of the frame numbers determines whether the image reception test passes. After the link stability test passes, latency detection begins. The slave device periodically sends its current timestamp or interactive events carrying the current timestamp via USB to the master device through a timestamp sending program. The master device uses an HID thread to poll for interactive events or timestamps sent by the slave device. Once found, it updates its saved timestamp to the found timestamp. The master device uses a rendering thread to render the magic number, saved timestamp, and frame number into the first 12 bytes of the first line of the screen data according to the screen refresh time, using single-buffering or double-buffering mode. The master device then transmits the screen data to the slave device via DP. The slave device performs latency detection based on the received screen data. Specifically, for each screen data, when the received screen data reaches a certain amount, the slave device triggers an interrupt, reads the first 12 bytes of the first line of the received screen data, and verifies the magic number in the first 4 bytes of these 12 bytes. If the verification passes, the corresponding screen is the one where latency needs to be detected. The slave device reads the timestamp of the last 8 bytes of the magic number and subtracts it from the slave device's current timestamp to obtain the time difference. Then read the frame number from the last 4 bytes and associate it with the time difference. If the magic number verification fails, it means that the delay detection is over and there is no need to collect subsequent screen data. Output the saved frame number and time difference to the local machine to calculate the display delay and periodic fluctuation delay.

[0141] Frame numbers can start from 0, and are reset to 0 after switching buffer modes. The time difference can then be divided into two groups using the frame number. Calculating the difference between the average values ​​of the two time differences yields the display latency caused by the display subsystem. By analyzing the correspondence between the time difference and the frame number within a single time difference group, the periodic fluctuation latency can be determined.

[0142] Corresponding to the delay detection method for slave devices provided in the foregoing embodiments, this application also provides a delay detection device for slave devices, the delay detection device comprising:

[0143] An interactive event sending module is used to send an interactive event carrying a timestamp of the current time to the main device, so that the main device adds the timestamp carried in the interactive event to the image data of the current frame; a time difference calculation module is used to receive the image data sent by the main device and calculate the time difference between the timestamp carried in the image data and the time when the image data is received; a delay determination module is used to determine the display delay on the display subsystem side of the main device based on the time difference of double-buffered image data and the time difference of single-buffered image data; wherein, the single-buffered image data and the double-buffered image data are image data obtained by the main device in single-buffered mode and double-buffered mode, respectively.

[0144] Optional, a delay determination module, specifically used for:

[0145] Calculate the average time difference of multiple double-buffered screen data and the average time difference of multiple single-buffered screen data; determine the display delay as the difference between the average time difference of the double-buffered screen data and the average time difference of the single-buffered screen data.

[0146] Optional, time difference calculation module, specifically used for:

[0147] The system receives multiple screen data sent by the master device; for each received screen data, after the magic number verification of the screen data passes, it reads the timestamp in the screen data and calculates the time difference between the read timestamp and the current time to obtain the time difference of the screen data.

[0148] Optionally, the delay detection device further includes a time difference output module for:

[0149] When the magic number verification of the screen data fails, the time difference of the obtained screen data is output to calculate the display delay.

[0150] Optionally, the delay detection device further includes a periodic fluctuation delay detection module, used for:

[0151] The periodic fluctuation delay is determined based on the calculated time difference between multiple single-buffered image data and / or the time difference between multiple double-buffered image data.

[0152] Optionally, the delay detection device further includes a delay compensation module for:

[0153] Based on the display latency and the periodic fluctuation latency, latency compensation is performed on the algorithm running on the slave device; the algorithm running on the slave device includes at least one of an intermediate frame generation algorithm and a pose prediction algorithm.

[0154] The delay detection device provided in this application embodiment can be used to execute the technical solution of the delay detection method applied to slave device provided in any of the above embodiments of this application. Its implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0155] Corresponding to the latency detection method for a host device provided in the foregoing embodiments, this application also provides a latency detection device for a host device, the latency detection device comprising:

[0156] An interaction event acquisition module is used to acquire interaction events sent by the slave device; a timestamp update module is used to read the timestamp in the interaction event and update the saved timestamp to the read timestamp; a timestamp addition module is used to add the currently saved timestamp to the current frame image data generated in single-buffered mode and double-buffered mode respectively to obtain single-buffered image data and double-buffered image data; and an image data sending module is used to send the single-buffered image data and the double-buffered image data to the slave device so that the slave device can determine the display latency on the main device display subsystem side based on the timestamps carried in the single-buffered image data and the double-buffered image data.

[0157] Optional, a timestamp addition module, specifically used for:

[0158] In single-buffered mode, multiple consecutive frames are displayed, and the timestamps and magic numbers saved when each frame is displayed are added to the frame data of each frame, resulting in multiple single-buffered frame data. Switching to double-buffered mode, multiple consecutive frames are displayed, and the timestamps and magic numbers saved when each frame is displayed are added to the frame data of each frame, resulting in multiple double-buffered frame data. The magic number added to the last double-buffered frame data is different from the magic numbers added to the other frame data.

[0159] The delay detection device provided in this application embodiment can be used to execute the technical solution of the delay detection method applied to the main device provided in any of the above embodiments of this application. Its implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0160] Figure 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 9, the electronic device provided in this embodiment may include: at least one processor 901 and a memory 902; at least one processor 901 is communicatively connected to the memory 902; wherein, the memory 902 stores code, and at least one processor 901 executes the code stored in the memory 902 to cause the electronic device to perform the method provided in any of the above embodiments.

[0161] Optionally, the memory 902 can be either standalone or integrated with the processor 901.

[0162] The electronic device can be either the aforementioned master device or slave device.

[0163] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0164] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the methods provided in any of the foregoing embodiments can be implemented.

[0165] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in any of the foregoing embodiments.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0167] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0168] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The memory may include high-speed memory, and may also include non-volatile memory, such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0169] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0170] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or a vehicle's sentry-mode control device.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0172] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods provided in the various embodiments of this application.

[0174] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0175] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A delay detection method, characterized by, The method is applied to a slave device, and the method includes: Send an interactive event carrying a timestamp of the current time to the master device, so that the master device adds the timestamp carried in the interactive event to the image data of the current frame; Receive the screen data sent by the main device, and calculate the time difference between the timestamp carried in the screen data and the moment the screen data was received; The display latency of the main device display subsystem is determined based on the time difference between double-buffered screen data and single-buffered screen data; wherein, the single-buffered screen data and the double-buffered screen data are the screen data obtained by the main device in single-buffered mode and double-buffered mode, respectively.

2. The method of claim 1, wherein, The display latency on the main device display subsystem side is determined based on the time difference between double-buffered and single-buffered image data, including: Calculate the average time difference of multiple double-buffered frame data and calculate the average time difference of multiple single-buffered frame data; The display delay is determined to be the difference between the average time difference of the double-buffered screen data and the average time difference of the single-buffered screen data.

3. The method according to claim 1 or 2, characterized in that, Receiving screen data sent by the main device and calculating the time difference between the timestamp carried in the screen data and the moment the screen data was received, including: Receive multiple screen data sent by the main device; For each received frame data, after the magic number verification of the frame data passes, the timestamp in the frame data is read, and the time difference between the read timestamp and the current time is calculated to obtain the time difference of the frame data.

4. The method of claim 3, wherein, The method further includes: When the magic number verification of the screen data fails, the time difference of the obtained screen data is output to calculate the display delay.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The periodic fluctuation delay is determined based on the calculated time difference between multiple single-buffered image data and / or the time difference between multiple double-buffered image data.

6. The method of claim 5, wherein, The method further includes: Based on the display delay and the periodic fluctuation delay, delay compensation is performed on the algorithm running by the slave device; The algorithm run by the slave device includes at least one of an intermediate frame generation algorithm and a pose prediction algorithm.

7. A method of delay detection, characterized by, The method is applied to a master device, and the method includes: Retrieve interactive events sent from the device; Read the timestamp from the interaction event and update the saved timestamp to the read timestamp; The currently saved timestamp is added to the current frame data in single-buffered mode and double-buffered mode respectively to obtain single-buffered frame data and double-buffered frame data. The single-buffered screen data and the double-buffered screen data are sent to the slave device so that the slave device can determine the display latency of the master device's display subsystem based on the timestamps carried in the single-buffered screen data and the double-buffered screen data.

8. The method of claim 7, wherein, The currently saved timestamp is added to the current frame's image data in single-buffered mode and double-buffered mode, respectively, to obtain single-buffered image data and double-buffered image data, including: In single-buffered mode, multiple consecutive frames are displayed, and the timestamps and magic numbers saved when each frame is displayed are added to the frame data of each frame to obtain multiple single-buffered frame data. Switch the mode to double buffering mode. In double buffering mode, display multiple consecutive frames and add the timestamp and magic number saved when each frame is displayed to the frame data of each frame to obtain multiple double buffered frame data. The magic number added to the last double-buffered frame data is different from the magic number added to the frame data of other frames.

9. A delay detection apparatus characterized by comprising: The device is used with a slave device, the device comprising: An interactive event sending module is used to send an interactive event carrying a timestamp of the current time to the main device, so that the main device adds the timestamp carried in the interactive event to the image data of the current frame. The time difference calculation module is used to receive the screen data sent by the main device and calculate the time difference between the timestamp carried in the screen data and the moment when the screen data is received. The delay determination module is used to determine the display delay of the main device display subsystem based on the time difference between the double-buffered screen data and the time difference between the single-buffered screen data; wherein the single-buffered screen data and the double-buffered screen data are the screen data obtained by the main device in single-buffered mode and double-buffered mode, respectively.

10. A delay detection apparatus characterized by comprising: The device is applied to the main equipment, and the device includes: The interaction event acquisition module is used to acquire interaction events sent from the device; The timestamp update module is used to read the timestamp in the interaction event and update the saved timestamp to the read timestamp. The timestamp addition module is used to add the currently saved timestamp to the current frame data generated in single-buffered mode and double-buffered mode, respectively, to obtain single-buffered frame data and double-buffered frame data; The screen data transmission module is used to send the single-buffered screen data and the double-buffered screen data to the slave device, so that the slave device can determine the display delay on the display subsystem side of the master device based on the timestamps carried in the single-buffered screen data and the double-buffered screen data.

11. An electronic device, comprising: include: A processor and a memory, wherein the memory stores code, and the processor executes the code stored in the memory to perform the method as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-8.

13. A computer program product, characterised in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images