Display method, head-mounted display device, computing device, and electronic device

Abstract

The present invention relates to the technical field of extended reality. Disclosed are a display method, a head-mounted display device, a computing device, and an electronic device. The disclosed display method comprises: sending a plurality of pieces of predicted pose data to a computing device, wherein the predicted pose data is determined on the basis of different preset on-screen times; determining the current communication delay with respect to the computing device; on the basis of the communication delay, sending to the computing device a selection instruction corresponding to the predicted pose data, such that the computing device selects, from the predicted pose data and on the basis of the selection instruction, target predicted pose data corresponding to the actual on-screen time, so as to generate target rendering data; and receiving the target rendering data sent by the computing device, and outputting a display picture on the basis of the target rendering data, thereby ensuring that the actual display time of the display picture is consistent with an expected display time, and preventing the display picture from being displayed in advance or with delay, and thus improving a display effect.

Description

Display methods, head-mounted display devices, computing devices, and electronic devices

[0001] This application claims priority to Chinese Patent Application No. 202411549327.4, filed on October 31, 2024, entitled "Display Method, Head-Mounted Display Device, Computing Device and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of extended reality technology, and more particularly to a display method, a head-mounted display device, a computing device, and an electronic device. Background Technology

[0003] With the rapid development of technology, Extended Reality (XR) technology, including Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), is gradually changing people's lifestyles with its unique charm. XR devices that utilize XR technology have display capabilities, and when displaying content, they need to use rendering technology to render the virtual content to generate the displayed image.

[0004] However, there is a time delay in the process of generating and displaying the image in XR devices. If the image is not displayed at the expected actual screen time, it may cause the image to be displayed earlier or later, affecting the display effect. Summary of the Invention

[0005] One objective of this invention is to provide a display method, a head-mounted display device, a computing device, and an electronic device, aiming to solve the technical problem that the process from the generation to the display of the display image in the prior art has a time delay, which causes the display image to be displayed earlier or later, affecting the display effect.

[0006] According to a first aspect of the present invention, a display method is provided, the display method comprising:

[0007] Multiple sets of attitude prediction data are sent to the computing device, and each set of attitude prediction data corresponds to a different preset screen display time.

[0008] Determine the current communication latency with the computing device;

[0009] The current communication delay is used to send the selection instruction corresponding to each of the attitude prediction data to the computing device, so that the computing device selects the target attitude prediction data corresponding to the actual screen time from each of the attitude prediction data based on the selection instruction to generate target rendering data.

[0010] The system receives the target rendering data sent by the computing device and outputs a display screen based on the target rendering data.

[0011] In one embodiment, the step of determining the current communication latency with the computing device includes:

[0012] Receive reference rendering data sent by the computing device based on the reference attitude prediction data in each of the attitude prediction data;

[0013] The actual communication delay is obtained by calculating the difference between the reception time of the reference rendering data and the generation time of the reference pose prediction data.

[0014] The difference between the actual communication delay and the theoretical communication delay corresponding to the preset screen display time is calculated to obtain the current communication delay.

[0015] In one embodiment, the step of sending the selection instructions corresponding to each of the attitude prediction data to the computing device via the current communication delay includes:

[0016] Determine the actual on-screen time corresponding to the baseline rendering data, and extract the baseline preset on-screen time corresponding to the baseline pose prediction data from the baseline rendering data;

[0017] Calculate the difference between the actual screen-on time and the baseline preset screen-on time to obtain the screen-on time difference;

[0018] Based on the time difference between the screen display and the current communication delay, a selection instruction corresponding to each of the attitude prediction data is generated.

[0019] In one embodiment, the step of generating selection instructions corresponding to each of the attitude prediction data based on the screen display time difference and the current communication delay includes:

[0020] Determine whether the time difference between the displayed screen and the actual screen refresh rate is greater than the duration corresponding to the current display refresh rate;

[0021] If so, determine whether the current communication delay is greater than a preset delay threshold;

[0022] If the delay exceeds the preset time threshold, a first selection instruction is generated. The first selection instruction is an instruction to select the first target attitude prediction data with the longest preset on-screen time from multiple sets of attitude prediction data.

[0023] If the delay is less than the preset time threshold, a second selection instruction is generated. The second selection instruction is an instruction to select a second target attitude prediction data from multiple sets of attitude prediction data that is greater than the baseline preset screen time and less than the longest preset screen time.

[0024] In one embodiment, after the step of determining whether the time difference between the displayed screens is greater than the duration corresponding to the current display refresh rate, the method further includes:

[0025] If not, then determine whether the time difference between the screen displays is greater than zero;

[0026] If the value is greater than zero, a third selection instruction is generated, which is an instruction to select reference attitude prediction data from multiple sets of attitude prediction times.

[0027] If it is less than zero, a selection instruction is generated based on the absolute value of the time difference between the screen displays.

[0028] In one embodiment, the step of generating a selection instruction based on the absolute value of the on-screen time difference if the difference is less than zero includes:

[0029] If it is less than zero, then determine whether the absolute value of the time difference between the screen displays is greater than the absolute value of the current screen refresh rate;

[0030] If so, a fourth selection instruction is generated, which is an instruction to select the third target attitude prediction data with the shortest preset on-screen time from multiple sets of attitude prediction data.

[0031] If not, a fifth selection instruction is generated, which is an instruction to select a fourth target attitude prediction data from multiple sets of attitude prediction data that is greater than the shortest preset screen time and less than the benchmark preset screen time.

[0032] In one embodiment, before the step of sending multiple sets of attitude prediction data to the computing device, the method further includes:

[0033] Send initial attitude prediction data to the computing device;

[0034] Receive initial rendering data sent by the computing device based on the initial pose prediction data;

[0035] Calculate the difference between the reception time of the initial rendering data and the generation time of the initial pose prediction data to obtain the current theoretical communication delay;

[0036] Multiple different preset screen display times are determined based on the current theoretical communication delay;

[0037] Add the preset on-screen times to the corresponding attitude prediction data.

[0038] In one embodiment, the step of determining multiple different preset screen-on times based on the current theoretical communication delay includes:

[0039] The target time indicator is determined based on the current theoretical communication latency and the current display refresh rate.

[0040] The target time index is updated by using preset parameters to obtain multiple updated time indices;

[0041] Different preset on-screen times are determined based on the current display refresh rate and multiple updated time metrics.

[0042] In one embodiment, the step of determining the target time index based on the current theoretical communication delay includes:

[0043] Obtain a preset deep learning model, which is a long short-term memory network model obtained by training based on training data consisting of parameters corresponding to multiple theoretical communication delays and multiple time indicators.

[0044] The initial time index is obtained based on the current theoretical communication latency using the preset deep learning model;

[0045] The initial time index is rounded to obtain the target time index.

[0046] Furthermore, to achieve the above objectives, the present invention also proposes a display method, the display method comprising:

[0047] Receive multiple sets of attitude prediction data sent by the head-mounted display device;

[0048] Receive the selection command sent by the head-mounted display device based on the current communication delay;

[0049] Based on the selection instruction, target pose prediction data corresponding to the actual screen time is selected from each of the pose prediction data, and target rendering data is generated based on the target pose prediction data;

[0050] The target rendering data is sent to the head-mounted display device so that the head-mounted display device outputs a display image based on the target rendering data.

[0051] Furthermore, to achieve the above objectives, the present invention also proposes a head-mounted display device, the head-mounted display device comprising:

[0052] The attitude prediction module is used to send multiple sets of attitude prediction data to the computing device, and each set of attitude prediction data is determined based on a different preset screen display time;

[0053] A latency determination module is used to determine the current communication latency with the computing device;

[0054] The time analysis module is used to send selection instructions corresponding to each of the attitude prediction data to the computing device through the current communication delay, so that the computing device selects target attitude prediction data corresponding to the actual screen time from each of the attitude prediction data based on the selection instructions to generate target rendering data;

[0055] The display module is used to receive the target rendering data sent by the computing device and output a display screen based on the target rendering data.

[0056] Furthermore, to achieve the above objectives, the present invention also proposes a computing device, the computing device comprising:

[0057] The data receiving module is used to receive multiple sets of attitude prediction data sent by the head-mounted display device;

[0058] The data receiving module is also used to receive the selection command sent by the head-mounted display device based on the current communication delay;

[0059] The data selection module is used to select target pose prediction data corresponding to the actual screen time from each of the pose prediction data based on the selection instruction, and generate target rendering data based on the target pose prediction data.

[0060] The rendering module is used to send the target rendering data to the head-mounted display device so that the head-mounted display device outputs a display image based on the target rendering data.

[0061] Furthermore, to achieve the above objectives, the present invention also proposes an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the display method as described above.

[0062] In addition, to achieve the above objectives, the present invention also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the display method described above.

[0063] One or more technical solutions proposed in this invention have at least the following technical effects:

[0064] This invention sends multiple sets of attitude prediction data to a computing device, each set determined based on a different preset screen-on time. It determines the current communication delay with the computing device, sends selection instructions corresponding to each attitude prediction data set to the computing device based on the current communication delay, and enables the computing device to select target attitude prediction data corresponding to the actual screen-on time from the various attitude prediction data sets based on the selection instructions to generate target rendering data. It then receives the target rendering data sent by the computing device and outputs a display screen based on the target rendering data. Because this invention's head-mounted display device sends multiple sets of attitude prediction data corresponding to different preset screen-on times to the computing device, and simultaneously controls the computing device to select target attitude prediction data corresponding to the actual screen-on time to generate target rendering data according to the selection instructions corresponding to the current communication delay, it can ensure that the display screen is displayed at the expected actual screen-on time, avoiding premature or delayed display, thereby improving the display effect.

[0065] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0067] Figure 1 is a flowchart illustrating the first embodiment of the display method of the present invention;

[0068] Figure 2 is a schematic diagram of the basic architecture of the preset deep learning model in the first embodiment of the display method of the present invention;

[0069] Figure 3 is a flowchart illustrating the second embodiment of the display method of the present invention;

[0070] Figure 4 is a flowchart illustrating the third embodiment of the display method of the present invention;

[0071] Figure 5 is a flowchart illustrating the fourth embodiment of the display method of the present invention;

[0072] Figure 6 is an example diagram of the modular structure of the split AR device of the present invention;

[0073] Figure 7 is a schematic diagram of the module structure of the head-mounted display device of the present invention;

[0074] Figure 8 is a schematic diagram of the module structure of the computing device of the present invention;

[0075] Figure 9 is a schematic diagram of the hardware operating environment involved in the display method in an embodiment of the present invention.

[0076] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0079] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0080] The main solution of this invention is as follows: a head-mounted display device sends multiple sets of attitude prediction data to a computing device, each attitude prediction data being determined based on a different preset screen-on time; the current communication delay between the head-mounted display device and the computing device is determined; a selection instruction corresponding to each attitude prediction data is sent to the computing device based on the current communication delay, so that the computing device selects target attitude prediction data corresponding to the actual screen-on time from each attitude prediction data based on the selection instruction to generate target rendering data; the target rendering data sent by the computing device is received, and a display screen is output based on the target rendering data.

[0081] In this embodiment, for ease of description, the following description uses a head-mounted display device as the execution subject.

[0082] Existing XR devices suffer from a time delay between the generation and display of the image. When the image does not appear at the expected actual display time, it may appear earlier or later, affecting the display effect. For example, if the display time obtained using attitude prediction data is 10 seconds without delay, and the optimal display time to reduce the impact of delay is 20 seconds, then using the existing method, the image may appear 10 seconds earlier, resulting in a poor display effect.

[0083] This invention provides a solution in which a head-mounted display device sends multiple sets of posture prediction data corresponding to different preset screen-on times to a computing device. At the same time, based on the selection command corresponding to the current communication latency, the computing device is controlled to select target posture prediction data corresponding to the actual screen-on time to generate target rendering data. This ensures that the display screen is displayed at the expected actual screen-on time, avoids the display screen from being displayed early or late, and thus improves the display effect.

[0084] Based on this, the present invention provides a display method. Referring to FIG1, FIG1 is a flowchart of the first embodiment of the display method of the present invention.

[0085] In this embodiment, the display method includes steps S10 to S40:

[0086] Step S10: Send multiple sets of attitude prediction data to the computing device, with each set of attitude prediction data corresponding to a different preset screen display time.

[0087] It should be noted that the executing entity in this embodiment can be a head-mounted display device with screen display, network communication, and program execution functions that applies XR technology, such as an AR device, VR device, MR device, etc., or an XR electronic device capable of achieving the above functions. The following description uses an AR device as an example, specifically a head-mounted display device within an AR device, to illustrate this embodiment and the subsequent embodiments.

[0088] This split-type AR device also includes a computing device with rendering capabilities, which utilizes rendering technology. The head-mounted display and the computing device form a split structure. The head-mounted display and the computing device can interact with each other. The computing device uses the data sent by the head-mounted display to render and generate rendering data. The head-mounted display can then correct the rendering data sent by the computing device and output the displayed image on the screen.

[0089] Understandably, the aforementioned attitude prediction data can be data characterizing the position and orientation of the head-mounted display device in three-dimensional space. Specifically, the attitude prediction data involves the accurate position of the head-mounted display device relative to its surrounding environment, as well as attitude changes such as tilt, rotation, and yaw of the head-mounted display device.

[0090] It should be noted that the above-mentioned preset on-screen time can be the time when the display image is finally displayed on the screen of the head-mounted display device under zero latency. That is, when there is no delay factor, the display image should be displayed on the display interface when the preset on-screen time is reached.

[0091] In practical implementation, when a user uses a split AR device consisting of a head-mounted display and a computing device, the head-mounted display can establish a communication connection with the computing device. The head-mounted display can track and determine its attitude in space through built-in sensors, such as an inertial measurement unit (IMU), generating attitude prediction data. The sensors involved can measure the acceleration and angular velocity of the head-mounted display, and then the data is processed by algorithms to estimate the position and attitude of the head-mounted display, thereby generating attitude prediction data. Simultaneously, a field for a preset display time can be appended to the attitude prediction data to ensure that the head-mounted display can accurately overlay the displayed image onto the user's real-world field of view when the preset display time is reached. The head-mounted display can acquire attitude prediction data at fixed intervals (e.g., 1 second or 5 seconds). After acquiring multiple sets of attitude prediction data, it can send multiple sets of attitude prediction data, including the preset display time, to the computing device. Different preset display times are added to different attitude prediction data sets.

[0092] In one feasible implementation, steps S01 to S05 may be included before step S10:

[0093] Step S01: Send initial attitude prediction data to the computing device.

[0094] It should be noted that the aforementioned initial attitude prediction data can be used to initialize the head-mounted display device and the computing device, and to detect whether the communication connection between the two has been successfully established.

[0095] In a specific implementation, the aforementioned head-mounted display device can generate initial attitude prediction data of the space it is in at the time of initialization, record the generation time of the initial attitude prediction data, and then add the generation time to the initial attitude prediction data before sending it to the computing device.

[0096] Step S02: Receive initial rendering data sent by the computing device based on the initial pose prediction data.

[0097] In a practical implementation, the aforementioned computing device can use the initial pose prediction data to render a virtual image, obtaining initial rendering data. This initial rendering data may include the generation time of the initial pose prediction data. Once the communication connection between the computing device and the head-mounted display device is successfully established and no anomalies occur, the computing device can feed back the initial rendering data to the head-mounted display device.

[0098] Step S03: Calculate the difference between the reception time of the initial rendering data and the generation time of the initial attitude prediction data to obtain the current theoretical communication delay.

[0099] In a specific implementation, after receiving the initial rendering data, the head-mounted display device can record the reception time of the initial rendering data, extract the generation time of the initial attitude prediction data from the initial rendering data, and then calculate the difference between the reception time of the initial rendering data and the generation time of the initial attitude prediction data to obtain the current theoretical communication latency when the network does not experience fluctuations or other delay factors at the current moment.

[0100] Step S04: Determine multiple different preset screen display times based on the current theoretical communication delay.

[0101] In practical implementation, the aforementioned head-mounted display device can allocate multiple different preset screen display times based on the calculated current theoretical communication latency and the number of vertical synchronization cycles. The vertical synchronization cycle can be the time interval between two consecutive refreshes of the head-mounted display device's screen, and can be the reciprocal of the screen's refresh rate. For example, on a 60Hz screen, the vertical synchronization cycle can be 1s / 60, or 16.67ms.

[0102] In one feasible implementation, step S04 may include steps S041 to S043:

[0103] Step S041: Determine the target time index based on the current theoretical communication delay.

[0104] In practical implementation, the aforementioned head-mounted display device can obtain the current vertical synchronization cycle by taking the reciprocal of the current display refresh rate, and then use the following formula to calculate the time index:

[0105] In the formula, N is the target time index, is the current theoretical communication delay, and is the vertical synchronization period.

[0106] It should be understood that the head-mounted display device can send initial attitude prediction data multiple times to repeat the above process, obtain multiple time indicators, and then calculate the average value to improve the accuracy of the time indicator determination.

[0107] In one feasible implementation, step S041 may include steps S0411 to S0413:

[0108] Step S0411: Obtain a preset deep learning model. The preset deep learning model is a long short-term memory network model obtained by training on training data consisting of multiple parameters corresponding to theoretical communication delays and multiple time indicators.

[0109] In the specific implementation, referring to Figure 2, which is a schematic diagram of the basic architecture of the preset deep learning model in the first embodiment of the display method of the present invention, Xt is the input of frame t, ht is the output of frame t, Xt-1 is the input of frame t-1, ht-1 is the output of frame t-1, Xt+1 is the input of frame t+1, and ht+1 is the output of frame t+1. Taking frame t as an example, the forget gate of the Long Short-Term Memory (LSTM) model includes a function on the left side. The function calculates the forget gate value by considering the input Xt of frame t and the hidden state of the previous time step. This value is between 0 and 1, where 0 represents complete forgetting and 1 represents complete retention. The input gate of the LSM model includes a function on the middle side and a tanh layer. The function controls the information in the input Xt of frame t that needs to be updated to the memory cell, and the tanh layer creates a new candidate value vector, which may be added to the memory cell. The output gate of the Long Short-Term Memory (LSTM) model consists of the function on the right and a tanh layer. The output gate value is calculated by taking the input Xt of frame t and the hidden state from the previous time step. This value is between 0 and 1, where 0 represents no output and 1 represents full output. Then, the cell state is processed by the tanh function, outputting a value between -1 and 1. This result is multiplied by the output gate value to obtain the final output ht of frame t. The descriptions of frames t-1 and t+1 can be found in the description of frame t above, and will not be repeated here.

[0110] It should be understood that when constructing a pre-defined deep learning model based on a long short-term memory network model, the parameter corresponding to the theoretical communication latency in the training data can be used as the input for each frame, and the time index can be used as the output for each frame. The parameter corresponding to the theoretical communication latency can be the difference between the reception time of the initial rendering data and the generation time of the corresponding initial pose prediction data.

[0111] Step S0412: Obtain the initial time index based on the current theoretical communication latency using the preset deep learning model.

[0112] In a specific implementation, the aforementioned head-mounted display device can use the parameters corresponding to the determined current theoretical communication delay as input to a preset deep learning model, and receive the initial time index output by the preset deep learning model based on the parameters corresponding to the current theoretical communication delay.

[0113] Step S0413: Round the initial time index to obtain the target time index.

[0114] In the specific implementation, the target time index is used to characterize the number of vertical synchronization cycles. The target time index should be an integer. Therefore, after the head-mounted display device obtains the initial time index output by the preset deep learning model, it can round the initial time index to obtain the target time index.

[0115] Step S042: Update the time index using preset parameters to obtain multiple updated time indices.

[0116] It should be noted that the above preset parameters can be pre-set according to requirements.

[0117] In a specific implementation, the above-mentioned preset parameters can be pre-configured in the head-mounted display device. After the head-mounted display device determines the time index, it can use the above-mentioned preset parameters to update the time index. For example, if the target time index is N, when the preset parameter is "-1", the updated time index is N-1.

[0118] Step S043: Determine different preset on-screen times based on the current display refresh rate and multiple updated time indicators.

[0119] In its implementation, the aforementioned head-mounted display device determines the current vertical synchronization cycle based on the current screen refresh rate. It then uses the updated time index as the target number and adds the next vertical synchronization cycle to the target number of current vertical synchronization cycles. The resulting time is the preset on-screen time. Due to network fluctuations, varying degrees of latency can occur; therefore, multiple preset parameters can be set to reduce the impact of latency.

[0120] For example, taking the time index as N, the vertical synchronization period as vsync, and the preset parameters including "-2", "-1", "0", "+1", and "+2" as an example, the updated time index obtained based on the above method is "N-2", "N-1", "N", "N+1", and "N+2". The resulting preset screen display times are: "next vsync + N-2 vsync", "next vsync + N-1 vsync", "next vsync + N vsync", "next vsync + N+1 vsync", and "next vsync + N+2 vsync". Each of the resulting preset screen display times can cover multiple durations before and after the vertical synchronization period to reduce latency caused by network fluctuations.

[0121] Step S05: Add the preset screen display time to the corresponding attitude prediction data.

[0122] In the specific implementation, after determining each preset screen time, each preset screen time can be added to the corresponding attitude prediction data, and then the attitude prediction data with the preset screen time added can be sent to the computing device.

[0123] Step S20: Determine the current communication latency with the computing device.

[0124] In a specific implementation, during initialization, the head-mounted display device can send arbitrary data to the computing device and record the time of data transmission. Then, after receiving the response data from the computing device based on the data, the head-mounted display device records the time of data reception and determines the current communication delay based on the time difference between the transmission and reception times.

[0125] Step S30: Send selection instructions corresponding to each of the attitude prediction data to the computing device through the current communication delay, so that the computing device selects target attitude prediction data corresponding to the actual screen time from each of the attitude prediction data based on the selection instructions to generate target rendering data.

[0126] In its specific implementation, the aforementioned head-mounted display device can compare the current communication latency with the vertical synchronization period, and determine the target screen time from each preset screen time based on the magnitude of the deviation of the current communication latency from the vertical synchronization period. This ensures that the display screen can compensate for the current communication latency when it is displayed at the target screen time. At the same time, it can generate a selection instruction for selecting target attitude prediction data from each attitude prediction data. The target attitude prediction data can be the attitude prediction data that includes the target screen time.

[0127] Furthermore, after determining the selection command, the aforementioned head-mounted display device can send the selection command to the computing device. After receiving multiple sets of attitude prediction data and the selection command, the computing device can select the target attitude prediction data containing the target on-screen time from the multiple sets of attitude prediction data through the selection command. Since the target on-screen time can compensate for the current delay, it can be determined that the target on-screen time corresponds to the actual on-screen time, which can ensure that when the display screen is displayed, the target on-screen time is consistent with or close to the actual on-screen time.

[0128] Step S40: Receive the target rendering data sent by the computing device, and output a display screen based on the target rendering data.

[0129] In its implementation, after selecting target pose prediction data, the aforementioned computing device can use this data to render a virtual image. The rendering process includes calculating the correct position, size, and orientation of the virtual image within the user's field of vision using the target pose prediction data. After the calculation is completed, target rendering data is generated and sent to the head-mounted display device. The head-mounted display device can then further compensate and correct the target rendering data based on the target pose prediction data to form the final display image. When the target on-screen time is reached, the display image is output on the display interface for the user to view.

[0130] The display method provided in this embodiment sends multiple sets of attitude prediction data to a computing device, each set of attitude prediction data being determined based on a different preset screen-on time; determines the current communication delay with the computing device; sends selection instructions corresponding to each attitude prediction data to the computing device based on the current communication delay, so that the computing device selects target attitude prediction data corresponding to the actual screen-on time from the attitude prediction data based on the selection instructions to generate target rendering data; receives the target rendering data sent by the computing device, and outputs the display screen based on the target rendering data. Because the head-mounted display device in this embodiment sends multiple sets of attitude prediction data corresponding to different preset screen-on times to the computing device, and simultaneously controls the computing device to select target attitude prediction data corresponding to the actual screen-on time to generate target rendering data according to the selection instructions corresponding to the current communication delay, it can ensure that the display screen is displayed at the expected actual screen-on time, avoiding the display screen from being displayed early or late, thereby improving the display effect.

[0131] Based on the first embodiment of the present invention, a second embodiment of the present invention is proposed. In the second embodiment, content that is the same as or similar to that in the first embodiment can be referred to the above description and will not be repeated hereafter. Based on this, please refer to Figure 3, which is a flowchart illustrating the second embodiment of the method of the present invention.

[0132] In this embodiment, step S20 may include steps S201 to S203:

[0133] Step S201: Receive reference rendering data sent by the computing device based on the reference attitude prediction data in each of the attitude prediction data.

[0134] It should be noted that the aforementioned baseline attitude prediction data can be the default attitude prediction data pre-selected from multiple sets of attitude prediction data. For example, the baseline attitude prediction data can be the attitude prediction data with a preset on-screen time of the next vsync + N vsyncs.

[0135] In a specific implementation, when the head-mounted display device does not send a selection command to the computing device, the computing device, after receiving multiple sets of attitude prediction data, will render the reference attitude prediction data from the multiple sets of attitude prediction data and feed back the rendered reference rendering data to the head-mounted display device.

[0136] Step S202: Calculate the difference between the reception time of the reference rendering data and the generation time of the reference attitude prediction data to obtain the actual communication delay.

[0137] In a specific implementation, the aforementioned head-mounted display device can record the generation time of the reference attitude prediction data and, upon receiving the reference rendering data, record the reception time of the reference rendering data. Then, it calculates the difference between the reception time of the reference rendering data and the generation time of the reference attitude prediction data to obtain the actual communication delay.

[0138] Step S203: Calculate the difference between the actual communication delay and the theoretical communication delay corresponding to the preset screen display time to obtain the current communication delay.

[0139] In practical implementation, the actual communication latency mentioned above can be the actual value affected by network fluctuations in a real-world scenario, while the theoretical communication latency mentioned above is the theoretical value when there are no network fluctuations. Therefore, to make the latency more accurate, the difference between the actual communication latency and the theoretical communication latency corresponding to the preset screen display time can be calculated to obtain a more precise current communication latency.

[0140] This embodiment receives reference rendering data sent by a computing device based on reference attitude prediction data from various attitude prediction data sets; calculates the difference between the reception time of the reference rendering data and the generation time of the reference attitude prediction data to obtain the actual communication latency; and calculates the difference between the actual communication latency and the theoretical communication latency corresponding to the preset on-screen time to obtain the current communication latency. Since the current communication latency is determined by the difference between the actual communication latency considering network fluctuations and the theoretical communication latency without network fluctuations, the obtained current communication latency accurately reflects network fluctuations, improving the selection accuracy of subsequent attitude prediction data.

[0141] Based on the second embodiment of the present invention, a third embodiment of the present invention is proposed. In the third embodiment of the present invention, the contents that are the same as or similar to those in the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, please refer to Figure 4, which is a flowchart illustrating the third embodiment of the method of the present invention.

[0142] In this embodiment, the step of sending the selection instructions corresponding to each attitude prediction data to the computing device through the current communication delay includes steps S301 to S303:

[0143] Step S301: Determine the actual screen time corresponding to the reference rendering data, and extract the reference preset screen time corresponding to the reference pose prediction data from the reference rendering data.

[0144] It should be noted that the above-mentioned actual screen-on time represents the optimal time to avoid screen delays in the event of network fluctuations. In other words, the head-mounted display device can minimize the impact of network fluctuations when displaying the image during the actual screen-on time, thus providing a better user experience.

[0145] In its implementation, the aforementioned head-mounted display device can employ Asynchronous Timewarp (ATW) technology to calculate the total time required for the image to be generated and displayed on the screen, i.e., the actual on-screen time. This actual on-screen time includes rendering time, data transmission time, and screen refresh time. Network fluctuations can increase transmission latency, leading to a longer actual on-screen time. The head-mounted display device can compensate for the time difference during rendering by asynchronously generating intermediate frames, thus reducing the perceived latency when the image is displayed at the actual on-screen time.

[0146] It should be understood that the aforementioned preset on-screen time is the preset on-screen time carried in the baseline attitude prediction data. After the computing device renders the baseline attitude prediction data to generate baseline rendering data, it can extract the preset on-screen time from the baseline attitude prediction data, add this preset on-screen time to the baseline rendering data, and then send it to the head-mounted display device. After receiving the baseline rendering data, the head-mounted display device can extract the preset on-screen time from the baseline rendering data.

[0147] Step S302: Calculate the difference between the actual screen time and the baseline preset screen time to obtain the screen time difference.

[0148] In its implementation, the aforementioned head-mounted display device can calculate the difference between the actual screen-on time and the preset screen-on time, which is the screen-on time difference. When the screen-on time difference is not zero, it is determined that there are network fluctuations or other latency factors affecting the device. Conversely, when the screen-on time difference is zero, meaning the actual screen-on time matches the preset screen-on time, it is determined that there are no network fluctuations or other latency factors affecting the device.

[0149] Step S303: Generate selection instructions corresponding to each of the attitude prediction data based on the screen display time difference and the current communication delay.

[0150] In the specific implementation, since the preset screen display time is different in each group of attitude prediction data, the optimal screen display time to reduce the current communication latency can be determined according to the range of the screen display time difference, and a selection instruction for selecting the attitude prediction data corresponding to the optimal screen display time can be generated.

[0151] In one feasible implementation, step S303 includes steps S3031 to S3034:

[0152] Step S3031: Determine whether the time difference between the displayed screen and the refresh rate is greater than the duration corresponding to the current display screen refresh rate.

[0153] It should be noted that the duration corresponding to the current display refresh rate is the vertical synchronization cycle.

[0154] In its implementation, the aforementioned head-mounted display device can determine the corresponding duration, i.e., the vertical synchronization period, by calculating the reciprocal of the current display refresh rate. The head-mounted display device can then compare the screen-on time difference with the duration corresponding to the current display refresh rate to determine whether the screen-on time difference is greater than the duration corresponding to the current display refresh rate, thereby confirming whether the current communication latency caused by network fluctuations exceeds the vertical synchronization period.

[0155] Step S3032: If yes, then determine whether the current communication delay is greater than the preset delay threshold.

[0156] It should be noted that the aforementioned preset latency threshold can be a pre-set threshold used to determine that the current network communication latency is too high.

[0157] In a specific implementation, when the head-mounted display device detects that the time difference between the screen display and the current screen refresh rate is greater than the duration corresponding to the current screen refresh rate, it can determine that the current communication latency caused by the current network fluctuation exceeds the vertical synchronization cycle. At this time, the current communication latency can be further compared with a preset latency threshold to determine whether the current communication latency is greater than the preset latency threshold.

[0158] Step S3033: If the delay is greater than the preset time threshold, a first selection instruction is generated. The first selection instruction is an instruction to select the first target attitude prediction data with the longest preset on-screen time from multiple sets of attitude prediction data.

[0159] In its implementation, when the head-mounted display device detects that the time difference between the displayed screen time and the refresh rate of the current screen is greater than the duration corresponding to the current display refresh rate, and the current communication latency is greater than a preset latency threshold, it can determine that the current communication latency is large. At this point, a first selection command can be generated. The head-mounted display device sends the first selection command to the computing device, which can then select the first target posture prediction data with the longest preset display time from multiple sets of posture prediction data for rendering, thereby reducing the impact of the current communication latency. The longest preset display time can be the maximum value among the preset display times.

[0160] Step S3034: If the delay is less than the preset time threshold, a second selection instruction is generated. The second selection instruction is an instruction to select a second target attitude prediction data from multiple sets of attitude prediction data that is greater than the baseline preset screen time and less than the longest preset screen time.

[0161] In its implementation, when the head-mounted display device detects that the time difference between the displayed screen and the actual screen refresh rate is greater than the duration corresponding to the current display refresh rate, and the current communication latency is less than a preset latency threshold, it can determine that the current communication latency is relatively low and generate a second selection instruction. The head-mounted display device sends the second selection instruction to the computing device, which can then select, based on the second selection instruction, a second target attitude prediction data set from multiple sets of attitude prediction data. This second target attitude prediction data set must have a preset display time greater than the baseline attitude prediction data and a maximum preset display time less than the longest preset display time for rendering. This reduces the impact of the current communication latency while avoiding delayed screen display.

[0162] In one possible implementation, after step S3031, step A3032 is further included:

[0163] Step A3032: If not, determine whether the time difference between the screen displays is greater than zero.

[0164] In a specific implementation, when the head-mounted display device detects that the time difference between the screen display and the current display refresh rate is less than or equal to the duration corresponding to the current display refresh rate, it can determine that the current communication delay caused by the current network fluctuation has not exceeded the vertical synchronization cycle. At this time, it can further determine whether the time difference between the screen display and the current display is greater than zero.

[0165] Step A3033: If the value is greater than zero, a third selection instruction is generated. The third selection instruction is an instruction to select reference attitude prediction data from multiple sets of attitude prediction times.

[0166] In its implementation, when the head-mounted display device detects that the time difference between the displayed screen and the screen refresh rate is less than or equal to the duration corresponding to the current display refresh rate, and the time difference is greater than zero, it can determine that the current communication latency is close to zero or that there are no network fluctuations or other latency factors. In this case, it should display at the preset screen time and generate a third selection instruction. The head-mounted display device sends the third selection instruction to the computing device, which can then select the default baseline attitude prediction data from multiple sets of attitude prediction data for rendering based on the third selection instruction.

[0167] Step A3034: If the difference is less than zero, a selection instruction is generated based on the absolute value of the time difference between the screen displays.

[0168] In a specific implementation, when the head-mounted display device detects that the time difference between the on-screen display and the on-screen display is less than zero, it determines that the display screen should be displayed in advance before the preset on-screen display time. At this time, the time for advance display can be determined based on the absolute value of the time difference between the on-screen display and the corresponding selection instruction can be generated.

[0169] In one possible implementation, after step A3032, the method further includes:

[0170] B3033, if it is less than zero, then determine whether the absolute value of the time difference between the screen displays is greater than the absolute value of the current screen refresh rate.

[0171] In a specific implementation, when the head-mounted display device detects that the time difference between the screen display and the screen is less than zero, it can compare the absolute value of the time difference with the duration corresponding to the current screen refresh rate to determine whether the absolute value of the time difference is greater than the duration corresponding to the current screen refresh rate.

[0172] B3034, if so, then generate a fourth selection instruction, which is an instruction to select the third target attitude prediction data with the shortest preset on-screen time from multiple sets of attitude prediction data.

[0173] In its implementation, when the absolute value of the time difference detected by the head-mounted display device is greater than the duration corresponding to the current display refresh rate, it generates a fourth selection instruction. The head-mounted display device sends the fourth selection instruction to the computing device, which then selects the third target attitude prediction data with the shortest preset time from multiple sets of attitude prediction data based on the fourth selection instruction for rendering, so as to display it in advance before the baseline preset time. The shortest preset time can be the minimum value among the preset time.

[0174] B3035, if not, then generate a fifth selection instruction, which is an instruction to select a fourth target attitude prediction data from multiple sets of attitude prediction data that is greater than the shortest preset screen time and less than the benchmark preset screen time.

[0175] In its implementation, when the absolute value of the detected screen-on time difference is less than or equal to the duration corresponding to the current screen refresh rate, the aforementioned head-mounted display device can generate a fifth selection instruction. The head-mounted display device sends the fifth selection instruction to the computing device, which can then select, based on the fifth selection instruction, a fourth target attitude prediction data from multiple sets of attitude prediction data that is greater than the shortest preset screen-on time and less than the benchmark preset screen-on time for rendering, so as to display it ahead of the benchmark preset screen-on time and at a distance far from the benchmark preset screen-on time.

[0176] For example, for ease of understanding, T_diff is used as the time difference between the screen display and the current display refresh rate, vsync is used as the duration corresponding to the current display refresh rate, T_ping_diff is used as the current communication latency, and T is used as the preset latency threshold. The preset duration of the first target attitude prediction data is the next vsync + N + 2 vsyncs, the preset duration of the first target attitude prediction data is the next vsync + N + 1 vsyncs, the preset duration of the reference attitude prediction data is the next vsync + N vsyncs, the preset duration of the third target attitude prediction data is the next vsync + N - 1 vsyncs, and the preset duration of the fourth target attitude prediction data is the next vsync + N - 2 vsyncs. However, this does not limit the solution.

[0177] When T_diff > vsync and T_ping_diff > T, a first selection instruction is sent to control the computing device to select the first target attitude prediction number with a preset duration of the next vsync + N + 2 vsyncs.

[0178] When T_diff > vsync and T_ping_diff < T, a second selection instruction is sent to control the computing device to select the second target attitude prediction data with a preset duration of the next vsync + N + 1 vsyncs.

[0179] When 0 < T_diff < vsync, a third selection instruction is sent to control the computing device to select the reference attitude prediction data with a preset duration of the next vsync + N vsyncs.

[0180] When -vsync < T_diff < 0, a fourth selection instruction is sent to control the computing device to select the third target attitude prediction data with a preset duration of the next vsync + N-1 vsyncs.

[0181] When T_diff < -vsync, a fifth selection instruction is sent to control the computing device to select the fourth target attitude prediction data with a preset duration of the next vsync + N-2 vsyncs.

[0182] This embodiment determines the actual on-screen time corresponding to the baseline rendering data and extracts the baseline preset on-screen time corresponding to the baseline attitude prediction data from the baseline rendering data; calculates the difference between the actual on-screen time and the baseline preset on-screen time to obtain the on-screen time difference; and generates selection instructions corresponding to each attitude prediction data based on the on-screen time difference and the current communication latency. Because this embodiment utilizes the on-screen time difference and the different ranges of the current communication latency to generate corresponding preset on-screen time selection instructions, it ensures that the image is displayed at the optimal time, effectively improving the display effect.

[0183] This embodiment also provides a display method. Referring to FIG5, FIG5 is a flowchart of the fourth embodiment of the display method of the present invention.

[0184] In this embodiment, the display method includes steps S10' to S40':

[0185] Step S10': Receive multiple sets of attitude prediction data sent by the head-mounted display device.

[0186] It should be noted that the executing entity in this embodiment can be a computing device with screen display, network communication, program execution functions, and rendering technology, such as a mobile phone, tablet computer, or personal computer. It can also be other electronic devices or terminals that perform the same or similar functions. The following uses an AR device as an example to illustrate this embodiment, with the computing device in the AR device forming a separate structure from the head-mounted display device.

[0187] In its implementation, the aforementioned head-mounted display device can track and determine its attitude in space using built-in sensors, such as an inertial measurement unit (IMU), generating attitude prediction data. These sensors measure the acceleration and angular velocity of the head-mounted display device, and algorithms process this data to estimate its position and attitude. The generated attitude prediction data ensures that the head-mounted display device accurately overlays the displayed image onto the user's field of vision. The head-mounted display device can acquire attitude prediction data at fixed intervals (e.g., 1 second or 5 seconds). After acquiring multiple sets of attitude prediction data, a preset on-screen time can be added to the corresponding attitude prediction data, and then the multiple sets of attitude prediction data containing the preset on-screen time are sent to a computing device.

[0188] Step S20': Receive the selection instruction sent by the head-mounted display device based on the current communication delay.

[0189] In its implementation, during initialization, the head-mounted display device can send arbitrary data to the computing device and record the data transmission time. After receiving response data from the computing device based on this data, it records the data reception time and determines the current communication latency based on the time difference between the transmission and reception times. The head-mounted display device can compare the current communication latency with the vertical synchronization period and determine the target display time from preset display times based on the deviation of the current communication latency from the vertical synchronization period. This ensures that the display compensates for the current communication latency when the image is displayed at the target display time. Simultaneously, it can generate a selection instruction for choosing the target attitude prediction data from various attitude prediction data and then send this selection instruction to the computing device.

[0190] Step S30': Select target pose prediction data corresponding to the actual screen time from each of the pose prediction data based on the selection instruction, and generate target rendering data based on the target pose prediction data.

[0191] In a specific implementation, after receiving multiple sets of attitude prediction data and the selection instruction, the aforementioned computing device can select target attitude prediction data containing the target on-screen time from the multiple sets of attitude prediction data through the selection instruction. Since the target on-screen time can compensate for the current delay, it can be determined that the target on-screen time corresponds to the actual on-screen time, which can ensure that when the display screen is displayed, the target on-screen time is consistent with or close to the actual on-screen time.

[0192] Step S40': The target rendering data is sent to the head-mounted display device so that the head-mounted display device outputs a display image based on the target rendering data.

[0193] In its implementation, after selecting target pose prediction data, the aforementioned computing device can use this data to render a virtual image. The rendering process includes calculating the correct position, size, and orientation of the virtual image within the user's field of vision using the target pose prediction data. After the calculation is completed, target rendering data is generated and sent to the head-mounted display device. The head-mounted display device can then further refine the target rendering data based on the target pose prediction data to form the final display image. When the target on-screen time is reached, the display image is output on the display interface for the user to view.

[0194] For ease of understanding, the following description refers to Figure 6, which is an example diagram of the modular structure of the split AR device of the present invention. Figure 6 shows a feasible example structure and does not limit the present invention. In Figure 6, the split AR device includes a head-mounted display device and a computing device. The head-mounted display device includes a time analysis module, a communication receiving module, a decoding module, an ATW module, a posture generation module, a posture prediction module, and a display module. The computing device's application runtime environment includes a posture usage module and a posture caching module. The computing device also includes an encoding module and a communication sending module. The communication receiving module of the head-mounted display device and the communication sending module in the computing device communicate via a network, involving protocols such as Transmission Control Protocol (TCP) and User Datagram Protocol (UDP).

[0195] As shown in Figure 6, the attitude generation module in the head-mounted display device generates attitude prediction data. The attitude usage module retrieves multiple sets of attitude prediction data from the attitude generation module at fixed intervals (e.g., 1 second or 5 seconds) and adds corresponding preset display times to each set of attitude prediction data. The attitude usage module can send multiple sets of attitude prediction data (e.g., 5 sets) to the attitude cache module in the computing device at once. The attitude cache module can cache the two most recent attitude prediction data. For example, if the attitude usage module sends 5 sets of attitude prediction data every 1 ms, the attitude cache module can cache 10 sets of attitude prediction data. By keeping the two most recent attitude prediction data, the attitude cache module can avoid conflicts between the latest attitude prediction data being written and read / write operations. The attitude utilization module first renders the baseline attitude prediction data. After generating the baseline rendering data, the encoding module encodes the baseline rendering data and then sends the encoded baseline data to the communication receiving module of the head-mounted display device through the communication sending module. The head-mounted display device decodes the baseline encoded data through the decoding module, obtains the baseline rendering data, and then sends it to the ATW module. The time analysis module can extract the baseline preset on-screen time from the baseline rendering data from the ATW module, and use the baseline preset on-screen time to determine the current communication delay before generating the corresponding selection command and sending the selection command to the attitude utilization module of the computing device.

[0196] The attitude utilization module in the computing device selects the corresponding target attitude prediction data from multiple sets of attitude prediction data based on selection instructions for rendering. After generating target rendering data, the target rendering data is encoded by the encoding module and then sent to the communication receiving module of the head-mounted display device via the communication sending module. The head-mounted display device decodes the target encoded data through the decoding module, obtains the target rendering data, and sends it to the ATW module. The ATW module can obtain the target attitude prediction data from the attitude generation module, compensate for the target rendering data, and output the compensated display image on the display interface of the display module. In addition, external devices can also connect to the computing device and send attitude prediction data to the computing device. The attitude utilization module of the computing device can combine the attitude prediction data sent by the external device and the target attitude prediction data to perform rendering and generate corresponding rendering data. External devices include: mobile phones, gamepads, or rings, etc., devices with attitude information.

[0197] The display method provided in this embodiment receives multiple sets of attitude prediction data sent by a head-mounted display device; receives a selection command sent by the head-mounted display device based on the current communication delay; selects target attitude prediction data corresponding to the actual screen-on time from the attitude prediction data based on the selection command, and generates target rendering data based on the target attitude prediction data; and sends the target rendering data to the head-mounted display device so that the head-mounted display device outputs a display screen based on the target rendering data. Because the head-mounted display device in this embodiment sends multiple sets of attitude prediction data corresponding to different preset screen-on times to the computing device, and simultaneously controls the computing device to select target attitude prediction data corresponding to the actual screen-on time to generate target rendering data according to the selection command corresponding to the current communication delay, it can ensure that the display screen is displayed at the expected actual screen-on time, avoiding the display screen from being displayed early or late, thereby improving the display effect.

[0198] It should be noted that the above embodiments are only for understanding the present invention and do not constitute a limitation on the application of the safety management method of the present invention. Any simple modifications based on this technical concept are within the protection scope of the present invention.

[0199] The present invention also provides a head-mounted display device. Please refer to Figure 7, which is a schematic diagram of the module structure of the head-mounted display device of the present invention. In Figure 7, the head-mounted display device includes:

[0200] The attitude prediction module 10 is used to send multiple sets of attitude prediction data to the computing device, and each set of attitude prediction data is determined based on a different preset screen display time.

[0201] The latency determination module 20 is used to determine the current communication latency with the computing device.

[0202] The time analysis module 30 is used to send selection instructions corresponding to each of the attitude prediction data to the computing device through the current communication delay, so that the computing device selects target attitude prediction data corresponding to the actual screen time from each of the attitude prediction data based on the selection instructions to generate target rendering data.

[0203] Display module 40 is used to receive the target rendering data sent by the computing device and output a display screen based on the target rendering data.

[0204] The head-mounted display device provided by this invention, employing the display method executed by the head-mounted display device in the above embodiments, can solve the technical problem in the prior art where the process from image generation to display has a time delay, causing the display image to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the head-mounted display device provided by this invention are the same as the beneficial effects of the display method executed by the head-mounted display device provided in the above embodiments, and other technical features in the head-mounted display device are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0205] The present invention also provides a computing device. Please refer to Figure 8, which is a schematic diagram of the modular structure of the computing device of the present invention. In Figure 8, the computing device includes:

[0206] The data receiving module 50 is used to receive multiple sets of attitude prediction data sent by the head-mounted display device.

[0207] The data receiving module 50 is also used to receive the selection instruction sent by the head-mounted display device based on the current communication delay.

[0208] The data selection module 60 is used to select target pose prediction data corresponding to the actual screen time from each of the pose prediction data based on the selection instruction, and generate target rendering data based on the target pose prediction data.

[0209] The rendering module 70 is used to send the target rendering data to the head-mounted display device so that the head-mounted display device outputs a display image based on the target rendering data.

[0210] The computing device provided by this invention, employing the display method executed by the computing device in the above embodiments, can solve the technical problem in the prior art where the process from image generation to display has a time delay, causing the image to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the computing device provided by this invention are the same as those of the display method executed by the computing device in the above embodiments, and other technical features in the computing device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0211] The present invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the display method in Embodiment 1 or 4 described above.

[0212] Referring now to Figure 9, which is a schematic diagram of the hardware operating environment involved in the display method of this embodiment of the invention, the electronic device in this embodiment may include, but is not limited to, head-mounted display devices such as AR glasses, VR glasses, and MR glasses, combined with terminals such as mobile phones, laptops, in-vehicle terminals, and desktop computers, forming an XR device. The electronic device shown in Figure 9 is merely an example and should not impose any limitations on the functionality and scope of use of this embodiment of the invention.

[0213] As shown in Figure 9, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. While electronic devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0214] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0215] The electronic device provided by this invention, employing the display method in the above embodiments, can solve the technical problem in the prior art where the process from image generation to display has a time delay, causing the image to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the electronic device provided by this invention are the same as those of the display method provided in the above embodiments, and other technical features in this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0216] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0217] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0218] The present invention provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, the computer-readable program instructions being used to execute the display method in the above embodiments.

[0219] The computer-readable storage medium provided by this invention may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0220] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0221] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0222] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0223] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules do not necessarily limit the specific unit itself.

[0224] The readable storage medium provided by this invention is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described display method. This solves the technical problem in the prior art where the process from image generation to display of the screen involves a time delay, causing the screen to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this invention are the same as those of the display method provided in the above embodiments, and will not be repeated here.

[0225] The above description is only a part of the embodiments of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the technical concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A display method, characterized in that, The display method includes: Multiple sets of attitude prediction data are sent to the computing device, and each set of attitude prediction data corresponds to a different preset screen display time. Determine the current communication latency with the computing device; The current communication delay is used to send the selection instruction corresponding to each of the attitude prediction data to the computing device, so that the computing device selects the target attitude prediction data corresponding to the actual screen time from each of the attitude prediction data based on the selection instruction to generate target rendering data. The system receives the target rendering data sent by the computing device and outputs a display screen based on the target rendering data.

2. The display method as described in claim 1, characterized in that, The step of determining the current communication latency with the computing device includes: Receive reference rendering data sent by the computing device based on the reference attitude prediction data in each of the attitude prediction data; The actual communication delay is obtained by calculating the difference between the reception time of the reference rendering data and the generation time of the reference pose prediction data. The difference between the actual communication delay and the theoretical communication delay corresponding to the preset screen display time is calculated to obtain the current communication delay.

3. The display method as described in claim 2, characterized in that, The step of sending the selection instructions corresponding to each attitude prediction data to the computing device through the current communication delay includes: Determine the actual on-screen time corresponding to the baseline rendering data, and extract the baseline preset on-screen time corresponding to the baseline pose prediction data from the baseline rendering data; Calculate the difference between the actual screen-on time and the baseline preset screen-on time to obtain the screen-on time difference; Based on the time difference between the screen display and the current communication delay, a selection instruction corresponding to each of the attitude prediction data is generated.

4. The display method as described in claim 3, characterized in that, The step of generating selection instructions corresponding to each attitude prediction data based on the screen display time difference and the current communication delay includes: Determine whether the time difference between the displayed screen and the actual screen refresh rate is greater than the duration corresponding to the current display refresh rate; If so, determine whether the current communication delay is greater than a preset delay threshold; If the delay exceeds the preset time threshold, a first selection instruction is generated. The first selection instruction is an instruction to select the first target attitude prediction data with the longest preset on-screen time from multiple sets of attitude prediction data. If the delay is less than the preset time threshold, a second selection instruction is generated. The second selection instruction is an instruction to select a second target attitude prediction data from multiple sets of attitude prediction data that is greater than the baseline preset screen time and less than the longest preset screen time.

5. The display method as described in claim 4, characterized in that, After the step of determining whether the time difference between the displayed screens is greater than the duration corresponding to the current display refresh rate, the method further includes: If not, then determine whether the time difference between the screen displays is greater than zero; If the value is greater than zero, a third selection instruction is generated, which is an instruction to select reference attitude prediction data from multiple sets of attitude prediction times. If it is less than zero, a selection instruction is generated based on the absolute value of the time difference between the screen displays.

6. The display method as described in claim 5, characterized in that, The step of generating a selection instruction based on the absolute value of the time difference on the screen if the difference is less than zero includes: If it is less than zero, then determine whether the absolute value of the time difference between the screen displays is greater than the absolute value of the current screen refresh rate; If so, a fourth selection instruction is generated, which is an instruction to select the third target attitude prediction data with the shortest preset on-screen time from multiple sets of attitude prediction data. If not, a fifth selection instruction is generated, which is an instruction to select a fourth target attitude prediction data from multiple sets of attitude prediction data that is greater than the shortest preset screen time and less than the benchmark preset screen time.

7. The display method as described in claim 1, characterized in that, Before the step of sending multiple sets of attitude prediction data to the computing device, the method further includes: Send initial attitude prediction data to the computing device; Receive initial rendering data sent by the computing device based on the initial pose prediction data; Calculate the difference between the reception time of the initial rendering data and the generation time of the initial pose prediction data to obtain the current theoretical communication delay; Multiple different preset screen display times are determined based on the current theoretical communication delay; Add the preset on-screen times to the corresponding attitude prediction data.

8. The display method as described in claim 7, characterized in that, The step of determining multiple different preset screen display times based on the current theoretical communication delay includes: Determine the target time index based on the current theoretical communication delay; The target time index is updated by using preset parameters to obtain multiple updated time indices; Different preset on-screen times are determined based on the current display refresh rate and multiple updated time metrics.

9. The display method as described in claim 8, characterized in that, The step of determining the target time index based on the current theoretical communication delay includes: Obtain a preset deep learning model, which is a long short-term memory network model obtained by training based on training data consisting of parameters corresponding to multiple theoretical communication delays and multiple time indicators. The initial time index is obtained based on the current theoretical communication latency using the preset deep learning model; The initial time index is rounded to obtain the target time index.

10. A display method, characterized in that, The display method includes: Receive multiple sets of attitude prediction data sent by the head-mounted display device; Receive the selection command sent by the head-mounted display device based on the current communication delay; Based on the selection instruction, target pose prediction data corresponding to the actual screen time is selected from each of the pose prediction data, and target rendering data is generated based on the target pose prediction data; The target rendering data is sent to the head-mounted display device so that the head-mounted display device outputs a display image based on the target rendering data.

11. A head-mounted display device, characterized in that, The head-mounted display device includes: The attitude prediction module is used to send multiple sets of attitude prediction data to the computing device, and each set of attitude prediction data is determined based on a different preset screen display time; A latency determination module is used to determine the current communication latency with the computing device; The time analysis module is used to send selection instructions corresponding to each of the attitude prediction data to the computing device through the current communication delay, so that the computing device selects target attitude prediction data corresponding to the actual screen time from each of the attitude prediction data based on the selection instructions to generate target rendering data; The display module is used to receive the target rendering data sent by the computing device and output a display screen based on the target rendering data.

12. A computing device, characterized in that, The computing device includes: The data receiving module is used to receive multiple sets of attitude prediction data sent by the head-mounted display device; The data receiving module is also used to receive the selection command sent by the head-mounted display device based on the current communication delay; The data selection module is used to select target pose prediction data corresponding to the actual screen time from each of the pose prediction data based on the selection instruction, and generate target rendering data based on the target pose prediction data. The rendering module is used to send the target rendering data to the head-mounted display device so that the head-mounted display device outputs a display image based on the target rendering data.

13. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the display method as described in any one of claims 1 to 9 or claim 10.

14. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the display method as described in any one of claims 1 to 9 or claim 10.

Images