Portable display device for displaying stereoscopic image and control method therefor

Abstract

A portable display device for displaying a stereoscopic image and a control method therefor may be provided. Specifically, there may be provided a portable display device comprising: a camera; an inertial measurement unit (IMU) sensor; a memory storing at least one instruction; and at least one processor, wherein the at least one processor executes the at least one instruction independently or collectively to cause the portable display device to: detect the position of the eye of a user through the camera; correct the pose of the eye of the user; obtain IMU information related to the orientation of the portable display device through the IMU sensor; primarily predict the position of the eye of the user with respect to a plurality of future situations; and secondarily predict the position of the eye of the user by using the IMU information.

Description

Portable display device for displaying stereoscopic images and control method thereof

[0001] The present disclosure relates to a portable display device for displaying stereoscopic images and a method for controlling the same. Specifically, the present disclosure relates to a technology for reducing crosstalk that occurs when a light field display (LFD) is applied to a portable display device such as a tablet or a smartphone.

[0002] A light field display (LFD) can be a 3D display that creates stereoscopic images by generating a light field, which is represented as a vector distribution of light in space, using a flat display and optical elements. A light field can be a vector function representing the direction of light propagation and the intensity of light at every point in 3D space. A light field display can display the depth and sides of an object, thereby enabling the realization of stereoscopic images more naturally.

[0003] Light field displays can have the characteristic of displaying different information depending on the direction the user is looking. To apply light field display technology, it may be necessary to accurately determine where the user's eyes are located. Conventional stereoscopic display devices that applied light field display technology used two cameras to determine the position of the user's eyes.

[0004] Recently, there have been attempts to apply light field displays to portable display devices such as tablets and smartphones. When applying light field displays to portable display devices, unlike conventional stereoscopic display devices, it may not be easy to accurately measure the distance between the portable display device and the eyes by using a single camera to determine the position of the user's eyes. If the distance between the portable display device and the eyes cannot be accurately measured, crosstalk may occur in the stereoscopic image.

[0005] Additionally, at least one of the portable display device and the user may move during use. If at least one of the portable display device and the user moves, a technology may be required to predict the position of the user's eyes and display a stereoscopic image that matches the predicted position of the user's eyes. If the position of the user's eyes is not accurately predicted, crosstalk may occur in the stereoscopic image.

[0006] A portable display device for displaying stereoscopic images according to one embodiment of the present disclosure comprises: a camera; an inertial measurement unit (IMU) sensor; a memory for storing at least one instruction; and at least one processor operably coupled to the memory and including processing circuitry, wherein the at least one processor executes the at least one instruction individually or collectively, thereby enabling the portable display device to detect the position of a user's eye through the camera, correct the pose of the user's eye, acquire IMU information related to the orientation of the portable display device through the inertial measurement unit sensor, primarily predict the position of the user's eye for a plurality of future situations, and secondarily predict the position of the user's eye using the IMU information.

[0007] A control method for a portable display device that displays a stereoscopic image according to one embodiment of the present disclosure may include: detecting the position of a user's eye through a camera of the portable display device; correcting the pose of the user's eye through the portable display device; acquiring IMU information related to the orientation of the portable display device through an inertial measurement unit (IMU) sensor of the portable display device; primarily predicting the position of the user's eye for a plurality of future situations through the portable display device; and secondarily predicting the position of the user's eye using the IMU information through the portable display device.

[0008] FIG. 1 is a drawing showing a stereoscopic image displayed on a portable display device according to one embodiment of the present disclosure, directed toward the user's eyes.

[0009] FIG. 2 is a block diagram showing a portable display device according to one embodiment of the present disclosure.

[0010] FIG. 3 is a drawing showing at least one of a portable display device and a user moving according to one embodiment of the present disclosure.

[0011] FIG. 4 is a flowchart illustrating a control method for a portable display device according to one embodiment of the present disclosure.

[0012] FIG. 5 is a flowchart illustrating in more detail a control method for a portable display device according to one embodiment of the present disclosure.

[0013] FIG. 6 is a drawing showing a portable display device according to one embodiment of the present disclosure correcting a user's pose.

[0014] FIG. 7 is a flowchart illustrating a method for a portable display device to estimate a user's pose according to one embodiment of the present disclosure.

[0015] FIG. 8 is a drawing showing a portable display device according to one embodiment of the present disclosure estimating a user's pose.

[0016] FIG. 9 is a drawing showing a portable display device according to one embodiment of the present disclosure correcting a user's pose.

[0017] FIG. 10 is a drawing showing a portable display device according to one embodiment of the present disclosure correcting a user's pose.

[0018] FIG. 11 is a drawing showing a portable display device according to one embodiment of the present disclosure correcting a user's pose.

[0019] FIG. 12 is a diagram showing a portable display device according to one embodiment of the present disclosure calculating the speed at which a user's eye moves.

[0020] FIG. 13 is a diagram showing a portable display device according to one embodiment of the present disclosure calculating the speed at which a user's eye moves.

[0021] FIG. 14 is a diagram showing a portable display device according to one embodiment of the present disclosure calculating the speed at which a user's eye moves.

[0022] FIG. 15 is a diagram showing a portable display device according to one embodiment of the present disclosure primarily predicting the position of a user's eyes.

[0023] FIG. 16 is a diagram showing a portable display device according to one embodiment of the present disclosure correcting an eye position predicted using inter-pupillary distance (IPD).

[0024] FIG. 17 is a diagram showing a portable display device according to one embodiment of the present disclosure secondarily predicting the position of a user's eyes.

[0025] FIG. 18 is a flowchart illustrating a method for a portable display device according to one embodiment of the present disclosure to display a stereoscopic image in accordance with the position of the user's eyes.

[0026] FIG. 19 is a flowchart illustrating a method in which a portable display device according to one embodiment of the present disclosure uses accumulated IMU information to correct the position of a user's eye and displays a stereoscopic image to match the corrected position of the eye.

[0027] FIG. 20 is a diagram showing a portable display device according to one embodiment of the present disclosure correcting the position of a user's eyes using accumulated IMU information.

[0028] FIG. 21 is a flowchart illustrating a method for a portable display device according to one embodiment of the present disclosure to display a stereoscopic image in accordance with the characteristics of a user's movement.

[0029] FIG. 22 is a drawing showing a portable display device according to one embodiment of the present disclosure that compensates for the position of the user's eyes to match the characteristics of the user's movement.

[0030] FIG. 23 is a diagram showing a portable display device according to one embodiment of the present disclosure predicting the position of a user's eyes and measuring the result of displaying a stereoscopic image according to the predicted result.

[0031] FIG. 24 is a graph showing the maximum error between the point in time when a portable display device according to a comparative example and an embodiment of the present disclosure displays a stereoscopic image and the position of the user's eyes in various situations.

[0032] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0033] In relation to the description of the drawings, similar reference numerals may be used for similar or related components.

[0034] The singular form of the noun corresponding to the item may include one or multiple items, unless the relevant context clearly indicates otherwise.

[0035] In this document, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0036] The term "and / or" includes a combination of multiple related described components or any of the multiple related described components.

[0037] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0038] Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0039] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this document, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0040] When it is said that a component is "connected," "combined," "supported," or "in contact" with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0041] When it is said that a component is located "on" another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0042] It should be understood that the blocks in each flowchart and combinations of flowcharts can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored all in a single memory or may be partitioned and stored in multiple different memories.

[0043] All functions or operations described in this document may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuitry that performs processing and may include circuitry such as an AP (Application Processor), CP (Communication Processor), GPU (Graphical Processing Unit), NPU (Neural Processing Unit), MPU (Microprocessor Unit), SoC (System on Chip), IC (Integrated Chip), etc.

[0044] FIG. 1 is a drawing showing a stereoscopic image displayed in a portable display device (100) according to one embodiment of the present disclosure, directed toward the user's eyes (131, 132).

[0045] A portable display device (100) can track the position of the user's eyes (131, 132). The portable display device (100) can detect the position of the user's eyes (131, 132) using a camera (110). The portable display device (100) can display a stereoscopic image at the detected position of the user's eyes (131, 132) using a display (120).

[0046] A portable display device (100) can render an image so that a three-dimensional stereoscopic image appears at the detected position of the user's eyes (131, 132). The stereoscopic image can be seen by the user's left eye (131) and right eye (132), respectively, to give the user a sense of depth.

[0047] A portable display device (100) may be an electronic device that displays a stereoscopic image so that a user can view a stereoscopic image while carrying it. For example, the portable display device (100) may be a tablet, a smartphone, or other mobile electronic device.

[0048] As the portable display device (100) or the user moves, the position of the user's eyes (131, 132) may change. When the position of the user's eyes (131, 132) changes, it may not be easy for the stereoscopic image provided in real time to reflect the changed position of the user's eyes (131, 132). Accordingly, if the stereoscopic image fails to reflect the changed position of the user's eyes (131, 132), a crosstalk phenomenon may occur in the stereoscopic image, causing discomfort when the user views the stereoscopic image. The portable display device (100) according to the present disclosure can continuously and accurately predict the position of the user's eyes (131, 132) and provide a stereoscopic image at the predicted position of the user's eyes (131, 132) to reduce the crosstalk phenomenon in the stereoscopic image.

[0049] FIG. 2 is a block diagram showing a portable display device (100) according to one embodiment of the present disclosure. A portable display device (100) according to one embodiment of the present disclosure may include a camera (110), a display (120), a communication circuit (210), an inertial measurement unit (IMU) sensor (220), a processor (230), and a memory (240).

[0050] The camera (110) can photograph the surroundings of the portable display device (100). For example, the camera (110) can photograph a user on the front side of the portable display device (100) to capture the user and the user's surrounding environment.

[0051] The display (120) can display a three-dimensional image that the portable display device (100) intends to display. The display (120) may be a light field display (LFD). A light field display may be a three-dimensional display that creates a three-dimensional image by generating a light field, which is represented as a vector distribution of light in space, using a flat display and an optical element. The light field may be a vector function representing the direction of light propagation and the intensity of light at every point in three-dimensional space. A light field display can display the depth and side of an object, thereby enabling a more natural three-dimensional image.

[0052] The communication circuit (210) can communicate with other electronic devices. The communication circuit (210) can receive data from other electronic devices. The communication circuit (210) can transmit the received data to a processor (230) and a memory (240). The communication circuit (210) may include signal processing circuits such as a transceiver, an encoder, and a decoder.

[0053] The inertial measurement unit sensor (220) can measure the movement of the portable display device (100). The inertial measurement unit sensor (220) can measure the acceleration and angular velocity of the portable display device (100). The inertial measurement unit sensor (220) can measure the orientation of the portable display device (100). The inertial measurement unit sensor (220) can measure changes in the direction in which the portable display device (100) is facing. The inertial measurement unit sensor (220) may include a 9-axis sensor, an accelerometer, and a gyroscope.

[0054] The processor (230) can process data received by the communication circuit (210) and data stored in memory (240). The processor (230) may include a circuit having a physical structure. The processor (230) may be a data processing device implemented in hardware. For example, the processor (230) may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), or a Field Programmable Gate Array (FPGA).

[0055] The memory (240) can store at least one program capable of operating the portable display device (100). The at least one program may include one or more computer-readable instructions.

[0056] The processor (230) can execute at least one program stored in memory (240). The processor (230) can execute one or more instructions included in at least one program.

[0057] The processor (120) may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include at least one processor and various processing circuitry. In at least one processor, one or more processors may be configured to perform the various functions described herein in a distributed manner, individually and / or collectively. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform various functions. However, these terms cover, for example but without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor can perform all functions. Additionally, at least one processor may include a combination of processors performing various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0058] The processor (115) is configured to control a series of processes to operate the electrostatic precipitator (110) according to the embodiments described below, and may be composed of one or more processors. The one or more processors included in the processor (115) may be circuitry such as a System on Chip (SoC) or an Integrated Circuit (IC). The one or more processors included in the processor (115) may be general-purpose processors such as a CPU (Central Processing Unit), MPU (Micro Processor Unit), AP (Application Processor), or DSP (Digital Signal Processor); graphics-dedicated processors such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit); artificial intelligence-dedicated processors such as an NPU (Neural Processing Unit); or communication-dedicated processors such as a CP (Communication Processor). If the one or more processors included in the processor (115) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0059] FIG. 3 is a drawing showing at least one of a portable display device (100) and a user (320) moving according to one embodiment of the present disclosure.

[0060] The camera (110) of the portable display device (100) may be positioned to face the front of the portable display device (100). The camera (110) may capture a user (320) viewing the stereoscopic image displayed by the portable display device (100).

[0061] As the user (320) sees a stereoscopic image while carrying the portable display device (100), at least one of the portable display device (100) and the user (320) may move. For example, the angle of the portable display device (100) may change as in the first motion (310). For example, the position of the user (320) may change as in the second motion (330).

[0062] When at least one of the portable display device (100) and the user (320) moves, the movement of the user's (320) eyes must be accurately detected in real time, and the position of the user's (320) eyes at the next point in time must be predicted to provide a stereoscopic image that matches the position of the user's (320) eyes. A single camera (110) may generally be installed in the portable display device (100). According to the present disclosure, by using a method of predicting multiple future situations, a technology can be provided to accurately predict the position of the user's (320) eyes even when there is only one camera (110) in the portable display device (100).

[0063] FIG. 4 is a flowchart illustrating a control method for a portable display device (100) according to one embodiment of the present disclosure.

[0064] In operation 410, a portable display device (100) according to one embodiment of the present disclosure can detect the position of a user's eyes through a camera (110). The portable display device (100) can capture an image including a user. The portable display device (100) can detect the user's eyes in the captured image. The processor (230) of the portable display device (100) can store eye position information including the detected position of the user's eyes in a memory (240).

[0065] In operation 420, a portable display device (100) according to one embodiment of the present disclosure can correct the pose of the user's eyes. The pose of the user's eyes may include information related to the direction in which the user's eyes are directed, such as the angle formed by the user's left and right eyes and the direction in which the user's gaze is directed. The processor (230) of the portable display device (100) can correct the pose of the user's eyes to obtain an accurate distance between the portable display device (100) and the user's eyes even when the user's eyes are directed obliquely rather than vertically toward the portable display device (100).

[0066] In operation 430, a portable display device (100) according to one embodiment of the present disclosure can obtain IMU information related to the orientation of the portable display device (100) through an inertial measurement unit sensor (220). The inertial measurement unit sensor (220) can measure the direction in which the portable display device (100) is facing, the acceleration of the portable display device (100), and the angular velocity of the portable display device (100). The processor (230) of the portable display device (100) can obtain IMU information including the direction, acceleration, and angular velocity values ​​of the portable display device (100) measured by the inertial measurement unit sensor (220).

[0067] In operation 440, a portable display device (100) according to one embodiment of the present disclosure can primarily predict the position of the user's eyes for a plurality of future situations. The processor (230) of the portable display device (100) can check in advance a plurality of future situations that may be expressed in the next frames that will follow the current frame of the stereoscopic image. The processor (230) can predict the position of the user's eyes in each of the plurality of future situations.

[0068] In operation 450, a portable display device (100) according to one embodiment of the present disclosure can secondarily predict the position of the user's eyes using IMU information. The processor (230) of the portable display device (100) can apply the acquired IMU information to each of a plurality of future situations that are primarily predicted. The processor (230) can predict the position of the user's eyes more accurately by synthesizing the results of applying the IMU information. The portable display device (100) can provide a stereoscopic image that matches the predicted position of the user's eyes. Accordingly, the portable display device (100) can reduce crosstalk phenomena occurring in stereoscopic images.

[0069] FIG. 5 is a flowchart illustrating in more detail a control method of a portable display device (100) according to one embodiment of the present disclosure.

[0070] In operation 510, a portable display device (100) according to one embodiment of the present disclosure can detect the position of the eyes and remove noise. The portable display device (100) can detect the position of the user's eyes through a camera (110). The portable display device (100) can capture an image including the user. The processor (230) of the portable display device (100) can determine that the part of the captured image excluding the user's face is noise. The processor (230) removes the part determined as noise and can detect the position of the user's eyes on the user's face.

[0071] In operation 520, a portable display device (100) according to one embodiment of the present disclosure can correct a pose in a first axis direction. The first axis direction may be an axis direction perpendicular to the display (120) of the portable display device (100) (more specifically, a plane defining the display screen of the display (120)). The position of the user's eyes may be the distance in the first axis direction between the portable display device (100) and the user's eyes. If the user's eyes are directed obliquely toward the portable display device (100), the pose in the first axis direction may be distorted. The processor (230) of the portable display device (100) can correct the pose in the first axis direction by taking into account the direction of the user's eyes.

[0072] In operation 530, a portable display device (100) according to one embodiment of the present disclosure can acquire IMU information. The IMU information may include information related to the orientation of the portable display device (100). The IMU information may include the direction, acceleration, and angular velocity values ​​of the portable display device (100). The processor (230) of the portable display device (100) can acquire IMU information through an inertial measurement unit sensor (220).

[0073] In operation 540, a portable display device (100) according to one embodiment of the present disclosure can predict the speed of the eye. The portable display device (100) can predict the acceleration of the eye. The portable display device (100) can measure the angular velocity of the eye.

[0074] In operation 550, a portable display device (100) according to one embodiment of the present disclosure can primarily predict the position of the eye for a plurality of future situations. The processor (230) of the portable display device (100) can check in advance a plurality of future situations that may be expressed in the next frames that will follow the current frame of the stereoscopic image. The processor (230) can predict the position of the user's eye in each of the plurality of future situations.

[0075] In operation 560, a portable display device (100) according to one embodiment of the present disclosure can correct the predicted eye position using the inter-pupillary distance (IPD).

[0076] In operation 570, a portable display device (100) according to one embodiment of the present disclosure can secondarily predict the position of the eye using IMU information. The processor (230) of the portable display device (100) can precisely predict the position of the user's eye at a desired point in time using IMU information. The portable display device (100) can provide a stereoscopic image that matches the predicted position of the user's eye. Accordingly, the portable display device (100) can reduce crosstalk phenomena occurring in stereoscopic images.

[0077] FIG. 6 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure correcting the user's pose.

[0078] The user's pose may include a distance in a first axial direction between the portable display device (100) and the position of the user's eyes. The first axial direction may be a direction from the portable display device (100) toward the user's eyes. The first axial direction may be a vertical direction of the front of the portable display device (100) (or a plane defining the display screen of the display).

[0079] The line of sight of the user's actual eye (610) may be directed obliquely toward the front of the portable display device (100). The camera (110) of the portable display device (100) can calculate the distance between the portable display device (100) and the position of the user's eye by assuming that the user's eye is directed vertically toward the portable display device (100). For example, the processor (230) can calculate the position of the user's eye (or, also referred to as a first position) by assuming that the user's eye is directed vertically toward the portable display device (100).

[0080] If the user's actual eye (610) is facing the portable display device (100) at an angle, an error (630) may occur in the distance to the user's eye measured by the camera (110). For example, if the user's actual eye (610) is facing the portable display device (100) at an angle, the camera (110) may measure that the user's eye is closer to the portable display device (100) than the actual eye (610), causing an error (630).

[0081] The processor (230) of the portable display device (100) can correct an error (630) resulting from the measurement of the camera (110). The processor (230) can correct an error (630) in the direction of the first axis of the user's eye position. The processor (230) can correct the user's eye position to a corrected position (640) that is the same as the actual eye (610). For example, the processor (230) can set the corrected position (640) (or also referred to as a second position) so that the user's eye is parallel to the portable display device (100).

[0082] FIG. 7 is a flowchart illustrating a method for a portable display device (100) according to one embodiment of the present disclosure to estimate a user's pose.

[0083] In operation 710, a portable display device (100) according to one embodiment of the present disclosure can capture a user using a camera (110). The camera (110) of the portable display device (100) can capture a user located in front of the portable display device (100). The processor (230) of the portable display device (100) can identify the face of the captured user.

[0084] In operation 720, a portable display device (100) according to one embodiment of the present disclosure can extract a plurality of three-dimensional landmarks from a captured user's face. The plurality of three-dimensional landmarks may be a plurality of points to be considered to determine the user's pose. A processor (230) of the portable display device (100) can determine a plurality of points of the captured user's face as three-dimensional landmarks.

[0085] In operation 730, a portable display device (100) according to one embodiment of the present disclosure can estimate a user's pose based on a plurality of three-dimensional landmarks. A processor (230) of the portable display device (100) can apply a perspective-n-points method to the plurality of three-dimensional landmarks. The processor (230) can estimate a virtual pose of the camera (110) with respect to an object by applying the perspective-n-points method. The processor (230) can convert the virtual pose of the camera (110) into a user's actual pose.

[0086] FIG. 8 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure estimating a user's pose.

[0087] The processor (230) of the portable display device (100) can estimate the pose of the camera (110) through a plurality of three-dimensional landmarks. The processor (230) can transform the coordinate system (R, t) representing the real world into a coordinate system based on the camera (110). The processor (230) can estimate the pose of the camera (110) through a plurality of landmarks (C) captured by the camera (110). i C, C2, C3, C4) can be mapped to the reference coordinate system (f). The processor (230) has a reference point (P) for projecting the image captured by the camera (110) onto the reference coordinate system (f). i ) can be set. The processor (230) can set a reference point (P i Multiple landmarks (C) projected onto a reference coordinate system (f) centered on ) i The pose of the camera (110) can be estimated based on the positions of C2, C3, and C4.

[0088] The processor (230) of the portable display device (100) can convert the pose of the camera (110) into the pose of the user. The processor (230) can obtain angle information regarding the direction from the camera (110) toward the user through the camera (110). For example, the processor (230) can determine through the image obtained from the camera (110) whether the user is positioned vertically in front of the camera (110), positioned obliquely to the left or right of the camera (110) by 15 degrees, or positioned obliquely to the left or right of the camera (110) by 30 degrees. Based on the obtained angle information, the processor (230) can convert the pose of the camera (110) into the pose of the user.

[0089] FIG. 9 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure correcting the user's pose.

[0090] The processor (230) of the portable display device (100) can correct the user's pose using a pinhole model. The pinhole model may be a model that mathematically represents the user's pose using the symmetry relationship of polygons, particularly triangles. The processor (230) can modify the pinhole model to implement a model that changes according to the user's pose.

[0091] The processor (230) can set each of the user's two eyes (611, 612) as two vertices of a virtual first right triangle. The processor (230) can set a virtual triangle so as to face the portable display device (100) from the virtual right triangle. The processor (230) can set a virtual rectangular prism so as to face the user's two eyes (611, 612) from the portable display device (100). The processor can set two virtual right triangles using the height of the virtual triangle and the height of each of the virtual rectangular prisms.

[0092] The processor (230) can set the central sensor vector (910). The central sensor vector (910) can be defined as in the following mathematical formula 1.

[0093]

[0094] The processor (230) has the heights (D, f) of each of the two imaginary right triangles. k The processor (230) can calculate the length (d) of the hypotenuse of the right triangle that is closer to the user's two eyes (611, 612) among two virtual right triangles. The processor (230) can determine the length (d) of the hypotenuse of the right triangle as the depth from the user's two eyes (611, 612) to the portable display device (100).

[0095] FIG. 10 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure correcting a user's pose.

[0096] The processor (230) of the portable display device (100) can approximate a three-dimensional model into a two-dimensional model. The processor (230) can approximate the three-dimensional model of FIG. 9 to generate a two-dimensional model including a right triangle and two symmetric triangles.

[0097] The processor (230) can calculate the angle (θ) of one vertex of a right triangle generated from a two-dimensional model. The processor (230) can calculate the heights (d, f) of each of the two generated symmetric triangles. k The processor (230) can calculate the angles (α, β) of two vertices located at the bases of two symmetric triangles. The processor (230) can calculate the length (a) of the base of the triangle that is farther from the user's two eyes (611, 612) among the two symmetric triangles.

[0098] The processor (230) can define the length of the hypotenuse (1010) of a right triangle created as a two-dimensional model as shown in the following mathematical formula 2.

[0099]

[0100] The processor (230) can define the height (1020) of a right triangle created as a two-dimensional model as shown in the following mathematical formula 3.

[0101]

[0102] The processor (230) can define the height (1030) of a right triangle created as a two-dimensional model as shown in the following mathematical formula 4.

[0103]

[0104] FIG. 11 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure correcting a user's pose.

[0105] The processor (230) of the portable display device (100) can calculate the lengths of the edges and the angles of the vertices of two symmetric triangles generated in a two-dimensional model to calculate the depth from both eyes (611, 612) of the user to the portable display device (100).

[0106] The processor (230) can define a vector (1110) at the user's right eye corresponding to the vertex of the base of the triangle generated from the 2D model generated from the 2D model as shown in the following mathematical formula 5.

[0107]

[0108] The processor (230) uses the vector (1110) at the user's right eye, the central sensor vector (910), and the height (f) of the triangle generated from the 2D model as shown in the following mathematical formula 6. k Based on ), the tangent angle (tanβ) of a triangle generated in a 2D model can be calculated.

[0109]

[0110] The processor (230) can calculate the depth from both eyes (611, 612) of the user to the portable display device (100) based on the lengths of the corners of the triangle and the angles of the vertices of the triangle defined using the preceding mathematical formulas as in the following mathematical formula 7.

[0111]

[0112] The depth from both eyes (611, 612) of the user calculated by the processor (230) to the portable display device (100) may be a value that corrects the user's pose. That is, the depth from both eyes (611, 612) of the user calculated by the processor (230) to the portable display device (100) may be a value that corrects the error that occurred in the user's pose calculated by the conventional method. Accordingly, the processor (230) can correct the user's pose by mathematically using the symmetrical relationship of virtual triangles generated using a pinhole model.

[0113] FIG. 12 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure calculating the speed at which a user's (320) eye moves.

[0114] The processor (230) of the portable display device (100) can capture an image of the user (320) using the camera (110). The processor (230) can obtain the position of the user's (320) eyes from the captured image.

[0115] The processor (230) can convert the domain representing the position of the user's (320) eyes from pixel units to millimeter (mm) units. The processor (230) can convert the domain representing the position of the user's (320) eyes from pixel units to millimeter units based on the field of view (FoV) of the camera (110) and the resolution of the display (120) of the portable display device (100) as shown in the following mathematical formula 8.

[0116]

[0117] In Equation 8, x can be the position of the eye expressed in pixels. In Equation 8, x' can be the position of the eye expressed in millimeters. In Equation 8, r wd may be the horizontal resolution of the camera (110). In Equation 8, d may be the depth from the user (320) to the portable display device (100). In Equation 8, θ may be the field of view of the camera (110).

[0118] The processor (230) can calculate the position of the user's (320) eyes that has changed by the rotation angle of the portable display device (100) detected by the inertial measurement unit sensor (220). The processor (230) can calculate the vertical distance to the portable display device (100) after rotation as shown in the following mathematical formula 9.

[0119]

[0120] In mathematical equation 9, d t may be the vertical distance to the portable display device (100) after rotation. In Equation 9, d t-1 θ' may be the vertical distance to the portable display device (100) before rotation. In Equation 9, θ' may be the angle at which the portable display device (100) has rotated.

[0121] The processor (230) can calculate the position of the central part of the portable display device (100) after rotation as shown in the following mathematical formula 10.

[0122]

[0123] x' in mathematical formula 10 t may be the position of the central part of the portable display device (100) after rotation. x' in Equation 10 t-1 ☐ may be the position of the central part of the initial portable display device (100). In Equation 10, θ' may be the angle at which the portable display device (100) is rotated.

[0124] The processor (230) can restore the domain representing the changed user (320)'s eye position from millimeter units to pixel units. The processor (230) can restore the domain representing the changed user (320)'s eye position from millimeter units to pixel units based on the field of view of the camera (110) and the resolution of the display (120) of the portable display device (100) as shown in the following mathematical formula 11.

[0125]

[0126] In Equation 11, x can be the position of the eye expressed in pixels. In Equation 11, x' can be the position of the eye expressed in millimeters. In Equation 11, r w d may be the horizontal resolution of the camera (110). In Equation 11, d may be the depth from the user (320) to the portable display device (100). In Equation 11, θ may be the field of view of the camera (110).

[0127] The processor (230) can calculate the speed at which the user's (320) eyes move by subtracting the changed user's (320) eye position obtained from the initially captured image from the changed user's (320) eye position obtained from Equation 11.

[0128] FIG. 13 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure calculating the speed at which a user's eye (320) moves. FIG. 14 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure calculating the speed at which a user's eye (320) moves.

[0129] The processor (230) of the portable display device (100) can capture an image of the user (320) using the camera (110). The processor (230) can obtain the position of the user's (320) eyes from the captured image.

[0130] The processor (230) can convert the domain representing the position of the user's (320) eyes from pixel units to millimeter units. The processor (230) can convert the domain representing the position of the user's (320) eyes from pixel units to millimeter units based on the field of view of the camera (110) and the resolution of the display (120) of the portable display device (100).

[0131] The processor (230) can calculate the position of the user's (320) eyes that has changed based on the acceleration of the portable display device (100) detected through the inertial measurement unit sensor (220). The processor (230) can calculate the position of the user's (320) eyes that has changed from the acceleration of the portable display device (100) by applying a double integration calculation method. In the first integration calculation, the processor (230) can calculate the speed at which the user's (320) eyes move as shown in the following mathematical formula 12.

[0132]

[0133] v in mathematical equation 12 t may be the speed of the user's (320) eyes after the change. In Equation 12, v t-k may be the initial speed of the user's (320) eyes. In Equation 12, t may be the total time the user's (320) eyes were tracked. In Equation 12, k may be the time for performing sampling by tracking the user's (320) eyes. In Equation 12, a t may be the acceleration of the user's (320) eye after the change. In Equation 12, a t-k This could be the acceleration of the initial user's (320) eyes.

[0134] The processor (230) can calculate the position of the user's (320) eyes that have changed as shown in the following mathematical formula 13.

[0135]

[0136] x' in mathematical equation 13 t may be the position of the central part of the portable display device (100) after rotation. x' in Equation 13 t-k may be the position of the central part of the initial portable display device (100). In Equation 13, t may be the total time the user (320)'s eyes were tracked. In Equation 13, k may be the time for sampling to be performed by tracking the user (320)'s eyes. In Equation 13, v t may be the speed of the user's (320) eyes after the change. In Equation 13, v t-k This could be the speed of the initial user's (320) eyes.

[0137] The processor (230) can restore the domain representing the changed user (320)'s eye position from millimeter units to pixel units. The processor (230) can restore the domain representing the changed user (320)'s eye position from millimeter units to pixel units based on the field of view of the camera (110) and the resolution of the display (120) of the portable display device (100).

[0138] The processor (230) can calculate the speed at which the user's (320) eyes move by subtracting the changed user's (320) eye position obtained from the initially captured image from the changed user's (320) eye position obtained from Equation 13.

[0139] FIG. 15 is a diagram showing that a portable display device (100) according to one embodiment of the present disclosure primarily predicts the position of a user's eyes.

[0140] A processor (230) of a portable display device (100) can obtain the position of the user's eyes using a user image (1510) at the current time point (t) using a camera (110). For example, the processor (230) can obtain the position of the user's eyes at the current time point (t) using a camera (110) that obtains images at 30 frames per second (30 FPS). The processor (230) can predict multiple future situations. Based on the multiple future situations, the processor (230) can generate a predicted image (1520) of a target time point (t+n) to be predicted. Based on the predicted image (1520), the processor (230) can predict the position of the user's eyes.

[0141] The processor (230) can predict a first future situation at a target time point (t+n) to be predicted. The processor (230) can predict a second future situation at a time point (t+n-0.5) 0.5 frames earlier than the target time point (t+n). The processor (230) can predict a third future situation at a time point (t+n+0.5) 0.5 frames later than the target time point (t+n).

[0142] The processor (230) can predict the position of the user's eyes in each of the first future situation, the second future situation, and the third future situation. For example, when the processor (230) acquires 30 frames per second of images from the camera (110), it can predict the position of the user's eyes at each of the target time point, the time point 1 / 60 second before the target time point, and the time point 1 / 60 second after the target time point. The processor (230) can predict the position of the user's eyes at the target time point (t+n) based on the prediction results of each of the first future situation, the second future situation, and the third future situation.

[0143] FIG. 16 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure correcting an eye position predicted using the inter-pupillary distance (IPD) (1610).

[0144] The processor (230) of the portable display device (100) can predict the position of each of the user's two eyes (131, 132). The processor (230) can calculate the interpupillary distance (1610), which is the distance between the predicted user's two eyes (131, 132).

[0145] After calculating the interpupillary distance (1610), the processor (230) can calculate a vertical vector (1620) perpendicular to the interpupillary distance (1610). The vertical vector (1620) may be a vector that vertically bisects the line forming the interpupillary distance (1610). The vertical vector (1620) may represent the direction of the user's gaze.

[0146] The processor (230) can correct the predicted eye position using the interpupillary distance (1610) and the vertical vector (1620). The processor (230) can correct the error in the result value of predicting the user's eye position at the target time point (t+n).

[0147] FIG. 17 is a diagram showing that a portable display device (100) according to one embodiment of the present disclosure secondarily predicts the position of the user's eyes.

[0148] The processor (230) of the portable display device (100) can obtain the position of the user's eyes using the user image (1710) at the current time point (t) using the camera (110). For example, the processor (230) can obtain the position of the user's eyes at the current time point (t) using the camera (110) which obtains images at 30 frames per second (30 FPS). The processor (230) can primarily predict the position of the user's eyes at the target time point (t+n).

[0149] The processor (230) can obtain IMU information from the inertial measurement unit sensor (220). The IMU information may include information related to the orientation of the portable display device (100). The IMU information may include the direction, acceleration, and angular velocity values ​​of the portable display device (100). The processor (230) can apply the IMU information to the result of primarily predicting the position of the user's eyes at a target time point (t+n). The processor (230) can secondarily predict the position of the user's eyes at a target time point (t+n) based on the IMU information. The processor (230) can obtain a result image (1720) reflecting the result of secondarily predicting the user's eyes.

[0150] The processor (230) can apply IMU information at a rate higher than the frame rate per second of the camera (110). For example, the processor (230) can acquire IMU information from the inertial measurement unit sensor (220) at 60 frames per second (60 FPS). For example, the processor (230) can apply IMU information at 60 frames (60 FPS) to the result of a primary prediction of the user's eye position. For example, the processor (230) can secondarily predict the user's eye position by applying IMU information to the result of a primary prediction of the user's eye position at each of the target time point, the time point 1 / 60 second before the target time point, and the time point 1 / 60 second after the target time point. Accordingly, the processor (230) can predict the user's eye position more accurately by applying IMU information to the result of a primary prediction.

[0151] FIG. 18 is a flowchart illustrating a method for a portable display device (100) according to one embodiment of the present disclosure to display a stereoscopic image in accordance with the position of the user's eyes.

[0152] In operation 1810, a portable display device (100) according to one embodiment of the present disclosure can acquire the movement of a user using a camera (110). The camera (110) of the portable display device (100) can photograph a user located in front of the portable display device (100). The camera (110) can detect the movement of the user in the captured image of the user. A processor (230) can acquire the movement of the user detected by the camera (110).

[0153] In operation 1820, a portable display device (100) according to one embodiment of the present disclosure can acquire the movement of the portable display device (100) using an inertial measurement unit sensor (220). The inertial measurement unit sensor (220) can acquire IMU information including information related to the movement of the portable display device (100). A processor (230) of the portable display device (100) can acquire IMU information from the inertial measurement unit sensor (220). The processor (230) can acquire the movement of the portable display device (100). For example, the processor (230) can acquire the movement of the portable display device (100) after approximately 95 ms has elapsed since acquiring the movement of the user.

[0154] In operation 1830, a portable display device (100) according to one embodiment of the present disclosure can reflect IMU information in the view-mapping of a stereoscopic image. A processor (230) of the portable display device (100) can predict the position of the user's eyes at a target viewpoint using the user's movement, the movement of the portable display device (100), and IMU information. The processor (230) can map the viewpoint of the stereoscopic image to correspond to the predicted position of the user's eyes. For example, the processor (230) can map the viewpoint of the stereoscopic image after approximately 107 ms has elapsed since acquiring the user's movement. The processor (230) can display the stereoscopic image through the display (120) to match the mapped viewpoint of the stereoscopic image. Accordingly, the processor (230) can reduce crosstalk phenomena occurring in the stereoscopic image by mapping the viewpoint of the stereoscopic image to correspond to the predicted position of the user's eyes.

[0155] FIG. 19 is a flowchart illustrating a method in which a portable display device (100) according to one embodiment of the present disclosure corrects the position of a user's eye using accumulated IMU information and displays a stereoscopic image to match the corrected position of the eye.

[0156] In operation 1910, a portable display device (100) according to one embodiment of the present disclosure can detect the position of the eyes and remove noise. A processor (230) of the portable display device (100) can capture a user using a camera (110). The processor (230) can detect the position of the user's eyes in an image captured by the camera (110). The processor (230) can determine that parts of the image captured by the camera (110), excluding the user's face, are noise. The processor (230) can remove the parts determined to be noise from the image captured by the camera (110).

[0157] In operation 1920, a portable display device (100) according to one embodiment of the present disclosure can correct a pose in a first axis direction. The first axis direction may be an axis direction perpendicular to the display (120) of the portable display device (100) (more specifically, a plane defining the display screen of the display (120)). The first axis direction may be a direction toward the portable display device (100) from the position of the user's eyes. An error may occur in the pose in the first axis direction when the user's eyes look at the portable display device (100) at an angle. A processor (230) of the portable display device (100) can correct the pose in the first axis direction by applying a mathematical model. For example, the processor (230) can correct the pose in the first axis direction by applying a pinhole model.

[0158] In operation 1930, a portable display device (100) according to one embodiment of the present disclosure can acquire IMU information. The processor (230) of the portable display device (100) can acquire IMU information using an inertial measurement unit sensor (220).

[0159] In operation 1940, a portable display device (100) according to one embodiment of the present disclosure can accumulate IMU information. A processor (230) of the portable display device (100) can periodically acquire IMU information from an inertial measurement unit sensor (220). The processor (230) can store the periodically acquired IMU information in a memory (240). The processor (230) can add newly acquired IMU information to the IMU information stored in the memory (240).

[0160] In operation 1950, a portable display device (100) according to one embodiment of the present disclosure can correct the position of the user's eyes using accumulated IMU information. A processor (230) of the portable display device (100) can primarily predict the position of the user's eyes for a plurality of situations. The processor (230) can apply the accumulated IMU information to the result of the primary prediction. The processor (230) can secondarily predict the position of the user's eyes by applying the accumulated IMU information. For example, when the portable display device (100) moves, the processor (230) can predict the position of the user's eyes by reflecting the degree of movement of the portable display device (100) based on the accumulated IMU information.

[0161] In operation 1960, a portable display device (100) according to one embodiment of the present disclosure can map the viewpoint of a stereoscopic image based on a corrected eye position. For example, when the portable display device (100) moves, the processor (230) of the portable display device (100) can map the viewpoint of a stereoscopic image based on the predicted eye position of the user, reflecting the result of the portable display device (100) moving.

[0162] In operation 1970, a portable display device (100) according to one embodiment of the present disclosure can periodically acquire camera (110) capture and IMU information to monitor the position of the eyes. A processor (230) of the portable display device (100) can acquire an image of the user captured from the camera (110) at designated intervals. A processor (230) can acquire IMU information from an inertial measurement unit sensor (220) at designated intervals. A processor (230) can periodically predict the position of the user's eyes based on the image and IMU information acquired at designated intervals. Accordingly, the processor (230) can continuously and accurately predict the position of the user's eyes even in an environment where the portable display device (100) or the user is moving, and map the viewpoint of the stereoscopic image based on the predicted position of the user's eyes.

[0163] FIG. 20 is a drawing showing that a portable display device (110) according to one embodiment of the present disclosure corrects the position of a user's eyes using accumulated IMU information.

[0164] The processor (230) of the portable display device (110) can capture an image of the user at the current time (t) using the camera (110). The processor (230) can obtain a first prediction result (2010) that primarily predicts the position of the user's eyes at the target time (t+n).

[0165] The processor (230) can periodically acquire IMU information (2020) using an inertial measurement unit sensor (220). The processor (230) can accumulate the periodically acquired IMU information (2020). The processor (230) can apply the accumulated IMU information (2020) to a primary prediction result (2010). The processor (230) can secondarily predict the position of the user's eyes using the IMU information (2020).

[0166] The processor (230) can periodically capture an image of the user using the camera (110). The processor (230) can periodically apply IMU information (2020) to the captured image. Accordingly, the processor (230) can continuously and accurately predict the position of the user's eyes even in a portable display device (100) or an environment where the user is moving.

[0167] The processor (230) can map the viewpoint of the stereoscopic image to correspond to the position of the user's eyes, which is continuously and accurately predicted. The processor (230) can display the stereoscopic image on the display (120) according to the mapped viewpoint of the stereoscopic image. Accordingly, the processor (230) can reduce crosstalk occurring in the stereoscopic image.

[0168] FIG. 21 is a flowchart illustrating a method in which a portable display device (100) according to one embodiment of the present disclosure displays a stereoscopic image to match the characteristics of a user's movement.

[0169] In operation 2110, a portable display device (100) according to one embodiment of the present disclosure can acquire a plurality of features from the movement of a user. The plurality of features may be parameter values ​​representing the user's movement in space. For example, the plurality of features may include the speed of the user's eyes, the acceleration of the user's eyes, and the direction in which the user's eyes are looking. A processor (230) of the portable display device (100) can photograph the user using a camera (110). The processor (230) can detect the user's movement in the captured image. The processor (230) can extract a plurality of features from the result of detecting the user's movement.

[0170] In operation 2120, a portable display device (100) according to one embodiment of the present disclosure can perform compensation for each of a plurality of features of a user's movement. A processor (230) of the portable display device (100) can perform compensation for each of the plurality of features based on the relationship between the extracted plurality of features. For example, the processor (230) can compensate for the position of the user's eyes based on the relationship between the extracted plurality of features.

[0171] In operation 2130, a portable display device (100) according to one embodiment of the present disclosure may reflect the result of performing compensation in the viewpoint mapping of a stereoscopic image. A processor (230) of the portable display device (100) may map the viewpoint of the stereoscopic image to correspond to the position of the user's eyes that has been compensated. The processor (230) may display the stereoscopic image through the display (120) to match the viewpoint of the mapped stereoscopic image. Accordingly, the processor (230) may provide a stereoscopic image corresponding to the position of the user's eyes that has been compensated, thereby reducing crosstalk.

[0172] FIG. 22 is a drawing showing that a portable display device (100) according to one embodiment of the present disclosure compensates for the position of the user's eyes to match the characteristics of the user's movement.

[0173] The inertial measurement unit sensor (220) of the portable display device (100) can detect movement of the user's face in the yaw direction, pitch direction, and roll direction. The inertial measurement unit sensor (220) can provide detection results related to movement in the yaw direction, pitch direction, and roll direction to the processor (230). For example, if the user's face rotates in the yaw direction and pitch direction, the inertial measurement unit sensor (220) can provide the processor (230) with a result of detecting that the user's face has rotated in the yaw direction and pitch direction.

[0174] The processor (230) can receive the results of detecting the movement of the user's face in the yaw, pitch, and roll directions from the inertial measurement unit sensor (220). Based on the detected results, the processor (230) can compensate for the movement of the user's face. For example, the processor (230) can compensate for the movement of the user's face rotating in the yaw and pitch directions. The processor (230) can compensate for the predicted position of the user's eyes based on the compensated movement of the user. The processor (230) can map the viewpoint of the stereoscopic image to correspond to the compensated position of the user's eyes. The processor (230) can display the stereoscopic image through the display (120) to match the viewpoint of the mapped stereoscopic image. Accordingly, the processor (230) can reduce crosstalk by providing a stereoscopic image corresponding to the compensated position of the user's eyes.

[0175] FIG. 23 is a drawing showing a portable display device (100) according to one embodiment of the present disclosure predicting the position of a user's eyes and measuring the result of displaying a stereoscopic image according to the predicted result.

[0176] A portable display device (100) can acquire a user image (2310) at a current time point (t) using a camera (110). The processor (230) of the portable display device (100) can primarily predict the position of the user's eyes at a time point adjacent to a target time point (t+n) (e.g., a time point 0.5 frames different from the target time point (t+n)) based on the acquired user image (2310).

[0177] A portable display device (100) can acquire IMU information using an inertial measurement unit sensor (220). A processor (230) can secondarily predict the position of the user's eyes at a target time point (t+n) based on the IMU information. The processor (230) can acquire a result image (2320) of the secondarily predicted position of the user's eyes at a target time point (t+n).

[0178] The processor (230) can measure the position of the user's eyes at a specific point in time after the target point in time (t+n) (e.g., a point in time 6 frames after the target point in time (t+n+6)). The processor (230) can acquire a measurement image (2330) representing the position of the user's eyes measured at the specific point in time.

[0179] The processor (230) can set the difference value between the user's eye position measured in the measurement image (2330) and the user's eye position secondarily predicted in the result image (2320) as the error value.

[0180] Experimental results showed that the error value obtained from the processor (230) was reduced in the portable display device (100) according to the present disclosure. For example, based on a camera (110) that obtains 30 frames per second (30 FPS), when the position of the user's eyes is predicted using only the image obtained from the camera (110) according to a comparative example, the error value was 16.43 at 418 frames, 19.79 at 371 frames, 13.67 at 536 frames, and the maximum error value was 46.93. On the other hand, when predicting the position of the user's eyes using both the image obtained from the camera (110) and the IMU information obtained from the inertial measurement unit sensor (220) according to the present disclosure, based on a camera (110) that acquires 30 frames per second (30 FPS), the error value was 16.14 at 418 frames, 19.73 at 371 frames, 13.50 at 536 frames, and the maximum error value was 45.68.

[0181] According to the present disclosure, when the position of a user's eye is predicted using both the image obtained from the camera (110) and the IMU information obtained from the inertial measurement unit sensor (220), it can be seen that the error value, which is the difference between the position of the user's eye measured in the measurement image (2430) and the position of the user's eye secondarily predicted in the result image (2420), is reduced. Accordingly, according to the present disclosure, the position of the user's eye can be predicted more accurately, and the crosstalk phenomenon can be reduced by providing a stereoscopic image that matches the predicted position of the user's eye.

[0182] FIG. 24 is a graph showing the maximum error between the point in time when a portable display device (100) according to a comparative example and an embodiment of the present disclosure displays a stereoscopic image and the position of the user's eyes in various situations.

[0183] The portable display device (100) can display a stereoscopic image in a first situation, a second situation, a third situation, a fourth situation, and a fifth situation. For example, the first situation may be a situation where the portable display device (100) is placed on a stand of the portable display device (100). For example, the second situation may be a situation where the portable display device (100) moves in a direction perpendicular to the user while on the stand of the portable display device (100). For example, the third situation may be a situation where the user's face rotates while the portable display device (100) is on the stand of the portable display device (100). For example, the fourth situation may be a situation where the portable display device (100) is held in the user's hand. For example, the fifth situation may be a situation where the portable display device (100) moves in a direction perpendicular to the user while on the user's hand. The portable display device (100) can produce a result by combining the first situation, the second situation, the third situation, the fourth situation, and the fifth situation.

[0184] The processor (230) of the portable display device (100) can calculate the maximum error between the point in time when the portable display device (100) displays a stereoscopic image and the position of the user's eyes in various situations. The maximum error may be the distance between the position of the user's eyes predicted by the processor (230) and the actual position of the user's eyes.

[0185] According to the comparative example, when the position of the user's eyes is predicted using only the image obtained from the camera (110), the maximum error according to the calculated comprehensive result may be 94.90. On the other hand, according to the embodiment of the present disclosure, when the position of the user's eyes is predicted using both the image obtained from the camera (110) and the IMU information obtained from the inertial measurement unit sensor (220), the maximum error according to the calculated comprehensive result may be 37.96.

[0186] According to the present disclosure, when the position of the user's eyes is predicted using both the image obtained from the camera (110) and the IMU information obtained from the inertial measurement unit sensor (220), it can be confirmed that the maximum error is reduced. Accordingly, according to the present disclosure, the position of the user's eyes can be predicted more accurately, and the crosstalk phenomenon can be reduced by providing a stereoscopic image that matches the predicted position of the user's eyes.

[0187] The present disclosure aims to provide a technology that reduces crosstalk in stereoscopic images by accurately measuring the distance between a portable display device and an eye even when a light field display is applied to a portable display device, and accurately predicting the position of the user's eye even when at least one of the portable display device and the user moves.

[0188] A portable display device for displaying stereoscopic images according to the present disclosure comprises: a camera (110); an inertial measurement unit (IMU) sensor (220); a memory (240) for storing at least one instruction; and at least one processor (230) operably coupled to the memory (240) and including processing circuitry, wherein the at least one processor (230) executes the at least one instruction individually or collectively, thereby enabling the portable display device (100) to detect the position of a user's eye through the camera (100), correct the pose of the user's eye, obtain IMU information related to the orientation of the portable display device (100) through the inertial measurement unit sensor (220), primarily predict the position of the user's eye for a plurality of future situations, and secondarily predict the position of the user's eye using the IMU information.

[0189] In one embodiment of the present disclosure, the IMU information may include the movement of the portable display device (100), the movement of the user's eyes, and the angle between the portable display device (100) and the user's line of sight.

[0190] In one embodiment of the present disclosure, by having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) can detect the position of the user's eye and the direction of the eye's gaze, calculate the first position of the user's eye by assuming that the direction of the user's eye's gaze is a first axis direction facing vertically toward the portable display device, correct the first position by correcting the distance in the first axis direction from the portable display device (100) to the first position, and set the second position so that the user's eye is parallel to the portable display device (100).

[0191] In one embodiment of the present disclosure, by having the at least one processor (230) execute the at least one instruction individually or collectively, the portable display device (100) can capture the user using the camera, extract a plurality of three-dimensional landmarks from the captured user's face, and estimate the user's pose based on the plurality of three-dimensional landmarks.

[0192] In one embodiment of the present disclosure, by having the at least one processor (230) execute the at least one instruction individually or collectively, the portable display device (100) can predict the speed of the user's eyes.

[0193] In one embodiment of the present disclosure, by having the at least one processor (230) execute the at least one instruction individually or collectively, the portable display device (100) can predict a first future situation at a target time point to be predicted, a second future situation at a time point 0.5 frames earlier than the target time point, and a third future situation at a time point 0.5 frames later than the target time point, and predict the position of the user's eye in each of the first future situation, the second future situation, and the third future situation.

[0194] In one embodiment of the present disclosure, by having the at least one processor (230) execute the at least one instruction individually or collectively, the portable display device (100) can correct the position of the eye that was initially predicted using the interpupillary distance (IPD).

[0195] In one embodiment of the present disclosure, by having the at least one processor (230) execute the at least one instruction individually or collectively, the portable display device (100) may apply the IMU information to each of the plurality of future situations and compare the results of the application to obtain the secondarily predicted position of the eye.

[0196] In one embodiment of the present disclosure, by having the at least one processor (230) execute the at least one instruction individually or collectively, the portable display device (100) can accumulate the IMU information and use the accumulated IMU information to correct the position of the eye.

[0197] In one embodiment of the present disclosure, by having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) may acquire a plurality of features from the movement of the user, perform compensation for each of the plurality of features, and reflect the result of the compensation in the viewpoint mapping of the stereoscopic image.

[0198] A control method for a portable display device that displays a stereoscopic image according to the present disclosure may include: detecting the position of a user's eye through a camera of the portable display device; correcting the pose of the user's eye through the portable display device; acquiring IMU information related to the orientation of the portable display device through an inertial measurement unit (IMU) sensor of the portable display device; primarily predicting the position of the user's eye for a plurality of future situations through the portable display device; and secondarily predicting the position of the user's eye using the IMU information through the portable display device.

[0199] In one embodiment of the present disclosure, the operation (420) for correcting the pose may include: detecting the position of the user’s eye and the direction of the eye’s gaze; calculating a first position of the user’s eye by assuming that the direction of the user’s eye’s gaze is a first axis direction that is vertically oriented toward the portable display device; correcting the first position by correcting the distance in the first axis direction from the portable display device to the first position; and setting a second position so that the user’s eye is parallel to the portable display device.

[0200] In one embodiment of the present disclosure, the operation (420) for correcting the pose may include: an operation (710) of capturing the user using the camera; an operation (720) of extracting a plurality of three-dimensional landmarks from the captured face of the user; and an operation (730) of estimating the pose of the user based on the plurality of three-dimensional landmarks.

[0201] In one embodiment of the present disclosure, a control method for a portable display device that displays a stereoscopic image may include an operation of predicting the speed of the user's eyes.

[0202] In one embodiment of the present disclosure, the operation (440) of primarily predicting the position of the user's eye for the plurality of future situations may include: an operation of predicting a first future situation at a target time point to be predicted, a second future situation at a time point 0.5 frames earlier than the target time point, and a third future situation at a time point 0.5 frames later than the target time point; and an operation of predicting the position of the user's eye in each of the first future situation, the second future situation, and the third future situation.

[0203] In one embodiment of the present disclosure, a control method for a portable display device that displays a stereoscopic image may include an operation of correcting the position of the eye that was primarily predicted using the interpupillary distance (IPD).

[0204] In one embodiment of the present disclosure, the operation (450) of secondarily predicting the position of the user's eye using the IMU information may include applying the IMU information to each of the plurality of future situations and comparing the results of the application to obtain the secondarily predicted position of the eye.

[0205] In one embodiment of the present disclosure, a control method for a portable display device that displays a stereoscopic image may include: an operation of accumulating the IMU information; and an operation of correcting the position of the eye using the accumulated IMU information.

[0206] In one embodiment of the present disclosure, a control method for a portable display device that displays a stereoscopic image may include: an operation (2110) of acquiring a plurality of features from the movement of the user; an operation (2120) of performing compensation for each of the plurality of features; and an operation (2130) of reflecting the result of performing compensation in the viewpoint mapping of the stereoscopic image.

[0207] According to the present disclosure, the position of the user's eyes can be accurately predicted by utilizing both the image acquired from the camera and the IMU information acquired from the inertial measurement unit sensor, thereby reducing crosstalk phenomena occurring in stereoscopic images.

[0208] A method according to one embodiment of the present disclosure may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present disclosure, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0209] Some embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include both computer storage media and communication media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. A communication medium typically includes computer-readable instructions, data structures, program modules, or other data of modulated data signals such as carrier waves, or other transmission mechanisms, and includes any information transmission medium. Additionally, some embodiments of the present disclosure may also be implemented as a computer program or computer program product containing computer-executable instructions, such as a computer program executed by a computer.

[0210] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0211] According to one embodiment of the present disclosure, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Claims

1. In a portable display device (100) that displays a stereoscopic image, Camera (110); Inertial measurement unit (IMU) sensor (220); Memory (240) for storing at least one instruction; and It includes at least one processor (230) operably coupled to the memory (240) and including a processing circuitry, and By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) The position of the user's eyes is detected through the above camera (100), and Correcting the pose of the above user's eyes, and IMU information related to the orientation of the portable display device (100) is obtained through the inertial measurement unit sensor (220), and Predicting the position of the user's eye in multiple future situations in a primary manner, A portable display device (100) that secondarily predicts the position of the user's eye using the above IMU information.

2. In Paragraph 1, The above IMU information is, A portable display device (100) comprising the movement of the portable display device (100), the movement of the user's eyes, and the angle between the portable display device (100) and the user's line of sight.

3. In either Paragraph 1 or Paragraph 2, By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) Detecting the position of the user's eye and the direction of the eye's gaze, Calculate the first position of the user's eye by assuming that the direction of the user's eye's gaze is a first axis direction that is vertically directed toward the portable display device (100), and The distance in the first axis direction from the above portable display device (100) to the first position is corrected to correct the first position, and A portable display device (100) that sets a second position so that the user's eye is parallel to the portable display device (100).

4. In at least one of paragraphs 1 to 3, By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) The user is captured using the camera (110), and Extract multiple 3D landmarks from the captured face of the above user, and A portable display device (100) that estimates the pose of the user based on the plurality of three-dimensional landmarks.

5. In at least one of paragraphs 1 to 4, By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) A portable display device (100) that predicts the speed of the user's eyes.

6. In at least one of paragraphs 1 to 5, By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) Predicting a first future situation at a target time point to be predicted, a second future situation at a time point 0.5 frames prior to the target time point, and a third future situation at a time point 0.5 frames after the target time point, and A portable display device (100) that predicts the position of the user's eye in each of the first future situation, the second future situation, and the third future situation.

7. In at least one of paragraphs 1 to 6, By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) A portable display device (100) that corrects the position of the eye primarily predicted using the interpupillary distance (IPD).

8. In at least one of paragraphs 1 to 7, By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) A portable display device (100) that applies the IMU information to each of the above multiple future situations and compares the results of the application to obtain the position of the eye that is secondarily predicted.

9. In at least one of claims 1 to 8, By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) Accumulate the above IMU information, and A portable display device (100) that corrects the position of the eye using the accumulated IMU information.

10. In at least one of claims 1 to 9, By having at least one processor (230) execute at least one instruction individually or collectively, the portable display device (100) Acquiring multiple features from the movement of the above user, and Compensation is performed for each of the above plurality of features, and A portable display device (100) that reflects the result of performing the above compensation in the viewpoint mapping of the above stereoscopic image.

11. A method for controlling a portable display device that displays stereoscopic images, The operation of detecting the position of the user's eyes through the camera of the above portable display device (410); The above portable display device performs an operation (420) to correct the pose of the user's eyes; An operation (430) of obtaining IMU information related to the orientation of the portable display device through an inertial measurement unit (IMU) sensor of the portable display device; The operation (440) of the portable display device primarily predicting the position of the user's eye regarding a plurality of future situations; and A method comprising the operation (450) of the portable display device secondarily predicting the position of the user's eye using the IMU information.

12. In Paragraph 11, The above IMU information is, A method comprising the movement of the portable display device, the movement of the user's eyes, and the angle between the portable display device and the user's line of sight.

13. In either of Paragraphs 11 and 12, The action (420) for correcting the above pose is, An operation to detect the position of the user's eye and the direction of the eye's gaze; An operation to calculate the first position of the user's eye by assuming that the direction of gaze of the user's eye is a first axis direction facing vertically toward the portable display device; An operation to correct the first position by correcting the distance in the first axis direction from the portable display device to the first position; and A method comprising the action of setting a second position so that the user's eye is parallel to the portable display device.

14. In at least one of claims 11 to 13, The operation (440) of primarily predicting the position of the user's eye regarding the plurality of future situations is, An operation to predict a first future situation at a target time point to be predicted, a second future situation at a time point 0.5 frames prior to the target time point, and a third future situation at a time point 0.5 frames after the target time point; and A method comprising an operation to predict the position of the user's eye in each of the first future situation, the second future situation, and the third future situation.

15. In at least one of claims 11 to 14, The operation (450) of secondarily predicting the position of the user's eye using the above IMU information is, A method comprising the operation of applying the IMU information to each of the plurality of future situations and comparing the results of the application to obtain the secondarily predicted position of the eye.

Images