Method and system for rendering images in a virtual environment viewed on a head-mounted display

Abstract

A method for rendering stereoscopic images, the method comprising: receiving minimal and maximal viewing distances; determining an allowed foreground parallax and / or an allowed background parallax; determining an allowed total parallax; for the left and right images, determining a respective size at an image plane based on a conversion factor and the image size, the conversion factor ensuring that a total parallax is at most equal to the allowed total parallax; determining an image offset for laterally translating the pair of stereoscopic images at the image plane distance, the image offset ensuring that: a given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax; and / or a given background parallax at the image plane distance is at most equal to the allowed background parallax; and causing rendering of the pair of stereoscopic images at the image plane distance within the virtual 3D viewing space.

Description

METHOD AND SYSTEM FOR RENDERING IMAGES IN A VIRTUAL ENVIRONMENT VIEWED ON A HEAD-MOUNTED DISPLAYCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority on US Provisional Patent Application No. 63 / 720,813 filed on November 15, 2024, the content of which is incorporated herein by reference.TECHNICAL FIELD

[0002] The present disclosure relates to methods, systems, and non-transitory storage mediums for generating and rendering a pair of stereoscopic images in a virtual environment displayed to a user on a head-mounted display (HMD).BACKGROUND

[0003] Virtual Reality (VR) is exciting technology which provides a unique 3-dimensional (3D) viewing experience. A user of a VR headset or display can feel immersed in the viewing environment.

[0004] However, the viewing experience of stereoscopic media (photos or videos) in VR is not always without its share of drawbacks. For example, viewing problems include objects in images perceived as jumping out of the viewing area onto a viewer (thereby causing framing violations), images or objects shown in images perceived as not proper 3D reproduction of such images or objects, excessive parallax, etc. All those drawbacks render the viewing experience uncomfortable and cause visual and / or cognitive stress.

[0005] There is therefore a need for methods and systems for rendering stereoscopic images within a virtual environment.SUMMARY

[0006] One or more implementations of the present technology have been developed based on developers’ appreciation that a restricted type and configuration of capturecameras are normally used to achieve a functional stereoscopic image with the common stereoscopic viewing techniques described in the prior art. For example, two identical capture cameras having lens arrangements specific to stereoscopic capturing / filming, which further requires them to be precisely positioned and orientated with a specific setup (e.g., viewer’s eye), are usually required to obtain functional stereoscopic images of the scene. In fact, if these criteria are not strictly respected, the viewer can misperceive the depth and / or develop a negative physiological response.

[0007] One or more implementations of the present technology have been developed based on developers’ appreciation that when the parameters of the original sensors (i.e., cameras) having acquired the pair of images are unknown, as is the case with stereographs and photographs taken with unidentified cameras, and there are no algorithms or methods to determine them, an alternative method based on content analysis of the image content could be used.

[0008] One or more implementations of the present technology provide for generation of a stereoscopic images having a geometry correspondence (converging point of the viewer’s eyes) between the reality and the virtual display for using images captured with a wide range of capture camera types (standard, wide angle, fisheye, etc.) without needing a restricted preselected positioning as long as they share a common portion of their respective field of view.

[0009] Further, one or more implementations of the present technology provide for processing, rendering and display of a stereoscopic image dynamically when magnification is used such that the viewing experience remains comfortable to the user wearing the HMD.

[0010] In accordance with a first broad aspect, there is provided a method for rendering a pair of stereoscopic images in a virtual 3D viewing space, the pair of stereoscopic images comprising a left image and a right image each having a respective image size, the method being executed by at least one processor operatively connected to a headmounted display (HMD), the method comprising: receiving a minimal viewing distance and a maximal viewing distance associated with the virtual 3D viewing space of the HMD; determining at least one of an allowed foreground parallax and an allowed background parallax both at an image plane distance based on an interocular distance, theimage plane distance and at least one of the minimal viewing distance and the maximal viewing distance; determining an allowed total parallax based on the at least one of the allowed foreground parallax and the allowed background parallax; for each one of the left and right images, determining a respective viewing space size at the image plane distance based on a given conversion factor and the respective image size, the given conversion factor ensuring that a total parallax at the image place distance is at most equal to the allowed total parallax and being at most equal to a maximal conversion factor; determining an image offset for laterally translating the pair of stereoscopic images at the image plane distance, the image offset ensuring that at least one of: a given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax; and a given background parallax at the image plane distance is at most equal to the allowed background parallax; and causing rendering of the pair of stereoscopic images at the image plane distance within the virtual 3D viewing space of the HMD using the respective viewing space size and the image offset.

[0011] In some embodiments, the method further comprises determining an overlapping portion of the pair of stereoscopic images positioned at the image plane and being shifted based on the respective shift, said causing the rendering of the pair of stereoscopic images comprising causing rendering of at least a section of the overlapping portion of the pair of stereoscopic images.

[0012] In some embodiments, the allowed total parallax is equal to the difference between the allowed background parallax and the allowed foreground parallax.

[0013] In some embodiments, the method further comprises calculating the maximal conversion factor based on the allowed total parallax and a total image parallax.

[0014] In some embodiments, the method further comprises receiving a zoom value and calculating the given conversion factor based on the zoom value and the maximal conversion factor.

[0015] In some embodiments, the step of receiving the minimal viewing distance and the maximal viewing distance comprises calculating the minimal viewing distance and the maximal viewing distance based on a virtual screen distance, the virtual screen distance corresponding to a distance of a virtual screen perceived by a user wearing the HMD.

[0016] 7 In some embodiments, the step of determining the at least one of the allowed foreground parallax and the allowed background parallax comprises: determining the allowed foreground parallax based on the interocular distance, the image plane distance and the minimal viewing distance; and determining the allowed background parallax based on the interocular distance, the image plane distance and the maximal viewing distance, wherein the image offset is chosen to ensure that the given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax and the given background parallax at the image plane distance is at most equal to the allowed background parallax.

[0017] In some embodiments, the image plane distance is equal to the minimal viewing distance, the step of determining the at least one of the allowed foreground parallax and the allowed background parallax comprises determining the allowed background parallax based on the interocular distance, the image plane distance and the maximal viewing distance; and the step of determining the allowed total parallax comprises determining the allowed total parallax based on the allowed foreground parallax, the allowed total parallax being equal to the allowed foreground parallax.

[0018] In some embodiments, the image offset is chosen to ensure that the given background parallax at the image plane distance is equal to the allowed background parallax, thereby ensuring that a background object appears at the maximal viewing distance.

[0019] In some embodiments, the given conversion factor is equal to the maximal conversion factor so that the total parallax at the image place distance is equal to the allowed total parallax and the given foreground parallax at the image plane distance is equal to zero, thereby ensuring that a foreground object appears at the minimal viewing distance.

[0020] In some embodiments, the image offset is chosen to ensure that the given foreground parallax at the image plane distance is equal to zero, thereby ensuring that a foreground object appears at the minimal viewing distance.

[0021] In some embodiments, the given conversion factor is equal to the maximal conversion factor so that the total parallax at the image place distance is equal to the allowed total parallax, thereby ensuring that a background object appears at the maximalviewing distance.

[0022] In accordance with another broad aspect, there is provided a system for rendering a pair of stereoscopic images in a virtual 3D viewing space, the pair of stereoscopic images comprising a left image and a right image each having a respective image size, the system comprising: at least one processing unit; and a non-transitory storage medium storing computer-readable instructions thereon, the at least one processing unit, upon executing the computer-readable instructions, being configured for: receiving a minimal viewing distance and a maximal viewing distance associated with the virtual 3D viewing space of the HMD; determining at least one of an allowed foreground parallax and an allowed background parallax both at an image plane distance based on an interocular distance, the image plane distance and at least one of the minimal viewing distance and the maximal viewing distance; determining an allowed total parallax based on the at least one of the allowed foreground parallax and the allowed background parallax; for each one of the left and right images, determining a respective viewing space size at the image plane distance based on a given conversion factor and the respective image size, the given conversion factor ensuring that a total parallax at the image place distance is at most equal to the allowed total parallax and being at most equal to a maximal conversion factor; determining an image offset for laterally translating the pair of stereoscopic images at the image plane distance, the image offset ensuring that at least one of: a given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax; and a given background parallax at the image plane distance is at most equal to the allowed background parallax; and causing rendering of the pair of stereoscopic images at the image plane distance within the virtual 3D viewing space of the HMD using the respective viewing space size and the image offset.

[0023] In some embodiments, the at least one processing unit is further configured for determining an overlapping portion of the pair of stereoscopic images positioned at the image plane and being shifted based on the respective shift, said causing the rendering of the pair of stereoscopic images comprising causing rendering of at least a section of the overlapping portion of the pair of stereoscopic images.

[0024] In some embodiments, the allowed total parallax is equal to the difference between the allowed background parallax and the allowed foreground parallax.

[0025] In some embodiments, the at least one processing unit is further configured for calculating the maximal conversion factor based on the allowed total parallax and a total image parallax.

[0026] In some embodiments, the at least one processing unit is further configured for receiving a zoom value and calculating the given conversion factor based on the zoom value and the maximal conversion factor.

[0027] In some embodiments, the at least one processing unit is configured for calculating the minimal viewing distance and the maximal viewing distance based on a virtual screen distance, the virtual screen distance corresponding to a distance of a virtual screen perceived by a user wearing the HMD.

[0028] In some embodiments, the at least one processing unit is configured for: determining the allowed foreground parallax based on the interocular distance, the image plane distance and the minimal viewing distance; and determining the allowed background parallax based on the interocular distance, the image plane distance and the maximal viewing distance, wherein the image offset is chosen to ensure that the given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax and the given background parallax at the image plane distance is at most equal to the allowed background parallax.

[0029] In some embodiments, the image plane distance is equal to the minimal viewing distance, the at least one processing unit is configured for determining the allowed background parallax based on the interocular distance, the image plane distance and the maximal viewing distance; and the at least one processing unit is configured for determining the allowed total parallax based on the allowed foreground parallax, the allowed total parallax being equal to the allowed foreground parallax.

[0030] In some embodiments, the image offset is chosen to ensure that the given background parallax at the image plane distance is equal to the allowed background parallax, thereby ensuring that a background object appears at the maximal viewing distance.

[0031] In some embodiments, the given conversion factor is equal to the maximal conversion factor so that the total parallax at the image place distance is equal to the allowed total parallax and the given foreground parallax at the image plane distance isequal to zero, thereby ensuring that a foreground object appears at the minimal viewing distance.

[0032] In some embodiments, the image offset is chosen to ensure that the given foreground parallax at the image plane distance is equal to zero, thereby ensuring that a foreground object appears at the minimal viewing distance.

[0033] In some embodiments, the given conversion factor is equal to the maximal conversion factor so that the total parallax at the image place distance is equal to the allowed total parallax, thereby ensuring that a background object appears at the maximal viewing distance.

[0034] In accordance with a further broad aspect, there is provided a non-transitory storage medium storing computer-readable instructions thereon, the computer-readable instructions, upon being executed by at least one processor, are configured for causing: receiving a minimal viewing distance and a maximal viewing distance associated with the virtual 3D viewing space of the HMD; determining at least one of an allowed foreground parallax and an allowed background parallax both at an image plane distance based on an interocular distance, the image plane distance and at least one of the minimal viewing distance and the maximal viewing distance; determining an allowed total parallax based on the at least one of the allowed foreground parallax and the allowed background parallax; for each one of the left and right images, determining a respective viewing space size at the image plane distance based on a given conversion factor and the respective image size, the given conversion factor ensuring that a total parallax at the image place distance is at most equal to the allowed total parallax and being at most equal to a maximal conversion factor; determining an image offset for laterally translating the pair of stereoscopic images at the image plane distance, the image offset ensuring that at least one of: a given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax; and a given background parallax at the image plane distance is at most equal to the allowed background parallax; and causing rendering of the pair of stereoscopic images at the image plane distance within the virtual 3D viewing space of the HMD using the respective viewing space size and the image offset.

[0035] In some embodiments, the computer-readable instructions are configured for causing: determining an overlapping portion of the pair of stereoscopic images positionedat the image plane and being shifted based on the respective shift, said causing the rendering of the pair of stereoscopic images comprising causing rendering of the overlapping portion of the pair of stereoscopic images.BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Embodiments of the disclosure will be described by way of example only with reference to the accompanying drawings, in which:

[0037] FIG. 1 illustrates a schematic diagram of a head-mounted display (HMD) system in accordance with one or more non-limiting embodiments of the present technology.

[0038] FIG. 2 illustrates a diagram of a perspective view of viewing planes and distances in a virtual viewing space in accordance with one or more non-limiting embodiments of the present technology.

[0039] FIG. 3 illustrates a diagram of the left and right eyes of the user focusing on the farthest perceived object (at the farthest plane distance Df) in the stereoscopic image, when the image plane is located between the nearest convergence plane and the farthest convergence plane, in accordance with one or more non-limiting embodiments of the present technology.

[0040] FIG. 4 illustrates a diagram of the left and right eyes of the user focusing on the nearest perceived object (at the nearest plane distance Dn) in the stereoscopic image, when the image plane is located between the nearest convergence plane and the farthest convergence plane, in accordance with one or more non-limiting embodiments of the present technology.

[0041] FIG. 5 illustrates a flowchart of a method of rendering a pair of stereoscopic images, in accordance with one or more non-limiting embodiments of the present technology.

[0042] FIG. 6 illustrates a diagram of the left and right eyes of the user focusing on the farthest perceived object (at the farthest plane distance Dff) in the stereoscopic image, when the image plane is at the nearest convergence plane, in accordance with one or more non-limiting embodiments of the present technology.

[0043] FIG. 7 shows a diagram of the left and right eyes of the user focusing on the farthest perceived object (at the farthest plane distance Dff) in the stereoscopic image at a given conversion factor in accordance with one or more non-limiting embodiments of the present technology.

[0044] FIGS. 8 A and 8B illustrate a flowchart of a method of rendering a pair of stereoscopic images at a maximum conversion factor in accordance with one or more non-limiting embodiments of the present technology.

[0045] FIG. 9 illustrates a flowchart of a method of rendering a pair of images at a given conversion factor in accordance with one or more non-limiting embodiments of the present technology.DETAILED DESCRIPTION

[0046] The examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the present technology and not to limit its scope to such specifically recited examples and conditions. It will be appreciated that those skilled in the art may devise various arrangements which, although not explicitly described or shown herein, nonetheless embody the principles of the present technology and are included within its spirit and scope.

[0047] Furthermore, as an aid to understanding, the following description may describe relatively simplified implementations of the present technology. As the person skilled in the art would understand, various implementations of the present technology may be of a greater complexity.

[0048] In some cases, what are believed to be helpful examples of modifications to the present technology may also be set forth. This is done merely as an aid to understanding, and, again, not to define the scope or set forth the bounds of the present technology. These modifications are not an exhaustive list, and a person skilled in the art may make other modifications while nonetheless remaining within the scope of the present technology. Further, where no examples of modifications have been set forth, it should not be interpreted that no modifications are possible and / or that what is described is the sole manner of implementing that element of the present technology.

[0049] Moreover, all statements herein reciting principles, aspects, andimplementations of the present technology, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof, whether they are currently known or developed in the future. Thus, for example, it will be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present technology. Similarly, it will be appreciated that any flowchart, flow diagrams, state transition diagrams, pseudo-code, and the like represent various processes which may be substantially represented in computer-readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0050] The foregoing and other features will become more apparent upon reading of the following non-restrictive description of illustrative embodiments thereof, given by way of example only with reference to the accompanying drawings. Like numerals represent like features on the various drawings.

[0051] Various aspects of the present disclosure generally address the rendering of a pair of stereographic images in a virtual environment (VE), which may be used in the context of virtual reality (VR), mixed reality (MR) and, in some implementations, augmented reality (AR).

[0052] More particularly, the present disclosure provides a method, a system, and a non-transitory computer-readable medium comprising instructions executable by a processing unit for rendering a pair of images for viewing in a virtual environment created by an HMD.

[0053] One or more implementations of the present technology are directed to positioning, processing and rendering of a pair of stereoscopic images such that the resulting 3D stereoscopic image in the virtual viewing scene perceived by the user wearing the HMD is rendered in an improved manner, even when magnification is used, without knowledge of the acquisition parameters of the cameras having captured or acquired the pair of images.

[0054] The present technology provides for a spatial control of the positioning of different images planes to render and integrate different objects (e.g., menus) in a virtual environment such that it is comfortably perceived by the user. As a result, one or more implementations of the present technology enable reducing or minimizing potentialadverse effects associated with prolonged viewing of stereo imagery which may include eye tiredness, altered vision, lightheadedness, confusion, nausea, and even in some cases convulsions.

[0055] Thus, in the context of at least some implementations of the present technology, a stereoscopic image can be defined as a stereoscopic image giving the right perception of depth, magnitude, scale, etc. without causing a significant negative physiological response from the viewer, such as dizziness, headaches, any uncomfortable feeling, and the like.

[0056] It will be understood that stereoscopic images may be frames of a video.

[0057] One or more implementations of the present technology may be used in the fields of entertainment (e.g., cinema, television, video games), communication (e.g., mobile devices, video conferencing, scientific visualization), and medicine (e.g., diagnosis, surgical planning and control, medical instruction).

[0058] The following terminology is used throughout the present disclosure:

[0059] Stereoscopy: Technique used for generating an illusion of depth in an image being displayed.

[0060] Virtual Reality (VR): Three-dimensional image or environment generated by a processing unit.

[0061] Wearable VR display: Electronic device adapted to display a three- dimensional image or environment. Wearable VR displays include VR helmets, VR goggles, VR glasses, 3D helmets, 3D goggles, 3D lenses, MR helmets, MR goggles, MR glasses, AR helmets, AR goggles, AR glasses and any other wearable electronic device adapted to display a three-dimensional image or three- dimensional environment stereoscopically.

[0062] Virtual viewing space: Field of view generated by a wearable VR display.

[0063] The following variables and acronyms are used throughout the present disclosure:Bg: Distance between the center of a pair of images on an image plane;Df or Dff: Farthest or maximal viewing or convergence distance in the virtual viewing space;Dg or Dgo: Distance of the image or geometry plane where the pair of images will be formed or positioned in the virtual viewing space;Dn or Dnf: Nearest or minimal viewing or convergence distance in the virtual viewing space;Dvs: Distance of the virtual screen, which is function of a focal distance of the lenses of the wearable VR display;Io: Interocular distance of the subject wearing the wearable VR display; LgG: Width of the left image;LgD: Width of the right image;LgGm: Width of the left image at a given conversion factor M;LgDm: Width of the right image at a given conversion factor M;LgGmax: Width of the left image at the maximum conversion factor Mmax; LgDmax: Width of the right image at the maximum conversion factor Mmax; Pin: Foreground or near image parallax at the image plane;Pif: Background or far image parallax at the image plane;Pit: Total image parallax;Pgfo: Allowed background or far parallax at the image plane;Pgno: Allowed foreground or near parallax at the image plane; and Pgto: Allowed total parallax at the image plane.

[0064] Referring to FIG. 1, the head-mounted display (HMD) system 100 provides immersive virtual reality experiences by presenting stereoscopic images to the user. The HMD system 100 includes a housing (not shown) to be worn on the user's head, supporting various components for virtual environment generation. The housing may have adjustable straps for securing the system on the user’ s head for ergonomic use duringextended periods.

[0065] In some non-limiting implementations, the HMD system 100 includes at least two display units, i.e., a left display unit 135 and a right display 145 to be viewed respectively by the left and right eyes of the user. The display units are located inside the housing and project images that together form a stereoscopic view of the virtual environment. Each display unit may include, as a non-limiting example, LCD, LED / LCoS, or microdisplay comprising an OLED, MicroLED or MicroOLED panel or other suitable display technology capable of rendering high-resolution images.

[0066] The display units 135 and 145 are electronically connected to at least one processing unit 110, which is configured to generate the virtual environment and provide the appropriate image data to each display. The display units 135 and 145 are placed to align with the lenses and the user's eyes to provide stereoscopic image rendering.

[0067] Each display unit 135, 145 is paired with a lens assembly 130, 140. The lenses adjust the focal distance and magnify the images displayed on the display units 135, 145. The lenses 130, 140 are typically convex and are positioned between the user's eyes and the respective display units. The lenses 135 and 145 may be adjustable to accommodate variations in users’ interocular distances.

[0068] The HMD system 100 generates stereoscopic images by presenting different views of the virtual environment to the left and right eyes. Each display unit 135, 145 shows a slightly different perspective of the same scene, simulating how the human eyes perceive depth in the real world. This difference in perspectives, commonly referred to as "binocular disparity" allows the user to perceive depth, creating a 3D effect.

[0069] In some embodiments, the HMD system 100 includes an array of sensors (not shown), such as accelerometers, gyroscopes, and magnetometers, to track the user's head movements. The sensors are connected to the one or more processing units 100, which may be configured to adjust the display images in real-time based on the detected head orientation and movement. In some embodiments, the HMD system 100 includes eyetracking sensors that monitor the user’s eye movements. The data collected by the eyetracking sensors can be used to adjust the image rendering process, providing adaptive resolution, and enhancing the overall experience of the virtual environment.

[0070] The HMD system 100 includes one or more processing units 110 operativelyconnected to one or more memories 115 and / or one or more storage mediums 120.

[0071] The one or more processing units 110 are configured to load and execute computer-readable instructions to render stereoscopic images as described hereinafter.

[0072] Non-limiting examples of processing units 110 may include one or more of: CPUs, GPUs, DSPs, ASICS and / or SOCs. Non-limiting examples of memories 115 include RAM, flash storage, VRAM, ROM. Non-limiting examples of storage mediums 120 include SSDs, eMMCs, UFS, microSD, optical storage, network attached storage and cloud storage.

[0073] The one or more processing units 110, one or more memories 115 and / or one or more storage mediums 120 may be located externally and connected to the HMD via a wired or wireless connection. Alternatively, the one or more processing units 110, one or more memories 115 and / or one or more storage medium 120 may be integrated within the housing or components of the HMD system 100.

[0074] The HMD system 100 may also include an integrated audio system (not shown), which consists of speakers positioned near the user’s ears or inbuilt headphones. The audio system may be synchronized with the virtual environment displayed on the display units 135 and 145 to enhance the immersive experience by providing spatial audio cues.

[0075] The HMD system 100 includes one or more communication interfaces (not shown) operatively connected to computing devices (not shown) and other components via wired or wireless communication links to transmit and / or receive data therefrom.

[0076] The HMD system 100 may also incorporate various input / output (I / O) interfaces (not shown), including controllers, buttons, and / or other interactive elements, configured to receive user interaction signals for the user to interact with the virtual environment. Non-limiting examples of IO interfaces buttons, joysticks, triggers, and touchpads that provide tactile feedback and fine control over actions. It will be appreciated that buttons or touch-sensitive areas may be integrated directly into the HMD housing to enable users to perform basic operations, such as adjusting volume, navigating through virtual interfaces (e.g., zooming), or powering the device on and off. The I / O interfaces may include gesture recognition sensors configured to detect hand movements or finger gestures as input and / or, haptic feedback mechanisms in the controllers or theHMD system 100.

[0077] FIG. 2 is a diagram of a perspective view of viewing planes and distances in a virtual viewing space 200 in accordance with one or more non-limiting embodiments of the present technology.

[0078] The virtual viewing space 200 corresponds to the 3D virtual environment generated and rendered on the display units 135, 145 of the HMD system 100 and perceived by the user. In FIG. 2, the user perceives a stereoscopic image 222 formed by a pair of superposed images which provides a maximum effect of perceived depth. It should be noted that FIG. 2 depicts planes and distances when maximum conversion is used, as will be explained hereinafter.

[0079] In FIG. 2, the virtual viewing space 200 includes a virtual camera plane 210, an image plane 220, a virtual screen plane 230, and a farthest convergence plane 240.

[0080] The virtual camera plane 210 corresponds to the plane where the virtual cameras 112 comprising a left camera 214 and right camera 216 are located. The virtual cameras 112 define a total field of view (FOV), where the left camera 214 defines a left FOV and the right camera 216 defines a right FOV. It will be appreciated that the left camera 214 corresponds to what is perceived by the left eye of the user on the left display unit 135 and the right camera 216 corresponds to what is perceived by the right eye of the user on the right display unit 145 via the respective lens 130, 140.

[0081] The distance between the left camera 214 and the right camera 216 corresponds to an interocular distance 218 (Io) of the eyes of the user wearing the HMD 105.

[0082] The stereoscopic image 222 is formed by a pair of images including a left image and right image (not shown separately) which are superposed and positioned (i.e., anchored) on the image or geometry plane 220 located at an image or geometry plane distance 228 (Dg).

[0083] It should be understood that in the context of the present technology, the stereoscopic image 222 is positioned on the image plane 220. However, the image plane distance 228 may vary based on optical parameters of the HMD 105 and the magnification used.

[0084] In some implementations, only the common or overlapping portions of the pairof images in the stereoscopic image 222 positioned at the image plane distance 228 (or at least a section of the overlapping portions) are rendered to be viewed by the user, with the left border of the left image and the right border of the right image being clipped to ensure an optimal stereoscopic effect and to reduce or minimize distortions when the stereoscopic image 222 is viewed by the user. It will be appreciated that the left and right images forming the stereoscopic image 222 are images of the same scene taken from left and right points of view, respectively.

[0085] Each image in the pair has respective dimensions, i.e., a height (y) and width (x), which may be expressed in pixels, e.g., 1024x768 pixels. It will be appreciated that the left and right images generally have the same dimension. In one or more alternative implementations, the left image and right image may have slight variations in their dimension, for example if the images were taken by using older types of cameras, different types of cameras, different acquisition parameters, different optics, etc.

[0086] The image plane 220 is located at the image plane distance 228 (Dg) from the camera plane 210. In some implementations, the nearest points in the stereoscopic image 222 are perceived by the user at the nearest convergence plane (corresponding to the image plane 220 at maximum conversion in FIG. 2) located at the nearest convergence plane distance (Dnf), and the farthest points in the stereoscopic image 222 are perceived at the farthest convergence plane 240 located at the farthest convergence plane distance 246 (Dff). It should be noted that in the context of the present technology, the image plane distance 228 is to be calculated based on a given conversion factor, as will be explained hereinafter.

[0087] The virtual screen plane 230 is located at the virtual screen distance 236 (Dvs) from the camera plane 210 and represents a screen in the virtual space perceived by the user wearing the HMD 105. The virtual screen distance 236 is determined based on the parameters of the HMD, including a focal length of the lenses used in the HMD 105.

[0088] In the context of the present disclosure, the virtual screen distance 236 is used to calculate the nearest convergence plane distance Dnf (corresponding to the image plane distance 228 in FIG. 2), the farthest convergence plane distance 146 (Dff) and the optimal image plane distance 228 (Dg) for comfortable viewing by the user. The virtual screen distance 236 is also used to calculate the maximum conversion factor Mmax (not shown)of the 3D stereoscopic image formed by the pair of images 222 in the virtual viewing space 200, such that the stereoscopic image does not overflow from the field of view and is comfortably perceived by the user. At maximum conversion, the optimal image plane distance 228 is equal to the nearest convergence plane distance (Dgo = Dnf), which is the scenario shown in FIG. 2.

[0089] The stereoscopic image 222 has a nearest parallax (Pin) (not shown) which corresponds to a parallax distance of the nearest perceived point in the stereoscopic image 222 and a farthest parallax (Pif), which corresponds to a parallax distance of the farthest perceived point in the image (not shown). The total image parallax is equal to a difference between the farthest image parallax and the nearest image parallax (Pit = Pif- Pin).

[0090] FIG. 3 illustrates an allowed background or far parallax Pgfo when an image plane 150 is positioned at an image plane distance Dg, i.e., the maximum background parallax at the image plane distance Dg for an object 152 to be viewed or perceived at the farthest viewing plane 154 (or farthest convergence plane 154) positioned at the farthest or maximum viewing distance Df.

[0091] The allowed background parallax Pgfo is then determined as follows.

[0092] The line 156 corresponds to the line that connects the right camera 158 to the object 152, and the line 160 corresponds to the line that connects the left camera 162 to the farthest perceived object 152. The line 156 intersects the image plane 150 at point 164 and the line 160 intersects the image plane 150 at point 166. The allowed background parallax Pgfo then corresponds to the difference between the lateral coordinate or position of the point 164 along the width of the image plane 150 and the lateral coordinate or position of the point 166 along the width of the image plane 150. Assuming that the lateral axis is oriented towards the right of FIG. 3, then the lateral coordinate of point 164 is greater than that of point 166, and the allowed background parallax Pgfo has a positive value.

[0093] The allowed background parallax Pgfo can then be determined using the following equation:Pgfo = Io * (Df - Dg) / Df

[0094] where Io is the interocular distance.

[0095] FIG. 4 illustrates an allowed foreground or near parallax Pgno when the image plane 150 is positioned at the image plane distance Dg, i.e., the minimal foreground parallax at the image plane distance Dg for an obj ect 170 to be viewed or perceived at the nearest viewing plane 172 (or nearest convergence plane 172) positioned at the nearest or minimal viewing distance Dn.

[0096] The allowed foreground parallax Pgno is then determined as follows.

[0097] The line 174 corresponds to the line that connects the right camera 158 to the object 170 and the line 176 corresponds to the line that connects the left camera 162 to the closest perceived object 170. The line 174 intersects the image plane 150 at point 180 and the line 176 intersects the image plane 150 at point 182. The allowed foreground parallax Pgno then corresponds to the difference between the lateral coordinate or position of the point 180 along the width of the image plane 150 and the lateral coordinate or position of the point 182 along the width of the image plane 150. Assuming that the width axis is oriented towards the right of FIG. 3, then the width coordinate of point 180 is less than that of point 182, and the allowed foreground parallax Pgno has a negative value.

[0098] The allowed foreground parallax Pgno can then be determined using the following equation:Pgno = Io * (Dg - Dn) / Dn

[0099] The total allowed parallax Pgto then corresponds to the difference between the allowed background or far parallax Pgfo and the allowed foreground or near parallax Pgno:Pgto = Pgfo - Pgno

[0100] It will be understood that if the orientation of the width axis is changed towards the left of FIGS . 3 and 4, then the allowed background or far parallax Pgfo has a negative value and the allowed foreground or near parallax Pgno has a positive value.

[0101] In this case, the total allowed parallax Pgto then corresponds to the difference between the allowed foreground or near parallax Pgno and the allowed background or far parallax Pgfo:Pgto = Pgno - Pgfo

[0102] so that the total allowed parallax Pgto has a positive value.

[0103] Now turning to FIG. 5, there is shown a flowchart of a method 250 of rendering a pair of stereoscopic images comprising a right image and a left image in a virtual 3D viewing space, in accordance with one or more non-limiting embodiments of the present technology.

[0104] The method 250 is executed to render a pair of stereoscopic images such that they can be viewed comfortably by a user wearing an HMD such as the HMD system 100.

[0105] The method 250 may be executed by at least one processing unit. The at least one processing unit may be, for example, a processing unit of a computing device operatively connected to the HMD system 100 or a processing unit 110 of the HMD system 100. In one or more other implementations, the method 250 may be executed in a distributed manner by a plurality of processing units.

[0106] In one or more implementations, the at least one processing unit is operatively connected to at least one non-transitory storage medium storing computer-readable instructions thereon. The at least one processing unit, upon loading the computer-readable instructions, is configured or operable to execute the method 200.

[0107] At step 252, a minimal viewing distance Dn and a maximal viewing distance Df associated with the virtual 3D viewing space of the HMD are received.

[0108] In some implementations, the step 252 consists in determining the minimal viewing distance Dn and the maximal viewing distance Df using the below described method for example.

[0109] At step 254, an allowed foreground parallax Pgno and / or an allowed background parallax Pgfo both at an image plane distance Dg is(are) determined based on an interocular distance Io, the image plane distance Dg and at least one of the minimal viewing distance Dn and the maximal viewing distance Df using the above-described method, i.e., Pgto = Pgfo - Pgno.

[0110] In some implementations in which the image plane is not located at the minimal viewing distance Dn or at the maximal viewing distance Df, both the allowed foreground parallax Pgno and an allowed background parallax Pgfo are determined at step 254.

[0111] In embodiments in which the image plane is not located at the minimal viewing distance Dn or at the maximal viewing distance Df, only one of the allowed foreground parallax Pgno and an allowed background parallax Pgfo is determined at step 204. If the image plane is located at the minimal viewing distance Dn, then the allowed foreground parallax Pgno is equal to zero and only the allowed background parallax Pgfo is determined at step 254. If the image plane is located at the maximal viewing distance Df, then the allowed background parallax Pgfo is equal to zero and only the allowed foreground parallax Pgno is determined at step 254.

[0112] At step 256, the allowed total parallax at the image plane distance Dg is determined based on at least one of the allowed foreground parallax Pgno and an allowed background parallax Pgfo.

[0113] In embodiments in which the image plane is not located at the minimal viewing distance Dn or at the maximal viewing distance Df, both the allowed foreground parallax Pgno and an allowed background parallax Pgfo are determined at step 204 and the allowed total parallax Pgto at the image plane distance Dg is determined base don both the allowed foreground parallax Pgno and an allowed background parallax Pgfo, as described above.

[0114] In embodiments in which the image plane is located at the minimal viewing distance Dn (Dg = Dn), then the allowed foreground parallax Pgno is equal to zero and the total allowed parallax at the image plane distance is determined only based on the allowed background parallax Pgfo, i.e., the allowed total parallax Pgto is equal to the allowed background parallax Pgfo (Pgto = Pgfo).

[0115] In embodiments in which the image plane is located at the maximal viewing distance Df (Dg = Df), then the allowed background parallax Pgfo is equal to zero and the total allowed parallax at the image plane distance is determined only based on the allowed foreground parallax Pgno, i.e., the allowed total parallax Pgto is equal to the inverse of the allowed foreground parallax Pgno (Pgto = -I * Pgno).

[0116] As described above, if the convention on the orientation of the lateral axis of the image plane is changed so as to be oriented towards the left of FIGS. 3 and, then the allowed total parallax at the image plane distance Dg is equal to the allowed foreground parallax Pgno at the image plane distance Dg minus the allowed background parallaxPgfo at the image plane distance Dg. If the image plane is located at the minimal viewing distance Dn, the allowed total parallax Pgto is equal to the inverse of the allowed background parallax Pgfo (Pgto = -1 * Pgfo). If the image plane is located at the maximal viewing distance Df, then the allowed total parallax Pgto is equal to the allowed foreground parallax Pgno (Pgto = Pgno).

[0117] At step 258, for each one of the left and right images, a respective size at the image plane distance Dg within the viewing space is determined based on a conversion or magnification factor M and the respective size of the left and right images, i.e. the size LgL for the left image at the plane distance Dg is determined based on the conversion or magnification factor M and the size LiL of the left image of the stereoscopic pair and the image size LgR for the right image at the plane distance Dg is determined based on the conversion or magnification factor M and the size LiR of the right image of the stereoscopic pair:LgL = M * LiLLgR = M * LiR

[0118] The conversion factor M allows for converting the size of an image into a given size for the image within the viewing space at the image plane distance Dg. For example, the conversion factor M may allow for converting the size of an image in pixels into a size for the image in the virtual 3D environment in meters or inches. The conversion factor M is greater than a minimal threshold value (such as zero) and at most equal to a maximal threshold Mmax. The size of the image within the viewing space is maximal when the conversion factor M is equal to Mmax. Furthermore, the conversion factor M is chosen so that the total parallax Pit (expressed in pixels) of the image at the image plane distance Dg is at most equal to the allowed total parallax Pgto (expressed in meters for example) at the image plane distance. When the orientation of the lateral axis of the image plane is oriented towards the right in FIGS. 3 and 4, the total image parallax Pit at the image plane distance is equal to the background image parallax Pif at the image plane distance Dg (which has a positive value) minus the foreground image parallax Pin at the image plane distance Dg (which has a negative value). However, if the convention on the orientation of the width axis of the image plane is changed so as to be oriented towards the left of FIGS. 3 and 4, then the total image parallax Pit at the image plane distance isequal to the foreground image parallax Pin at the image plane distance Dg (which has a positive value) minus the background image parallax Pif at the image plane distance Dg (which has a negative value).

[0119] In some embodiments, the conversion factor M is chosen to be equal to Mmax.

[0120] In some embodiments, the value of the conversion factor M is predefined.

[0121] In other embodiments, the method 250 further comprises a step of receiving a zoom value Z from the user and determining the value of the conversion factor M based on the zoom value Z and the maximal conversion factor Mmax, as follows:M = Z * Mmax

[0122] Where Z is greater than zero and at most equal to one.

[0123] In some implementations, the method 200 further comprises a step of determining the value of the maximal conversion factor Mmax based on the total parallax Pit of the image at the image plane distance Dg and the allowed total parallax at the image plane distance Dg, as follows:Mmax = Pgto / Pit

[0124] Referring back to FIG. 5, step 260 consists in determining a lateral offset or shift for translating the left and right images, one relative to the other, at the image plane distance Dg along the lateral axis of the image plane. For example, the lateral position of the left image may remain unchanged, and the lateral offset is to be applied to the right image only. In another example, the lateral position of the right image may remain unchanged, and the lateral offset is to be applied to the left image only. In a further example, both the left and right images may be translated in opposite directions so that the total relative lateral displacement of the images corresponds to the determined lateral offset. For example, the left image may be moved towards the left by a distance equal to half the determined lateral offset and the right image may be moved towards the right also by a distance equal to half the determined lateral offset, so that the total offset between the left and right images is equal to the determined lateral offset.

[0125] The lateral offset or shift is chosen so that at least one of the two following conditions be met:

[0126] Condition 1 : the image foreground parallax Pin at the image plane distance Dg is at least equal to the allowed foreground parallax Pgno (in this case, the lateral offset or shift is at least to a minimal offset or shift Smin); and

[0127] Condition 2: the image background parallax Pif at the image plane distance Dg is at most equal to the allowed background parallax Pgfo (in this case, the lateral offset or shift is at most equal to a maximal offset or shift Smax).

[0128] The minimal and maximal offsets or shifts Smin and Smax are obtained as follows:Smin = (Pgno - Pin * M) / 2Smax = (Pgfo - Pif * M) / 2

[0129] Where Pgno is the allowed foreground parallax at the image plane distance Dg, Pin is the image foreground parallax at the image plane distance Dg, M is the conversion factor, Pgfo is the allowed background parallax Pgfo at the image plane distance Dg, and Pif is the image background parallax at the image plane distance Dg.

[0130] In some embodiments, the lateral offset is chosen so that both Conditions 1 and 2 are met. In this case: S G [ (Pgno - Pin * M) / 2; (Pgfo - Pif * M) / 2], In some embodiments, the lateral offset is chosen so that the image foreground parallax Pin at the image plane distance Dg is equal to the allowed foreground parallax Pgno. In this case, a foreground or nearest object of the images appear at the minimal viewing distance Dn. In the same or other embodiments, the lateral offset is chosen so that the image background parallax Pif at the image plane distance Dg is equal to the allowed background parallax Pgfo. In this case, a background or farthest object of the images appear at the maximal viewing distance Df.

[0131] In some embodiments, the lateral offset is chosen so that only one of Conditions 1 and 2 is met. For example, the lateral offset may be chosen so that only the image foreground parallax Pin at the image plane distance Dg is at least equal to the allowed foreground parallax Pgno, as long as the total image parallax Pit at the image plane distance Dg is at most equal to the allowed total parallax Pgto at the image plane distance Dg. In another example, the lateral offset may be chosen so that only the image background parallax Pif at the image plane distance Dg is at most equal to the allowedbackground parallax Pgfo as long as the total image parallax Pit at the image plane distance Dg is at most equal to the allowed total parallax Pgto at the image plane distance Dg.

[0132] In some embodiments, the image plane is positioned at the minimal viewing distance Dn (Dg = Dn). In some embodiments, the lateral offset is chosen so that the image background parallax Pif at the image plane distance Dg is equal to the allowed background parallax Pgfo. In this case, the background or farthest object in the images appears at the maximal viewing distance Df. Furthermore, if the conversion factor M is chosen to be equal to Mmax, then the total image parallax Pit of the image plane distance Dg is equal to the allowed total parallax Pgto and a foreground or nearest object in the image appears at the minimal viewing distance Dn. In some embodiments, the lateral offset is chosen so that the image foreground parallax at the image plane distance Dg is equal to zero. In this case, a foreground or nearest object in the image appears at the minimal viewing distance Dn. Furthermore, if the conversion factor M is chosen to be equal to Mmax, then the image total parallax Pit at the image place distance Dg is equal to the allowed total parallax Pgto and therefore the image background parallax Pif at the image plane distance Dg is equal to the allowed total parallax Pgto, thereby ensuring that a background or farthest object in the images appears at the maximal viewing distance Df.

[0133] Referring back to FIG. 5, step 262 consists in causing the rendering of the pair of stereoscopic images at the image plane distance Dg within the virtual 3D viewing space of the HMD using the respective viewing space size of the left and right images and the lateral offset.

[0134] In some embodiments, step 262 consists in transmitting the determined size for the left and right images at the image plane distance Dg within the 3D viewing space and the determined lateral offset of the HMD. In this case, the processing unit of the HMD renders the left and right image at the image plane distance Dg within the virtual 3D viewing space, the left and right rendered images having the received size and being laterally and relatively translated by the lateral offset value.

[0135] In other embodiments, step 262 consists in controlling the processing unit of the HMD to cause the rendering of the images according to the determined respective sizeof the images and the lateral offset.

[0136] In other embodiments such as embodiments in which the method 200 is executed by the processor of the HMD, step 262 consists in rendering the images according to the determined respective size of the images and the lateral offset.

[0137] It will be understood that in at least some embodiments, the method 250 further comprises receiving the left and right images.

[0138] In the following, there is described an exemplary embodiment of the method 250 when the image plane is positioned at the nearest viewing distance Dn, i.e., Dg = Dn.

[0139] In addition to FIG. 2, reference is also made to FIG. 6, which shows a diagram of the left and right eyes of the user focusing on the farthest perceived object (at the farthest plane distance Dff) in the stereoscopic image, when the image plane corresponds to the nearest convergence plane (i.e., Dgo = Dnf). The total parallax in the stereoscopic image is equal to a difference between the farthest object parallax and the nearest object parallax (Pgto = Pgfo - Pgno).

[0140] In this scenario, the nearest object parallax in the stereoscopic image at the image plane (Pgno) is equal to zero, and the farthest object parallax in the stereoscopic image (Pgfo) may be determined.

[0141] Since (Dff-Dgo) / Pgfo = Dff-Io, and Pgfo = Io * (Dff - Dgo) / Dff, the total object parallax may be determined as being:Pgto = Pgfo = Io * (Dff - Dgo) / Dff

[0142] If the user wishes to magnify the 3D image (i.e., scale down or up) by a given zoom factor (Z), the size of the stereoscopic image changes, which affects the distance of the nearest and farthest convergence planes, the size of the perceived objects in the stereoscopic image and the perceived parallax.

[0143] When superposed images are magnified by a conversion factor M, the parallaxes defined by corresponding points on the two images are magnified by the same conversion factor M. Since Pit = Pif- Pin, the optimal total parallax on the image plane may be obtained using:Pit * Mmax = Pgto

[0144] Where Mmax is the maximum conversion expressed in pixels per virtual environment unit.

[0145] In the case of the unity engine, the virtual environment unit is equal to 1 meter, and Mmax may be expressed in meters per pixel. It will be appreciated that other values of virtual units may be used for other engines.

[0146] The maximum conversion factor may be obtained using the following equation:Mmax = (Dff- Dnf) * Io / ([Pif — Pin] * Dff)

[0147] The width of the left image at the maximum conversion factor (LgGmax) and the width of the right image at the maximum conversion factor (LgDmax) may thus be obtained using:LgGmax = LiG * MmaxLgDmax = Lid * Mmax

[0148] Since the optimal image plane distance (Dgo = Dnf) at maximum conversion, and the width of the pair of images at maximum conversion are known, the shift value (translation) between the pair of images may be determined based on the optimal total parallax at the image plane (Pgto) being equal to a difference between the maximum parallax (Pgfo) at the image plane and the minimum parallax (Pgno), where Pgno = 0. In this case, the optimal total parallax at the image plane (Pgto) is equal to the maximum parallax at the image plane (Pgfo). The shift may be calculated interchangeably based on the nearest parallax (Pin) or the farthest parallax (Pif)

[0149] Since Pif * Mmax - 2 * SHIFTgd = Pgfo, then:SHIFTgdo = (Pif * Mmax - Pgfo) / 2

[0150] Since Pin * Mmax - 2 * SHIFTgd = 0, then:SHIFTgdo = Pin * Mmax / 2

[0151] Thus, the pair of images may be positioned and rendered at the optimal image plane distance (Dgo = Dnf), with the left image at width LgGmax, the right image at width LgDmax being shifted by distance SHIFTgdo.

[0152] Conversion factor

[0153] If magnification (i.e., scale up or down) is used in the virtual space, magnification of the image geometry leads to a proportional change in parallaxes. It will be appreciated that a user may conceptually perceive a zoom as a change in the width of the images which results in a variation of the perceived depth. As a result, a zoom factor may be determined as follows:Z = M / Mmax; andM = Z * Mmax

[0154] with M being the conversion factor, which may be in the range from Mmin to Mmax. It should be noted that Mmin is a value greater than 0 and may be determined by operators of the present technology. Z is the zoom factor, which may be in the range from Mmin / Mmaxto 1.

[0155] FIG. 7 shows a diagram of the left and right eyes of the user focusing on the farthest perceived object (at the farthest plane distance Dff) in the stereoscopic image at a given conversion factor in accordance with one or more non-limiting embodiments of the present technology.

[0156] It will be appreciated that zooming changes the distances in the x and y planes, as the size of the pair of images varies by factor M, and thus distances on the z axis (i.e., perceived depth) must be recalculated to preserve proportionality. As a result, when zooming is performed, the farthest convergence distance (Dff) of the farthest convergence plane does not change, however the magnified nearest convergence distance (Dnn) of the nearest convergence plane, the magnified image plane distance (Dgf) of the image plane, and the magnified shift between images must be recalculated to provide for comfortable viewing and a maximal depth effect when rendering the magnified stereoscopic image.

[0157] Since Pif * Mmax - 2 * SHIFTgd = Pgfo = PGfo, the magnification results in Pif*M -2* SHIFTgd = Pgff, which is the final or magnified parallax at magnified image plane distance Dgf. Thus, the magnified shift may be determined as:SHIFTgd = (Pif* M - Pgff) / 2; orSHIFTgd = (Pif * Z * Mmax - Pgff) / 2

[0158] Thus, the magnified image plane distance (Dgf) may be determined. Since Dgn = Dgf, the magnified image parallax is equal to 0. Thus, Pgnf = 0 = Pin * M - 2 *SHIFTgd, and via substitution, it can be determined that:Pgff = (Pif- Pin) / 2 = (Dff-Dgf) / Dff

[0159] As a result, the magnified image plane distance (Dgf) may be obtained as follows:Dgf= Dff * (1 - (M * (Pif-Pin)) / Io)

[0160] The pair of images may be positioned and rendered at the optimal image plane distance (Dgo = Dnf), with the left image at width (LgG), the right image at width (LgD) being shifted by respective magnified shift distance (SHIFTgdo).

[0161] It should be understood that the magnified image plane distance, the magnified left image width, the magnified right image width, and the magnified shift may be calculated for a plurality of zoom / conversion factor values.

[0162] The pair of images may then be rendered to the user on the HMD 105 using techniques known in the art.

[0163] Now turning to FIGs. 8 A and 8B, there is shown a flowchart of a method 500 of rendering a pair of stereoscopic images at a maximum conversion factor in accordance with one or more non-limiting embodiments of the present technology.

[0164] The method 500 is executed to render a pair of stereoscopic images such that they can be viewed comfortably by a user wearing an HMD such as the HMD system 100.

[0165] It should be understood that the method 500 is executed to determine optimal distances and size of the pair of images at maximum conversion such that the resulting stereoscopic image can comfortably viewed by the user while the size and perceived depth of the rendered image is maximized without overflowing from the FOV.

[0166] The method 500 may be executed by at least one processing unit. The at least one processing unit may be, for example, a processing unit of a computing device operatively connected to the HMD system 100 or a processing unit 110 of the HMD system 100. In one or more other implementations, the method 500 may be executed in a distributed manner by a plurality of processing units.

[0167] In one or more implementations, the at least one processing unit is operativelyconnected to at least one non-transitory storage medium storing computer-readable instructions thereon. The at least one processing unit, upon loading the computer-readable instructions, is configured or operable to execute the method 500.

[0168] The method begins at processing step 502.

[0169] At processing step 502, the at least one processing unit receives a pair of images comprising a left image and right image.

[0170] The left image has a left image width LiG, and the right image has a right image width LiD.

[0171] At processing step 504, the at least one processing unit receives a virtual screen distance.

[0172] The virtual screen distance (Dvs) corresponds to the distance of the virtual screen rendered in the virtual space and perceived by the user. The virtual screen distance is a function of the parameters of the HMD system 100.

[0173] In one or more implementations, the virtual screen distance (Dvs) may be calculated based on the parameters of the lenses used in the HMD system 100 transmitted by the HMD system 100 or by another computing device.

[0174] As a non-limiting example, the virtual screen distance may be about 1.3 m.

[0175] At processing step 506, the at least one processing unit receives an interocular distance.

[0176] In one or more implementations, the interocular distance (Io) may be received from the HMD system 100 or from another computing device. In one or more other implementations, the interocular distance may be input via a user interface operatively connected to the HMD system 100.

[0177] In one or more alternative implementations, the interocular distance may be an average interocular distance. As a non-limiting example, the average interocular distance for adults is 63 millimeters with the interocular distance for most adults ranging between 50 and 75 millimeters.

[0178] At processing step 508, the at least one processing unit receives a nearest image parallax and farthest image parallax of points of the pair of images.

[0179] The nearest parallax (Pin) and the farthest parallax (Pit) may be provided based on the pair of images to be rendered. In one or more implementations, the nearest parallax (Pin) and the farthest parallax (Pit) may be received from a computing device.

[0180] In one or more other embodiments, the nearest image parallax and the farthest image parallax may be estimated or calculated based on theoretical values. In one or more alternative implementations, the total image parallax (Pit = Pif-Pin) may be received instead of the farthest image parallax and the nearest image parallax.

[0181] It will be appreciated that processing steps 504-508 may be executed sequentially in any order or may be executed in parallel.

[0182] At processing step 510, the at least one processing unit calculates, based on the virtual screen distance, a farthest converge distance in the virtual viewing space.

[0183] The farthest convergence distance (Dff) corresponds to a distance of the farthest plane from the viewer in the view frustrum (i.e., virtual viewing space). The farthest convergence distance defines a back boundary of the virtual viewing space within a comfort viewing zone. It will be appreciated that objects beyond that plane may be not viewed or rendered properly and may be clipped or masked.

[0184] It should be noted that the farthest convergence plane is located at the farthest convergence distance independently of the conversion used.

[0185] In some implementations, the farthest convergence distance may be calculated as follows, based on a formula provided in Shibata (Shibata, Takashi, et al. "The zone of comfort: Predicting visual discomfort with stereo displays." Journal of vision 11.8 (2011): 11-11.):Df = Dvs * 1.2286 / (1 - Dvs * 0.442)

[0186] Where: Df is the farthest convergence distance for the HMD; and Dvs refers to the virtual screen distance in virtual viewing space, which is a function of the focal distance of the lenses of the HMD and the distance between the display and the eyes in the HMD.

[0187] In one or more other implementations, for example in a mixed reality (MR) context, the farthest convergence distance may be a distance of the farthest object in the virtual scene (e.g., wall). In such implementations, the farthest convergence distance maybe set as the minimum distance between the theorical farthest convergence distance and a distance of the farthest object in the virtual scene.

[0188] At processing step 512, the at least one processing unit calculates, based on the virtual screen distance, a nearest convergence distance in the virtual viewing space.

[0189] The nearest convergence distance (Dn) corresponds to a distance of the closest plane to the viewer in the view frustum (i.e., virtual viewing space). The closest plane defines the front boundary of the virtual viewing space within a comfort viewing zone.

[0190] In some implementations, the nearest convergence distance Dn is calculated as a function of the focal distance of the lenses of the wearable VR display as follows, based on Shibata:Dn = Dvs * 1.0365 / (1 - Dvs * -0.626)

[0191] Where Dvs refers to the virtual screen distance in virtual viewing space, which is a function of the focal distance of the lenses of the wearable VR display and the distance between the display and the eyes in the wearable VR display.

[0192] In some implementations, the farthest convergence distance Df and the nearest convergence distance Dn may be determined based on the teachings of the following publication: Visual Discomfort and Visual Fatigue of Stereoscopic Displays: A Review; May 2009; Journal of Imaging Science and Technology 53(3); DOI:10.2352 / J.ImagingSci.Technol.2009.53.3.030201.

[0193] It will be appreciated that in some implementations, objects and points closer than the nearest plane may not be viewed and rendered properly and may be clipped. In one or more other implementations of the present technology, objects and points may be rendered.

[0194] In one or more other implementations, for example in a mixed reality (MR) context, the nearest convergence distance Dn may be set as the minimum distance between: the theorical nearest convergence distance and a predetermined nearest convergence distance.

[0195] At processing step 514, the at least one processing unit calculates, based on the farthest convergence distance, the nearest convergence distance, the interocular distance, the nearest parallax and the farthest parallax, a maximum conversion factor.

[0196] The maximum conversion factor (Mmax) corresponds to the maximum enlargement of the stereoscopic image that may be rendered within the virtual viewing space such that the stereoscopic image can be viewed comfortably by the user. At the maximum conversion factor, the closest object in the rendered stereoscopic image is perceived at the nearest convergence distance (Dnf or Dn), and the farthest object in the rendered stereoscopic image is perceived at the farthest convergence distance (Dff or Df) when the pair of images are positioned at the image plane (Dg or Dgo), thus maximizing the perceived depth. At maximum conversion, the optimal image plane distance may be set to be equal to the nearest convergence distance (i.e., Dgo = Dnf) with the parallax of the nearest object (Pgno) being equal to zero (i.e., Pgno = 0).

[0197] In some implementations, the maximum conversion factor Mmax is calculated as follows:Mmax = (Dff- Dnf) * Io / ([Pif — Pin] * Dff)

[0198] Thus, the maximum width of the left image LgGmax and the maximum width of the right image LgDmax may be determined by multiplying the original left image size LiG and the original right image size LiD respectively by the maximum conversion factor Mmax (i.e., LgG = Lig * Mmax and LgD = Lid * Mmax).

[0199] At processing step 516, the at least one processing unit calculates, based on the maximum conversion factor, the nearest parallax and the farthest parallax, a respective shift for the pair of images to be rendered on an image plane positioned at the nearest vergence distance.

[0200] The respective shift is a translation of the images at maximum conversion determined based on the optimal total parallax at the image plane (Pgto) being equal to a difference between the maximum parallax (Pgfo) and the minimum parallax (Pgno), where Pgno = 0. In this case, the optimal total parallax at the image plane (Pgto) is equal to the maximum parallax at the image plane (Pgfo).

[0201] In one or more implementations, the respective shift may be determined based on the nearest image parallax (Pin) or the farthest image parallax (Pif).SHIFTgdo = (Pif * Mmax - Pgfo) / 2; orSHIFTgdo = Pin * Mmax / 2

[0202] The respective shift provides a distance by which to translate the left image on the left and the right image on the right on the image plane at image plane distance (Dgo) at the maximum conversion factor.

[0203] At processing step 518, the at least one processing unit determines overlapping portions of the pair of images positioned at the image plane and being shifted based on the respective shift.

[0204] The at least one processing unit determines the non-overlapping portions of the pair of images when the pair of images are positioned at the respective shift on the image plane. The at least one processing unit trims or clips the non-overlapping portions of the image at the image plane. It will be appreciated that the images have respective sizes LgGmax and LgDmax at the maximum conversion factor.

[0205] At processing step 520, the at least one processing unit causes rendering of at least a section of the overlapping portions of the pair of images, which are positioned at the image plane distance, the image plane distance being equal to the nearest vergence distance.

[0206] In one or more implementations, the at least one processing unit transmits, to a rendering engine, an indication of the maximum conversion factor, an indication of the image plane distance and an indication of at least the section of the overlapping portions of the pair of images together with an indication of the widths of the images. The rendering engine may then display the stereoscopic images according to the image plane distance and the indication of at least the section of the overlapping portions of the pair of images.

[0207] In some embodiments, prior to executing processing step 520, the at least one processing unit may cause execution of method 600, which will be described hereinafter.

[0208] It should be understood that by executing method 500, the stereoscopic image rendered and displayed to the user wearing the HMD may be viewed comfortably with maximum perceived depth according to the parameters of the images and optical parameters of the HMD at maximum conversion, without knowledge of the acquisition parameters of the cameras or sensor having acquired the pair of stereoscopic images.

[0209] With reference to FIG. 9, there is shown a method 600 of rendering a pair ofimages at a given conversion factor in accordance with one or more non-limiting embodiments of the present technology.

[0210] At processing step 602, the at least one processing unit receives a zoom factor Z.

[0211] In one or more implementations, the zoom factor Z may be received via a user interface connected to the HMD system 100. In one or more other implementations, the zoom factor may be predetermined to one or more default values.

[0212] The zoom factor Z may be a value in the range from ]0, 1] where the maximum value of one (i.e., 100%) corresponds to Mmax:Z = M / Mmax;M = Z * Mmax

[0213] Where M is the conversion factor corresponding to the zoom factor Z.

[0214] As a non-limiting example, the zoom factor Z may have a predetermined value of0.75.

[0215] It should be appreciated that zooming changes the size of the 3D virtual viewing space in x, y, and z coordinates. When zooming, the proportionality in the x, y and z axis should be preserved to provide a linear perception of depth change.

[0216] At processing step 604, the at least one processing unit receives a magnified farthest parallax, the magnified farthest parallax being a parallax at the farthest plane.

[0217] At processing step 606, the at least one processing unit calculates a respective magnified image size for the pair of images based on the conversion factor M.

[0218] In some embodiments, the magnified left image size is calculated by multiplying the left image size by the conversion factor M and the magnified right image size is calculated by multiplying the right image size the conversion factor M.

[0219] At processing step 608, the at least one processing unit calculates, based on the zoom factor Z (or the corresponding conversion factor M), the nearest parallax and the magnified farthest parallax, a respective magnified shift.

[0220] In one or more implementations, the magnified respective shift is calculated using:SHIFTgd = (Pif * M - Pgff) / 2; orSHIFTgd = (Pif * Z * Mmax - Pgff) / 2

[0221] At processing step 610, the at least one processing unit calculates a magnified image plane distance based on the conversion factor M.

[0222] In one or more implementations, the magnified image plane distance based on the conversion factor M is calculated using:Dgf= Dff * (1 - (M * (Pif-Pin)) / Io)

[0223] At processing step 612, the at least one processing unit determines magnified overlapping portions of the pair of images positioned at the magnified image plane and being shifted based on the respective magnified shift.

[0224] At processing step 614, the at least one processing unit causes rendering of the magnified overlapping portions of the magnified pair of images at the magnified image plane on the HMD.

[0225] It should be understood that the conversion factor M and the zoom factor may be interchangeably used for the calculations occurring in the method 600 since Z = M / Mmax. Therefore, knowing the zoom factor Z is equivalent to knowing the conversion factor M, and vice versa. For example, processing step 602 may consists in receiving the conversion factor M and the calculation of the magnified left image size, the magnified right image size, the respective magnified shift and the magnified image plane distance may be performed based on the received conversion factor M. Alternatively, the zoom factor F may be calculated based on the received conversion factor M, and the calculation of the magnified left image size, the magnified right image size, the respective magnified shift and the magnified image plane distance may be performed based on the calculated zoom factor. In another example, processing step 602 may consists in receiving the zoom factor F and the calculation of the magnified left image size, the magnified right image size, the respective magnified shift and the magnified image plane distance may be performed based on the received zoom factor F. Alternatively, the conversion factor M may be calculated based on the received zoom factor F, and the calculation of the magnified left image size, the magnified right image size, the respective magnified shift and the magnified image plane distance may be performed based on the calculatedconversion factor M.

[0226] It should be understood the magnified overlapped portions correspond to overlapping portions of the magnified image.

[0227] It will be appreciated that prior to rendering the pair of images at a given conversion factor, the maximum conversion factor has to be calculated.

[0228] The pair of images may be rendered using the present methods in a virtual viewing space which does not display other visual information. However, the present method is not limited to such an implementation. For example, the pair of images may be rendered using the present method in a virtual viewing space which further displays real visual information, e.g., surrounding visual information superposed with or underlaid the rendered pair of images for mixed reality and / or augmented reality applications. In such applications, the expression ‘virtual viewing space’ is further bound by the real visual elements displayed by the wearable VR display. For such implementations, the expression ‘visual viewing space’ corresponds to the visual viewing space as affected by the real visual elements. Those skilled in the art of mixed reality and augmented reality are familiar with the techniques used for displaying images in such mixed reality and augmented reality, and modifying the wearable VR display virtual viewing space accordingly.

[0229] Relying on an optical characteristic of the lenses of the wearable VR display for positioning the image plane enables minimizing calculations and image processing, as no information on the conditions for capturing the pair of images is necessary. Further, embodiments of the present enable preventing overflow of the pair of images from the virtual viewing space and provide for comfortable zooming of the pair of images in the virtual viewing space, while maximizing the 3D effect and minimizing negative physiological responses from the user.

[0230] As described above, if the convention on the orientation of the lateral axis of the image plane is changed so that the lateral axis is oriented towards the left of FIGS. 3 and 4, then the allowed total parallax at the image plane distance Dg is equal to the allowed foreground parallax Pgno at the image plane distance Dg minus the allowed background parallax Pgfo at the image plane distance Dg. Also, Condition 1 becomes: the image foreground parallax Pin at the image plane distance Dg is at most equal to theallowed foreground parallax Pgno. Furthermore, Condition 2 becomes: the image background parallax Pif at the image plane distance Dg is at least equal to the allowed background parallax Pgfo.

[0231] Similarly, if the convention for calculating a parallax is changed, the definition of the total parallax and Conditions 1 and 2 change. In the above-description, it is assumed that the allowed background parallax Pgfo then corresponds to the lateral coordinate or position of the point 164 along the width of the image plane 150 minus the lateral coordinate or position of the point 166 along the width of the image plane 150, and that the allowed foreground parallax Pgno then corresponds to the lateral coordinate or position of the point 180 along the width of the image plane 150 minus the lateral coordinate or position of the point 182 along the width of the image plane 150. However, if the convention is changed so that the allowed background parallax Pgfo then corresponds to the lateral coordinate or position of the point 166 along the width of the image plane 150 minus the lateral coordinate or position of the point 164 along the width of the image plane 150, and that the allowed foreground parallax Pgno then corresponds to the lateral coordinate or position of the point 182 along the width of the image plane 150 minus the lateral coordinate or position of the point 180 along the width of the image plane 150, then the allowed total parallax at the image plane distance Dg is equal to the allowed foreground parallax Pgno at the image plane distance Dg minus the allowed background parallax Pgfo at the image plane distance Dg, and Conditions 1 and 2 become: the image foreground parallax Pin at the image plane distance Dg is at most equal to the allowed foreground parallax Pgno (Condition 1); and the image background parallax Pif at the image plane distance Dg is at least equal to the allowed background parallax Pgfo (condition 2), respectively.

[0232] Although the present disclosure has been described hereinabove by way of non-restrictive, illustrative embodiments thereof, these embodiments may be modified at will within the scope of the appended claims without departing from the spirit and nature of the present disclosure.

Claims

1. CLAIMS2.What is claimed is:

1. A method for rendering a pair of stereoscopic images in a virtual 3D viewing space, the pair of stereoscopic images comprising a left image and a right image each having a respective image size, the method being executed by at least one processor operatively connected to a head-mounted display (HMD), the method comprising: receiving a minimal viewing distance and a maximal viewing distance associated with the virtual 3D viewing space of the HMD;4.determining at least one of an allowed foreground parallax and an allowed background parallax both at an image plane distance based on an interocular distance, the image plane distance and at least one of the minimal viewing distance and the maximal viewing distance;5.determining an allowed total parallax based on the at least one of the allowed foreground parallax and the allowed background parallax;6.for each one of the left and right images, determining a respective viewing space size at the image plane distance based on a given conversion factor and the respective image size, the given conversion factor ensuring that a total parallax at the image place distance is at most equal to the allowed total parallax and being at most equal to a maximal conversion factor;7.determining an image offset for laterally translating the pair of stereoscopic images at the image plane distance, the image offset ensuring that at least one of:8.a given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax; and9.a given background parallax at the image plane distance is at most equal to the allowed background parallax; and causing rendering of the pair of stereoscopic images at the image plane distance within the virtual 3D viewing space of the HMD using the respective viewing space size and the image offset.

2. The method of claim 1, further comprising determining an overlapping portion of the pair of stereoscopic images positioned at the image plane and being shifted based on the respective shift, said causing the rendering of the pair of stereoscopic images comprising causing rendering of at least a section of the overlapping portion of the pair of stereoscopic images.

3. The method of claim 1 or 2, wherein the allowed total parallax is equal to a difference between the allowed background parallax and the allowed foreground parallax.

4. The method of any one of claims 1 to 3, further comprising calculating the maximal conversion factor based on the allowed total parallax and a total image parallax.

5. The method of any one of claims 1 to 4, further comprising receiving a zoom value and calculating the given conversion factor based on the zoom value and the maximal conversion factor.

6. The method of any one of claims 1 to 5, wherein said receiving the minimal viewing distance and the maximal viewing distance comprises calculating the minimal viewing distance and the maximal viewing distance based on a virtual screen distance, the virtual screen distance corresponding to a distance of a virtual screen perceived by a user wearing the HMD.

7. The method of any one of claims 1 to 6, wherein said determining the at least one of the allowed foreground parallax and the allowed background parallax comprises: determining the allowed foreground parallax based on the interocular distance, the image plane distance and the minimal viewing distance; and16.determining the allowed background parallax based on the interocular distance, the image plane distance and the maximal viewing distance, and wherein the image offset is chosen to ensure that the given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax and the given background parallax at the image plane distance is at most equal to the allowed background parallax.

8. The method of any one of claims 1 to 6, wherein:18.the image plane distance is equal to the minimal viewing distance,19.said determining the at least one of the allowed foreground parallax and the allowed background parallax comprises determining the allowed background parallax based on the interocular distance, the image plane distance and the maximal viewing distance; and20.said determining the allowed total parallax comprises determining the allowed total parallax based on the allowed foreground parallax, the allowed total parallax being equal to the allowed foreground parallax.

9. The method of claim 8, wherein the image offset is chosen to ensure that the given background parallax at the image plane distance is equal to the allowed background parallax, thereby ensuring that a background object appears at the maximal viewing distance.

10. The method of claim 9, wherein the given conversion factor is equal to the maximal conversion factor so that the total parallax at the image place distance is equal to the allowed total parallax and the given foreground parallax at the image plane distance is equal to zero, thereby ensuring that a foreground object appears at the minimal viewing distance.

11. The method of claim 8, wherein the image offset is chosen to ensure that the given foreground parallax at the image plane distance is equal to zero, thereby ensuring that a foreground object appears at the minimal viewing distance.

12. The method of claim 11, wherein the given conversion factor is equal to the maximal conversion factor so that the total parallax at the image place distance is equalto the allowed total parallax, thereby ensuring that a background object appears at the maximal viewing distance.

13. A system for rendering a pair of stereoscopic images in a virtual 3D viewing space, the pair of stereoscopic images comprising a left image and a right image each having a respective image size, the system comprising:26.at least one processing unit; and27.a non-transitory storage medium storing computer-readable instructions thereon, the at least one processing unit, upon executing the computer-readable instructions, being configured for:28.receiving a minimal viewing distance and a maximal viewing distance associated with the virtual 3D viewing space of the HMD;29.determining at least one of an allowed foreground parallax and an allowed background parallax both at an image plane distance based on an interocular distance, the image plane distance and at least one of the minimal viewing distance and the maximal viewing distance;30.determining an allowed total parallax based on the at least one of the allowed foreground parallax and the allowed background parallax;31.for each one of the left and right images, determining a respective viewing space size at the image plane distance based on a given conversion factor and the respective image size, the given conversion factor ensuring that a total parallax at the image place distance is at most equal to the allowed total parallax and being at most equal to a maximal conversion factor;32.determining an image offset for laterally translating the pair of stereoscopic images at the image plane distance, the image offset ensuring that at least one of:33.a given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax; and a given background parallax at the image plane distance is at most equal to the allowed background parallax; and34.causing rendering of the pair of stereoscopic images at the image plane distance within the virtual 3D viewing space of the HMD using the respective viewing space size and the image offset.

14. The system of claim 13, wherein the at least one processing unit is further configured for determining an overlapping portion of the pair of stereoscopic images positioned at the image plane and being shifted based on the respective shift, said causing the rendering of the pair of stereoscopic images comprising causing rendering of at least a section of the overlapping portion of the pair of stereoscopic images.

15. The system of claim 13 or 14, wherein the allowed total parallax is equal to a difference between the allowed background parallax and the allowed foreground parallax.

16. The system of any one of claims 13 to 15, wherein the at least one processing unit is further configured for calculating the maximal conversion factor based on the allowed total parallax and a total image parallax.

17. The system of any one of claims 13 to 16, wherein the at least one processing unit is further configured for receiving a zoom value and calculating the given conversion factor based on the zoom value and the maximal conversion factor.

18. The system of any one of claims 13 to 5, wherein the at least one processing unit is configured for calculating the minimal viewing distance and the maximal viewing distance based on a virtual screen distance, the virtual screen distance corresponding to a distance of a virtual screen perceived by a user wearing the HMD.

19. The system of any one of claims 13 to 18, wherein the at least one processing unit is configured for:41.determining the allowed foreground parallax based on the interocular distance, the image plane distance and the minimal viewing distance; and determining the allowed background parallax based on the interocular distance, the image plane distance and the maximal viewing distance, and42.wherein the image offset is chosen to ensure that the given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax and the given background parallax at the image plane distance is at most equal to the allowed background parallax.

20. The system of any one of claims 13 to 18, wherein:44.the image plane distance is equal to the minimal viewing distance,45.the at least one processing unit is configured for determining the allowed background parallax based on the interocular distance, the image plane distance and the maximal viewing distance; and46.the at least one processing unit is configured for determining the allowed total parallax based on the allowed foreground parallax, the allowed total parallax being equal to the allowed foreground parallax.

21. The system of claim 20, wherein the image offset is chosen to ensure that the given background parallax at the image plane distance is equal to the allowed background parallax, thereby ensuring that a background object appears at the maximal viewing distance.

22. The system of claim 21, wherein the given conversion factor is equal to the maximal conversion factor so that the total parallax at the image place distance is equal to the allowed total parallax and the given foreground parallax at the image plane distance is equal to zero, thereby ensuring that a foreground object appears at the minimal viewing distance.

23. The system of claim 20, wherein the image offset is chosen to ensure that the given foreground parallax at the image plane distance is equal to zero, thereby ensuring that a foreground object appears at the minimal viewing distance.

24. The system of claim 23, wherein the given conversion factor is equal to the maximal conversion factor so that the total parallax at the image place distance is equal to the allowed total parallax, thereby ensuring that a background object appears at the maximal viewing distance.

25. A non-transitory storage medium storing computer-readable instructions thereon, the computer-readable instructions, upon being executed by at least one processor, are configured for causing:51.receiving a minimal viewing distance and a maximal viewing distance associated with the virtual 3D viewing space of the HMD;52.determining at least one of an allowed foreground parallax and an allowed background parallax both at an image plane distance based on an interocular distance, the image plane distance and at least one of the minimal viewing distance and the maximal viewing distance;53.determining an allowed total parallax based on the at least one of the allowed foreground parallax and the allowed background parallax;54.for each one of the left and right images, determining a respective viewing space size at the image plane distance based on a given conversion factor and the respective image size, the given conversion factor ensuring that a total parallax at the image place distance is at most equal to the allowed total parallax and being at most equal to a maximal conversion factor;55.determining an image offset for laterally translating the pair of stereoscopic images at the image plane distance, the image offset ensuring that at least one of:56.a given foreground parallax at the image plane distance is at least equal to the allowed foreground parallax; and57.a given background parallax at the image plane distance is at most equal to the allowed background parallax; and causing rendering of the pair of stereoscopic images at the image plane distance within the virtual 3D viewing space of the HMD using the respective viewing space size and the image offset.

26. The non-transitory storage medium of claim 25, wherein the computer-readable instructions are configured for causing: determining an overlapping portion of the pair of stereoscopic images positioned at the image plane and being shifted based on the respective shift, said causing the rendering of the pair of stereoscopic images comprising causing rendering of at least a section of the overlapping portion of the pair of stereoscopic images.

Images