Hybrid rendering of virtual reality content using depth-based occlusion

Abstract

A virtual reality rendering system includes a processor, memory storing a stereo panoramic image with color and depth data captured using above-horizon and below-horizon projections representing static virtual environment components, and a graphics processing unit. The GPU renders the stereo panoramic image as a full-screen quad using equirectangular projection coordinates, extracts depth values and reconstructs world-space positions by applying inverse projection and view matrices corresponding to capture parameters, seeds a runtime depth buffer with reconstructed positions for depth-based occlusion, and renders dynamic three-dimensional objects using standard rasterization techniques. The dynamic objects interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships.

Description

[0001] TSZ-00325

[0002] HYBRID RENDERING OF VIRTUAL REALITY CONTENT USING DEPTH-BASED OCCLUSION

[0003] TECHNICAL FIELD

[0004] The present disclosure relates to virtual reality rendering systems, and more particularly to a hybrid rendering technique that combines stereo panoramic images with dynamic three-dimensional content using depth-based occlusion for efficient rendering on mobile virtual reality hardware.

[0005] BACKGROUND

[0006] Virtual reality technology has become increasingly prevalent in consumer markets, with mobile and standalone virtual reality headsets offering immersive experiences without requiring external hardware or internet connectivity. These devices typically provide six degrees of freedom tracking, allowing users to experience three-dimensional environments that respond dynamically to head position and orientation. The rendering of virtual reality content involves complex graphics processing techniques to generate stereoscopic images at high frame rates while maintaining visual fidelity across wide fields of view.

[0007] Current virtual reality rendering approaches generally fall into two categories: traditional three-dimensional rasterization techniques that render geometric models in real-time, and panoramic image-based rendering that displays pre-captured spherical imagery. Three-dimensional rasterization provides full interactive capability but places substantial computational demands on mobile processors, while panoramic rendering offers high visual quality with lower computational overhead but lacks dynamic interaction capabilities. Mobile virtual reality systems often employ various optimization strategies including level-of-detail management, occlusion culling, and texture compression to balance visual quality with performance constraints inherent in portable hardware platforms. TSZ-00325

[0008] SUMMARY

[0009] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0010] In a first aspect, the technology provides a virtual reality rendering system. The system comprises a processor configured to execute instructions, a memory storing a stereo panoramic image comprising color data and depth data captured using an above-horizon projection and a below-horizon projection, wherein the stereo panoramic image represents static components of a virtual environment generated through a first process utilizing standard rasterization pipelines, and a graphics processing unit. The graphics processing unit may implement a second process that renders the stereo panoramic image as a full-screen quad using equirectangular projection coordinates, extract depth values from the depth data and reconstruct original world-space positions by applying an inverse projection matrix and an inverted view matrix corresponding to capture parameters used during panoramic image generation, seed a runtime depth buffer with the reconstructed world-space positions to enable depth-based occlusion, and render dynamic three-dimensional objects using standard rasterization techniques, wherein the dynamic objects interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships.

[0011] In some embodiments, the above-horizon projection captures imagery with a camera pointed 45 degrees above a horizon and the below-horizon projection captures imagery with a camera pointed 45 degrees below the horizon. In other embodiments, the stereo panoramic image is captured by rotating the camera in discrete steps of 360 degrees divided by a target image width to generate column-wise image strips. In yet other embodiments, the stereo panoramic image comprises a left eye view and a right eye view captured with an inter-pupillary distance baseline, and the left eye view and right eye view are vertically stacked in a single texture. In further TSZ-00325

[0012] embodiments, the graphics processing unit is configured to separate the left eye view and right eye view by converting texture coordinates from a range of 0 to 0.5 or 0.5 to 1.0 to a range of 0 to 1.0 based on a stereo eye index. In additional embodiments, reconstructing the original worldspace positions comprises snapping UV coordinates to pixel centers to match exact capture parameters, calculating original camera position and orientation based on column position and above-horizon or below-horizon designation, constructing clip-space coordinates with X set to 0.0, Y derived from UV coordinates, and Z set to the extracted depth value, and applying the inverse projection matrix and inverted view matrix to transform the clip-space coordinates to world-space positions. In some embodiments, snapping UV coordinates to pixel centers comprises multiplying UV coordinates by pixel dimensions, taking a floor operation, adding 0.5, and dividing by the pixel dimensions. In other embodiments, the depth data is stored as 32-bit floating point values using a reversed Z-buffer format where 1.0 represents near plane and 0.0 represents far plane. In further embodiments, the graphics processing unit is configured to render the stereo panoramic image using a fragment shader that samples both color data and depth data from the stereo panoramic image during a single full-screen quad pass. In additional embodiments, the fragment shader performs equirectangular coordinate transformation using a function that converts UV coordinates to spherical coordinates based on above-horizon and below-horizon regions. In some embodiments, the virtual reality rendering system is implemented on a mobile virtual reality headset comprising a SNAPDRAGON XR2, a system-on-chip processor, manufactured by Qualcomm Incorporated, of San Diego, California, and supports VULKAN, a low-level graphics application programming interface, developed by the Khronos Group, of Beaverton, Oregon. In other embodiments, the stereo panoramic image has a resolution of 8192x8192 pixels and uses ASTC, an adaptive scalable texture compression format, developed by ARM Limited, of Cambridge, England, 4x4 block compression for color data to reduce memory bandwidth requirements. In yet other embodiments, the dynamic three-dimensional objects are constrained to move only within predetermined regions of the virtual environment while a viewer position remains fixed to rotational movement only. In further TSZ-00325

[0013] embodiments, transparent objects are rendered back-to-front after the stereo panoramic image and dynamic three-dimensional objects, and transparency is not supported within the stereo panoramic image itself.

[0014] In a second aspect, the technology provides a method for hybrid rendering of virtual reality content. The method comprises storing a stereo panoramic image comprising color data and depth data, wherein the stereo panoramic image represents static components of a virtual environment and is captured using an above- horizon projection and a below-horizon projection through a first process employing standard rasterization pipelines, rendering the stereo panoramic image as a full-screen quad using equirectangular projection coordinates, extracting depth values from the depth data, reconstructing original world-space positions from the extracted depth values by applying an inverse projection matrix and an inverted view matrix corresponding to capture parameters used during panoramic image generation, seeding a runtime depth buffer with the reconstructed world-space positions, and rendering dynamic three-dimensional objects using standard rasterization techniques, wherein the dynamic objects interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships, and wherein the extracting, reconstructing, seeding, and rendering steps comprise a second process that enables occlusion calculations with traditionally rasterized geometry.

[0015] In some embodiments, the above-horizon projection captures imagery with a camera pointed 45 degrees above a horizon and the below-horizon projection captures imagery with a camera pointed 45 degrees below the horizon. In other embodiments, the stereo panoramic image is captured by rotating the camera in discrete steps of 360 degrees divided by a target image width to generate column-wise image strips. In yet other embodiments, reconstructing the original world-space positions comprises snapping UV coordinates to pixel centers to match exact capture parameters, calculating original camera position and orientation based on column position and above-horizon or below-horizon designation, constructing clip-space coordinates with X set to 0.0, Y derived from UV coordinates, and Z set to the extracted depth value, and TSZ-00325

[0016] applying the inverse projection matrix and inverted view matrix to transform the clip-space coordinates to world-space positions. In further embodiments, snapping UV coordinates to pixel centers comprises multiplying UV coordinates by pixel dimensions, taking a floor operation, adding 0.5, and dividing by the pixel dimensions. In additional embodiments, the stereo panoramic image comprises a left eye view and a right eye view captured with an inter-pupillary distance baseline, and the method further comprises separating the left eye view and right eye view by converting texture coordinates from a range of 0 to 0.5 or 0.5 to 1.0 to a range of 0 to 1.0 based on a stereo eye index. In some embodiments, the depth data is stored as 32-bit floating point values using a reversed Z-buffer format where 1.0 represents near plane and 0.0 represents far plane, and the dynamic three-dimensional objects are constrained to move only within predetermined regions of the virtual environment while a viewer position remains fixed to rotational movement only.

[0017] In a third aspect, the technology provides a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to access a stereo panoramic image comprising color data and depth data, wherein the stereo panoramic image represents static components of a virtual environment and is captured using an above-horizon projection and a below-horizon projection through a first process utilizing standard rasterization pipelines, render the stereo panoramic image as a full-screen quad using equirectangular projection coordinates, extract depth values from the depth data, reconstruct original world-space positions from the extracted depth values by applying an inverse projection matrix and an inverted view matrix corresponding to capture parameters used during panoramic image generation, seed a runtime depth buffer with the reconstructed world-space positions, and render dynamic three-dimensional objects using standard rasterization techniques, wherein the dynamic objects interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships, and wherein the rendering, extracting, reconstructing, seeding, and rendering steps TSZ-00325

[0018] comprise a second process that reconstructs world space positions of each pixel within the panorama to enable occlusion with traditionally rasterized geometry.

[0019] In some embodiments, the above-horizon projection captures imagery with a camera pointed 45 degrees above a horizon and the below-horizon projection captures imagery with a camera pointed 45 degrees below the horizon. In other embodiments, the stereo panoramic image is captured by rotating the camera in discrete steps of 360 degrees divided by a target image width to generate column-wise image strips. In yet other embodiments, reconstructing the original world-space positions comprises snapping UV coordinates to pixel centers by multiplying UV coordinates by pixel dimensions, taking a floor operation, adding 0.5, and dividing by the pixel dimensions, calculating original camera position and orientation based on column position and above-horizon or below-horizon designation, constructing clip-space coordinates with X set to 0.0, Y derived from UV coordinates, and Z set to the extracted depth value, and applying the inverse projection matrix and inverted view matrix to transform the clip-space coordinates to world-space positions.

[0020] The foregoing general description of the illustrative embodiments and the following detailed description thereof are merely exemplary aspects of the teachings of this disclosure and are not restrictive.

[0021] The foregoing general description of the illustrative embodiments and the following detailed description thereof are merely exemplary aspects of the teachings of this disclosure and are not restrictive.

[0022] BRIEF DESCRIPTION OF FIGURES

[0023] Non-limiting and non-exhaustive examples are described with reference to the following figures. The purpose and advantages of the disclosed subject matter will be set forth in and apparent from the description that follows, as well as will be learned by practice of the disclosed subject matter. Additional advantages of the disclosed subject matter will be realized and attained by the TSZ-00325

[0024] methods and systems particularly pointed out in the written description and claims hereof, as well as from the appended drawings.

[0025] FIG. 1 illustrates a flowchart for a method for hybrid rendering of virtual reality content, according to aspects of the present disclosure.

[0026] FIG. 2 depicts a flowchart for a stereo panoramic image capture method, according to an embodiment.

[0027] FIG. 3 illustrates a flowchart for a depth reconstruction method, according to aspects of the present disclosure.

[0028] FIG. 4 depicts a flowchart for a virtual reality rendering method, according to an embodiment. FIG. 5 illustrates a block diagram of a virtual reality rendering system, according to aspects of the present disclosure.

[0029] FIG. 6 depicts a block diagram of a stereo panoramic capture system for virtual reality rendering applications, according to an embodiment.

[0030] It is to be understood that both the foregoing general description and the following detailed description are exemplary and are intended to provide further explanation of the disclosed subject matter claimed.

[0031] The accompanying drawings, which are incorporated in and constitute part of this specification, are included to illustrate and provide a further understanding of the method and system of the disclosed subject matter. Together with the description, the drawings serve to explain the principles of the disclosed subject matter.

[0032] DETAILED DESCRIPTION

[0033] The following description sets forth exemplary aspects of the present disclosure. It should be recognized, however, that such description is not intended as a limitation on the scope of the TSZ-00325

[0034] present disclosure. Rather, the description also encompasses combinations and modifications to those exemplary aspects described herein.

[0035] A detailed description of systems, devices, and methods consistent with embodiments of the present disclosure is provided below. While several embodiments are described, it should be understood that disclosure is not limited to any one embodiment, but instead encompasses numerous alternatives, modifications, and equivalents. In addition, while numerous specific details are set forth in the following description in order to provide a thorough understanding of the embodiments disclosed herein, some embodiments can be practiced without some or all of these details. Moreover, for the purpose of clarity, certain technical material that is known in the related art has not been described in detail in order to avoid unnecessarily obscuring the disclosure.

[0036] Referring to FIG. 1, a method 100 for hybrid rendering of virtual reality content may be implemented to combine static panoramic imagery with dynamic three-dimensional objects through depth-based occlusion techniques. The method 100 may provide an approach for displaying complex, dynamic geometry at high framerates within the computational constraints of mobile virtual reality hardware by reconstructing world space positions of each pixel within the captured panorama to enable proper occlusion calculations with traditionally rasterized geometry.

[0037] The method 100 begin by rendering the stereo panoramic image as a full-screen quad using equirectangular projection coordinates (step 102). The stereo panoramic image may be accessed through various means, including retrieval from memory elements such as volatile memory (e.g., system RAM, graphics memory), non-volatile memory (e.g., flash storage, solid-state drives, hard disk drives), or static memory (e.g., ROM, embedded storage). Alternatively, the stereo panoramic image may be received in real-time from sources such as streaming data feeds, live panoramic camera inputs, network transmissions, or cloud-based storage services. In some cases, the panoramic image may be accessed from the stereo panoramic image storage 512 within the TSZ-00325

[0038] memory subsystem 508, where the color data 514 and depth data 516 are stored for efficient retrieval during rendering operations.

[0039] The full-screen quad rendering may involve drawing a single rectangular primitive that covers the entire viewport, serving as a canvas for displaying the panoramic content. In some cases, the full-screen quad rendering may transform UV coordinates according to an equirectangular projection to obtain sampling coordinates for the stereo panoramic image data. The UV coordinate transformation may utilize the EquirectangularCoords shader function that converts standard texture coordinates to spherical coordinates by calculating phi values using modulo operations and tangent functions. The equirectangular projection coordinates may enable proper mapping of the panoramic content to the virtual reality display by accounting for the spherical nature of the captured imagery and the planar nature of the display surface.

[0040] Depth values may be extracted from the depth data (step 104). The extracted depth values may correspond to distance measurements from the camera position for each pixel in the stereo panoramic image, providing the spatial information necessary for reconstructing three-dimensional world positions during runtime rendering operations. The depth extraction process may involve sampling the depth texture at coordinates corresponding to the equirectangular projection transformation performed in the step 102, where each depth sample may represent the original distance from the capture camera to the nearest surface visible at that pixel location. In some cases, the depth values may be stored as 32-bit floating point values using a reversed Z-buffer format where 1.0 may represent the near plane and 0.0 may represent the far plane. The reversed Z-buffer format may provide enhanced depth precision compared to traditional Z-buffer arrangements, particularly for distant objects in the virtual environment, by distributing floating point precision more evenly across the depth range. Alternatively, the depth values may be stored using 24-bit fixed-point formats such as D24_UNorm_S8_UInt, which may provide sufficient precision for most virtual reality applications while reducing memory bandwidth requirements. In other embodiments, the depth data may utilize 16-bit formats including D16_UNorm for TSZ-00325

[0041] applications with strict memory constraints, though this may result in reduced depth precision and potential depth fighting artifacts in complex scenes. The depth values may also be stored using logarithmic depth encoding, where depth values are transformed using logarithmic functions to achieve more uniform precision distribution across the entire depth range, particularly beneficial for scenes with large depth variations. In some cases, the depth information may be compressed using specialized depth compression algorithms that exploit spatial coherence in depth values, reducing storage requirements while maintaining acceptable reconstruction quality. The depth buffer format may be selected dynamically based on the target hardware capabilities, where high-end mobile processors may utilize 32-bit floating point formats while lower-end devices may fall back to 16-bit or 24-bit representations to maintain performance targets. The 32-bit floating point storage may ensure maximum depth precision during the capture process, enabling accurate world-space position reconstruction even for complex geometric details and fine surface variations within the panoramic imagery.

[0042] Original world-space positions may be reconstructed from the extracted depth values (step 106). In some implementations, this is done by applying an inverse projection matrix and an inverted view matrix corresponding to capture parameters used during panoramic image generation. The reconstruction process may transform the two-dimensional depth information stored in the stereo panoramic image back into three-dimensional world coordinates that correspond to the original geometry captured during the panoramic image generation process. In some cases, the reconstruction process may involve snapping UV coordinates to pixel centers to match exact capture parameters, calculating original camera position and orientation based on column position and above-horizon or below-horizon designation, and constructing clip-space coordinates with specific values derived from the UV coordinates and extracted depth values. The UV coordinate snapping operation may be implemented using a SnapToGrid shader function that multiplies UV coordinates by pixel dimensions, takes a floor operation to snap to the nearest integer pixel boundary, adds 0.5 to center the coordinate within the pixel, and divides by the TSZ-00325

[0043] pixel dimensions to return to normalized UV space. This snapping process may ensure that the reconstructed view parameters match exactly with the parameters used during the original panoramic capture process, preventing interpolation artifacts that could compromise the accuracy of the world-space position calculations. The pixel-perfect alignment may be critical for maintaining proper occlusion relationships between the static panoramic content and dynamic three-dimensional objects rendered subsequently.

[0044] The stereo camera positions may be calculated using the baseline offset formulas: P left = P_center - (baseline / 2) x R_right and P_right = P_center + (baseline / 2) x R_right, where P center represents the central capture position, baseline equals 0.064 meters, and R right is the normalized right vector [sin(O col), 0, -cos(O col)] for azimuth angle O col. The vergence adjustment may modify the camera orientations to converge at the target depth, with left eye forward vector F_left = normalize([cos(0_col + a_vergence / 2), sin(tilt_angle), sin(0_col + a_vergence / 2)]) and right eye forward vector F_right = normalize([cos(0_col - a_vergence / 2), sin(tilt_angle), sin(0_col - a_vergence / 2)]), where tilt_angle equals ±π / 4 for above / below-horizon projections and a vergence represents the calculated vergence angle for the current column position.

[0045] In an alternative embodiment, the system may operate without calculating a vergence angle. In this configuration, each camera position may be oriented to look straight ahead, with the viewing direction determined by multiplying the forward vector by the eye's rotation matrix. The left eye forward vector may be calculated as F left = normalize([cos(0_col), sin(tilt angle), sin(O col)]) and the right eye forward vector as F right = normalize([cos(0_col), sin(tilt angle), sin(O col)]), where both eyes maintain identical forward orientations while being offset only by the baseline distance. This approach may simplify the computational requirements during panoramic capture while maintaining effective stereo imaging capabilities through the inter-pupillary distance separation alone. TSZ-00325

[0046] The calculation of original camera position and orientation may utilize a getRays shader function that determines the camera's view position, up direction vectors, down direction vectors, and forward vectors based on the column position within the panoramic image and whether the sample corresponds to an above-horizon or below- horizon region. For above-horizon samples, the camera orientation may correspond to the 45-degree upward tilt used during capture, while below-horizon samples may correspond to the 45-degree downward tilt. The camera position may be adjusted by the inter-pupillary distance baseline, where the camera may be moved along the right vector by half the inter-pupillary distance for the right eye view and along the left vector by half the inter-pupillary distance for the left eye view.

[0047] The stereo camera positions may be calculated using the baseline offset formulas: P left = P_center - (baseline / 2) x R_right and P_right = P_center + (baseline / 2) x R_right, where P center represents the central capture position, baseline equals 0.064 meters, and R right is the normalized right vector [sin(O col), 0, -cos(O col)] for azimuth angle O col. The vergence adjustment may modify the camera orientations to converge at the target depth, with left eye forward vector F_left = normalize([cos(0_col + a_vergence / 2), sin(tilt_angle), sin(0_col + a_vergence / 2)]) and right eye forward vector F_right = normalize([cos(0_col - a_vergence / 2), sin(tilt_angle), sin(0_col - a_vergence / 2)]), where tilt_angle equals ±π / 4 for above / below-horizon projections and a vergence represents the calculated vergence angle for the current column position.

[0048] The clip-space coordinate construction may set the X component to 0.0 since each column in the panoramic capture may be one pixel wide, making the horizontal center of the column the appropriate X coordinate. The Y component may be derived from the UV coordinates by converting from the 0 to 1 texture coordinate range to the -1 to 1 clip-space range using the transformation Y = (correctedYx2.(Y) — 1.0. The Z component may be set directly to the extracted depth value sampled from the depth texture. The resulting clip-space coordinates may then be transformed through the inverse projection matrix and inverted view matrix to obtain the TSZ-00325

[0049] final world-space positions that represent the original three-dimensional geometry captured in the stereo panoramic image.

[0050] With continued reference to FIG. 1, a runtime depth buffer may be seeded with the reconstructed world-space positions (step 108). The seeded depth buffer may enable depth-based occlusion relationships between the static panoramic content and subsequently rendered dynamic objects. In some cases, the seeding process may write depth values to the depth buffer such that the color and depth buffer contain values as if the scene were generated with regular rasterization draw calls.

[0051] The depth buffer seeding operation may involve transforming the reconstructed world-space positions back into the current camera's clip space using the runtime projection and view matrices. The transformed positions may undergo perspective division to obtain normalized device coordinates, where the resulting Z-component may be written directly to the depth buffer at the corresponding pixel locations. In some cases, the depth buffer seeding may utilize the same depth buffer format queried from the system's depth pass descriptor, typically D32_SFloat or D24_UNorm_S8_UInt on mobile virtual reality platforms, ensuring compatibility with subsequent rendering operations.

[0052] The seeding process may write depth values as 32-bit floating point values, though the graphics hardware may automatically convert these values to match the currently bound depth buffer format. The depth buffer seeding may respect the reversed Z-buffer convention where applicable, maintaining consistency with the system's depth testing configuration. In some cases, the seeded depth values may be written using standard depth testing operations, where closer objects may overwrite farther objects according to the established depth comparison function.

[0053] The runtime depth buffer seeding may establish a foundation for proper occlusion testing between the static panoramic content and dynamic three-dimensional objects rendered in subsequent passes. The seeded depth information may enable the graphics processing unit to perform standard depth testing operations during dynamic object rendering, where fragments TSZ-00325

[0054] from dynamic objects may be discarded if they fail the depth test against the seeded panoramic depth values. In some cases, the depth buffer seeding may create the illusion that the panoramic content was rendered using traditional geometric primitives, allowing standard rasterization techniques to interact naturally with the pre- captured environmental content.

[0055] The runtime depth buffer seeding may establish a foundation for proper occlusion testing between the static panoramic content and dynamic three-dimensional objects rendered in subsequent passes. The seeded depth information may enable the graphics processing unit to perform standard depth testing operations during dynamic object rendering, where fragments from dynamic objects may be discarded if they fail the depth test against the seeded panoramic depth values. In some cases, the depth buffer seeding may create the illusion that the panoramic content was rendered using traditional geometric primitives, allowing standard rasterization techniques to interact naturally with the pre- captured environmental content.

[0056] Dynamic three-dimensional objects may be rendered using standard rasterization techniques (step 110). The dynamic objects may interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships. In some cases, the dynamic three-dimensional objects may be constrained to move within predetermined regions of the virtual environment while a viewer position remains fixed to rotational movement. The standard rasterization techniques may enable the dynamic objects to naturally occlude or be occluded by the static panoramic content based on the depth information established in the step 108.

[0057] The standard rasterization techniques may utilize Unity's standard rendering pipeline, where the color and depth buffers have been prepared with panoramic content such that subsequent rendering operations proceed as if the scene were generated with regular rasterization draw calls. The graphics processing unit may employ the same projection and view parameters for rendering dynamic objects as used for the panoramic content, ensuring consistent coordinate systems and depth relationships throughout the hybrid rendering process. In some cases, the dynamic object TSZ-00325

[0058] rendering may utilize Unity's layer system with cullingMask parameters to determine which geometric elements are processed during the rasterization operations.

[0059] The occlusion relationships may be established through standard depth testing operations, where fragments from dynamic objects may be compared against the seeded depth values in the runtime depth buffer. Dynamic object fragments that are positioned behind static panoramic surfaces may be discarded during the depth test, while fragments positioned in front of the panoramic content may overwrite the corresponding depth buffer values and contribute to the final rendered image. The depth testing may respect the reversed Z-buffer convention where applicable, maintaining consistency with the depth format established during the panoramic depth buffer seeding process.

[0060] The constraint of dynamic objects to predetermined regions may enable efficient resource allocation while maintaining the illusion of a fully interactive three-dimensional environment. The predetermined regions may correspond to areas where dynamic activity is expected or required by the application design, such as the surface of a pool for water simulation or specific zones for fire and flame effects. The regional constraints may allow the rendering system to focus computational resources on areas with dynamic content while leveraging the high-resolution static panoramic imagery for environmental details that do not require real-time modification.

[0061] The viewer position constraint to rotational movement may align with the three degrees of freedom limitation inherent in the panoramic capture approach, where the static content represents the environment from a fixed spatial location. The rotational movement may include yaw, pitch, and roll rotations that allow users to look around the virtual environment while maintaining the spatial reference point established during the panoramic capture process. In some cases, the rotational constraint may prevent translational movement that could reveal the limitations of the static panoramic representation, such as parallax inconsistencies or missing geometric information that would be visible from different spatial positions. TSZ-00325

[0062] The integration of dynamic objects with panoramic content may enable complex visual effects including shadows, reflections, and lighting interactions between dynamic and static elements. Dynamic objects may cast shadows onto the static panoramic surfaces through standard shadow mapping techniques, where the shadow calculations may utilize the reconstructed world-space positions from the panoramic depth data. The lighting calculations may treat the panoramic content as if it were composed of traditional geometric surfaces, enabling dynamic objects to receive lighting contributions from the static environment and vice versa.

[0063] Transparent objects may be rendered using back-to-front sorting after the opaque dynamic objects have been processed, ensuring proper alpha blending relationships with both the panoramic content and other dynamic elements. The transparency rendering may utilize standard alpha blending operations where transparent dynamic objects may be composited over the panoramic background content. In some cases, transparency may not be supported within the panoramic content itself due to the nature of the capture and storage process, limiting transparent effects to the dynamically rendered three-dimensional objects.

[0064] The performance characteristics of the dynamic object rendering may benefit from the reduced geometric complexity compared to traditional fully three-dimensional environments, as the static environmental details are represented through the panoramic imagery rather than geometric primitives. The computational savings from the hybrid approach may enable higher quality dynamic object rendering, including enhanced lighting models, particle effects, or more complex geometric detail for the dynamic elements. In some cases, the rendering system may achieve frame rate targets of 90fps on mobile virtual reality hardware by balancing the computational load between the efficient panoramic display and the focused dynamic object processing.

[0065] The method 100 may enable a hybrid rendering approach where computational resources may be allocated efficiently between static high-resolution panoramic content and dynamic three-dimensional elements. In some cases, a non-transitory computer-readable medium may store instructions that, when executed by a processor, may cause the processor to perform the TSZ-00325

[0066] operations described in the method 100, including rendering the full-screen quad, extracting depth values, reconstructing world-space positions, seeding the runtime depth buffer, and rendering the dynamic objects with proper occlusion relationships.

[0067] Referring now to FIG. 2, a method 200 for capturing stereo panoramic images may be implemented to generate the static panoramic content used in the hybrid rendering approach. The method 200 may provide a systematic approach for capturing high-resolution panoramic imagery with associated depth information using above-horizon and below- horizon projections. In some implementations, the captured panoramic data from this process may subsequently be processed by the process described in method 100 to enable occlusion relationships with dynamic three-dimensional content.

[0068] The method 200 begins by setting a target resolution for a three hundred sixty degree image (step 202). The target resolution may be selected from various standard display resolutions including, but not limited to, 640x480 pixels (VGA), 800x600 pixels (SVGA), 1024x768 pixels (XGA), 1920x1080 pixels (Full HD), 3840x2160 pixels (4K UHD), and 8192x4096 pixels for high-resolution virtual reality applications. In some cases, the target resolution may be 8192x4096 pixels for the capture, providing sufficient detail for high-quality virtual reality rendering.

[0069] Alternatively, the target resolution may be set to 1920x1080 pixels for applications with moderate computational requirements, or 3840x2160 pixels for 4K UHD virtual reality experiences. The system may automatically select the optimal target resolution based on the capabilities of the target virtual reality hardware platform, available memory bandwidth, and desired frame rate performance. Lower resolutions such as 640x480 pixels or 800x600 pixels may be utilized for legacy systems or applications with strict performance constraints, while higher resolutions including 8192x4096 pixels may be employed for premium virtual reality experiences requiring maximum visual fidelity. The target resolution may determine the number of columns that need to be rendered during the panoramic capture process, where higher TSZ-00325

[0070] resolutions require more discrete rotation steps and longer capture times but provide enhanced image quality and reduced pixelation artifacts in the final virtual reality presentation.

[0071] The camera may be positioned at forty-five degrees above horizon direction for the image (step 204). The above-horizon projection may capture imagery with the camera pointed 45 degrees above a horizon to cover the upper portion of the vertical field of view. The 45 -degree tilt angle may be selected to provide optimal spherical coverage while maintaining computational efficiency, as the tangent of 45 degrees equals 1.0, simplifying the trigonometric calculations required for equirectangular coordinate transformations. Alternatively, the above-horizon projection may utilize tilt angles ranging from 30 degrees to 60 degrees above the horizon, where 30-degree tilts may provide enhanced overlap with horizon-level content for seamless blending, while 60-degree tilts may offer increased coverage of overhead environmental features such as sky domes or ceiling structures. In some embodiments, the tilt angle may be dynamically adjusted based on the content characteristics of the virtual environment, where environments with significant overhead detail may benefit from steeper tilt angles approaching 75 degrees, while environments with minimal overhead content may utilize shallower angles of 15 to 30 degrees to allocate more resolution to horizon-level features.

[0072] The camera positioning may utilize a standard pinhole camera model with configurable intrinsic and extrinsic parameters to achieve the desired upward tilt orientation. In alternative implementations, the intrinsic matrix may be configured with varying focal length parameters to provide different field of view coverage, ranging from wide-angle capture with expanded coverage to narrow-angle capture with enhanced detail resolution. The principal point coordinates may be adjusted to accommodate different sensor formats, including configurations for various aspect ratios such as 16:9 sensors or square sensor configurations, enabling flexibility in the capture system design based on the specific requirements of the virtual reality application. The camera model may alternatively employ fisheye lens distortion models with radial distortion coefficients kl, k2, k3 and tangential distortion coefficients pl, p2, enabling capture of wider TSZ-00325

[0073] fields of view exceeding 180 degrees per projection half while requiring additional undistortion processing during the coordinate transformation operations. In some cases, the system may utilize omnidirectional camera models including equidistant projection with r = f x 0, equisolid angle projection with r = 2f x sin(0 / 2), or orthographic projection with r = f x sin(0), where f represents the focal length and 0 represents the incident angle from the optical axis. The extrinsic rotation matrices may incorporate additional rotation axes, such as R compound = R_z(cp) x R_y(0) x R_x(v) for compound rotations including roll, pitch, and yaw adjustments, enabling precise camera orientation control for complex capture scenarios or non-standard mounting configurations.

[0074] The projection system may support adaptive resolution scaling where the intrinsic matrix parameters are dynamically adjusted based on the target output resolution, computational constraints, or quality requirements. High-resolution configurations may utilize K_4K = [1024.0, 0.0, 2048.0; 0.0, 1024.0, 2048.0; 0.0, 0.0, 1.0] for 4096x4096 pixel capture, while mobile-optimized configurations may employ K_mobile = [256.0, 0.0, 512.0; 0.0, 256.0, 512.0; 0.0, 0.0, 1.0] for 1024x1024 pixel capture to reduce memory bandwidth and processing requirements. The system may implement multi-resolution capture where different regions of the panoramic image are captured at varying resolutions based on perceptual importance, with central viewing areas captured at higher resolution and peripheral areas captured at reduced resolution to optimize the balance between visual quality and computational efficiency.

[0075] The top half of a column may be rendered using upward perspective (step 206). The rendering process may generate 1 x2048 pixel column chunks that correspond to the upper portion of the vertical field of view. In some cases, the upward perspective rendering may capture the portion of the environment visible above the horizon line using the forty-five degree camera angle established in the step 204.

[0076] The upward perspective rendering may utilize the intrinsic camera matrix K above = [512.0, 0.0, 1024.0; 0.0, 512.0, 1024.0; 0.0, 0.0, 1.0] with focal length components of 512.0 pixels TSZ-00325

[0077] corresponding to the 90-degree field of view coverage. The extrinsic rotation matrix R above = [1.0, 0.0, 0.0; 0.0, 0.707, -0.707; 0.0, 0.707, 0.707] may be applied to achieve the upward tilt orientation, where the rotation values correspond to cos(?r / 4) and sin(7r / 4) = 0.707. The principal point coordinates (1024.0, 1024.0) may be centered within the 2048x2048 capture resolution per projection half, ensuring proper geometric alignment during the column rendering process.

[0078] The column chunk generation may involve systematic pixel-by-pixel rendering where each vertical strip represents a discrete azimuthal slice of the panoramic environment. The 1 x2048 pixel dimensions may provide sufficient vertical resolution to capture fine environmental details while maintaining computational efficiency during the capture process. In some cases, the column chunks may be processed sequentially as the camera rotates through discrete angular increments, building the complete panoramic image through systematic accumulation of vertical strips.

[0079] The upward perspective may capture environmental features including sky domes, overhead architectural elements, ceiling structures, and elevated terrain features that are positioned above the horizon reference plane. The forty-five degree upward angle may provide optimal coverage of overhead content while maintaining geometric consistency with the complementary below-horizon projection used for the lower portion of each column. The rendering process may utilize standard rasterization techniques with depth buffer capture to preserve both color information and distance measurements for each pixel within the column chunk.

[0080] The stereo baseline adjustments may be applied during the upward perspective rendering, where the camera position may be offset by half the inter-pupillary distance along the appropriate horizontal vector to generate separate left and right eye perspectives. The vergence calculations may modify the camera orientation to ensure proper stereo convergence at the target depth, with vergence angles calculated as avergence= arctan(0.032 / target_depth) where 0.032 represents half the 64mm baseline distance. The stereo adjustments may maintain consistent vertical TSZ-00325

[0081] alignment between left and right eye views to prevent vertical disparity artifacts that could cause viewer discomfort during virtual reality presentation.

[0082] The camera may be positioned at forty-five degrees below horizon direction of the image (step 208). The below-horizon projection may capture imagery with the camera as described above with respect to above-horizon projections, for example, the camera may be pointed 45 degrees below the horizon to cover the lower portion of the vertical field of view. The camera positioning may maintain the same focal length parameters as the above-horizon capture while adjusting the orientation downward using intrinsic matrix K_below = [512.0, 0.0, 1024.0; 0.0, -512.0, 1024.0; 0.0, 0.0, 1.0] with inverted Y-axis scaling and extrinsic rotation matrix R_below = [1.0, 0.0, 0.0; 0.0, 0.707, 0.707; 0.0, -0.707, 0.707] to achieve the downward tilt orientation. The consistent focal length of 512 pixels across both projections may ensure seamless integration at the horizon boundary while maintaining the 90-degree field of view coverage per projection half.

[0083] The bottom half of the column may be rendered using downward perspective (step 210) in a manner similar to the rendering of the top half of a column, as described above. For example, the downward perspective rendering may generate the remaining portion of the 1 2048 pixel column chunk corresponding to the lower portion of the vertical field of view. In some cases, the combination of the step 206 and the step 210 may provide a total vertical field of view of 180 degrees with 90 degrees above horizon and 90 degrees below horizon.

[0084] With continued reference to FIG. 2, the method 200 may continue to a step 212 which may present a decision point asking whether all columns are rendered for full rotation. The decision point may determine whether the panoramic capture process has completed a full 360-degree rotation around the capture position. In some cases, the decision may be based on whether the camera has been rotated through all discrete steps needed to capture the complete panoramic environment.

[0085] If the decision in the step 212 indicates that all columns have been rendered, the method 200 may conclude with a step 214 where complete stereo color and depth texture data may be saved. The TSZ-00325

[0086] saved data may include both the color information and depth information for each pixel in the panoramic image, providing the static content needed for the hybrid rendering system. In some cases, the saved data may be stored in formats suitable for real-time access during virtual reality rendering operations.

[0087] If the decision in the step 212 indicates that additional columns remain to be rendered, the method 200 may proceed to a step 216 where the camera may be rotated by a column increment. The camera rotation may occur in discrete steps of 360 degrees divided by the target image width to generate column-wise image strips. The discrete rotation step Δθ = 2π / image\_width may equal 0.044 degrees for an 8192 pixel wide image, ensuring systematic azimuthal coverage without gaps or overlaps between adjacent columns. For alternative resolutions, the rotation increment may be adjusted accordingly, with 4096 pixel wide images requiring 0.088 degree steps, 2048 pixel wide images requiring 0.176 degree steps, and 1024 pixel wide images requiring 0.352 degree steps to maintain complete horizontal coverage.

[0088] The camera's world position may remain fixed at the capture origin throughout the entire rotation sequence, while the orientation matrices may be updated by multiplying the base rotation

[0089] ' cos(0col) 0 sin(0col)’ matrices Raboveand Rbelowwith the azimuthal rotation matrix Razimuth= 0 1 0 -sin(0col) 0 cos(0col). for each column position 0col. The azimuthal rotation matrix represents a rotation about the Y-axis (vertical axis) in a right-handed coordinate system, where positive angles correspond to counterclockwise rotation when viewed from above. The combined rotation matrices for each Column may be calculated as Rabove.col= Razimuth(θcol) × Raboveand Rbelow.col= Razimuth(θcol) × Rbelow, ensuring that the 45-degree elevation angles are maintained while the horizontal viewing direction is systematically adjusted for each column position.

[0090] The stereo baseline offsets may be recalculated for each azimuth to maintain proper inter-pupillary distance relationships throughout the rotation sequence. The left and right eye camera baseline positions may be calculated using the formulas Pleft= Pcenter-

[0091]

[0092] (baseline / 2) × Rright(θcol); Pleft= TSZ-00325

[0093]

[0094] Pright= Pcenter+ (baseline / 2) × Rright(θcol)

[0095]

[0096] and Pright= P center + x Rright(dcol), where Pcenter represents the central capture position, baseline equals 0.064 meters, and Rright ficoi = [sin(^coj) isthe normalized right vector for the current azimuth angle. The vergence angles may be adjusted according to αvergence= arctan(0.032 / target_depth) where 0.032 represents half the 64mm baseline distance and target depth represents the desired convergence distance, typically set to 2.0 meters for optimal stereo viewing comfort. The vergence adjustment may modify the camera forward vectors to Fleft=

[0097] normalize([cos(

[0098]

[0099] θcol+αvergence / 2), sin(tilt_angle), sin(θcol+αvergence / 2)]) and Fleft= normalize([cos(

[0100]

[0101] θcol+αvergence / 2), sin(tilt_angle)]) Fright=

[0102] normalize([cos(θcol- αvergence / 2), sin(tilt_angle), sin(θcol- αvergence / 2)]) and Fright=

[0103] n

[0104]

[0105] ormalize([cos(θcol— αvergence / 2), sin(tilt_angle), sin(θcol— αvergence / 2)]), where tilt_angle equals ±π / 4 for above / below-horizon projections respectively, ensuring that the left and right eye views converge at the target depth while maintaining identical vertical orientations to prevent vertical disparity artifacts.

[0106] In an alternative embodiment, the stereo baseline calculations may be performed without vergence adjustments. In this configuration, both left and right eye cameras may be oriented to look straight ahead at each azimuth position, with forward vectors calculated as Fleft= normalize([cos(θcol), sin(tilt_angle), sin(θcol)]) and Fright= normalize([cos(θcol), sin(tilt_angle), sin(θcol)]), where both eyes maintain identical orientations while being separated only by the baseline distance. This approach may eliminate the computational overhead associated with vergence calculations while relying on the natural stereo disparity created by the 64mm baseline separation to provide depth perception cues during virtual reality viewing.

[0107] The column increment process may also involve updating the projection matrices to account for any lens distortion correction or field of view adjustments needed for the current azimuthal position. In some cases, the system may implement adaptive sampling where the angular TSZ-00325

[0108] increment may be dynamically adjusted based on the geometric complexity or visual importance of different azimuthal regions, with areas containing fine details receiving smaller angular increments and areas with uniform content receiving larger increments to optimize capture efficiency. The rotation mechanism may utilize high-precision computational positioning algorithms to ensure accurate virtual positioning, with verification systems providing confirmation that the desired angular position has been achieved before initiating the rendering sequence for each column.

[0109] After the step 216, the method 200 may return to the step 204 to continue the capture process for the next column position, creating a systematic loop that processes each azimuthal slice of the panoramic environment until the complete 360-degree capture has been accomplished. The iterative process may continue until all columns have been rendered, at which point the decision in the step 212 will indicate completion and the method will proceed to the step 214 for final data storage and processing.

[0110] The method 200 may enable systematic capture of panoramic imagery where each column may be rendered in two halves using different camera orientations. In some cases, the method 200 may be implemented as part of a stereo panoramic capture system that generates the static content used by virtual reality rendering systems, computer-implemented methods, and non-transitory computer-readable media for hybrid rendering applications.

[0111] Referring now to FIG. 3, a method 300 for depth reconstruction to transform depth information from the stereo panoramic image into world-space positions suitable for seeding a runtime depth buffer is described. The method 300 forms a component of the process that reconstructs world space positions of each pixel within panoramic imagery previously captured through the process utilizing standard rasterization pipelines. The method 300 may provide a systematic approach for reconstructing original world-space positions by applying inverse matrices corresponding to capture parameters used during panoramic image generation, thereby enabling proper occlusion calculations with traditionally rasterized dynamic geometry. TSZ-00325

[0112] The method 300 may begin with a step 302 where quad UV coordinates may be transformed using equirectangular projection coordinates. In some cases, the transformation may convert the quad's UV coordinates to sampling coordinates for the stereo panoramic image data using an EquirectangularCoords shader function. The EquirectangularCoords function may perform coordinate conversion by calculating phi values using modulo operations and tangent functions to map UV coordinates to spherical coordinates based on above-horizon and below-horizon regions.

[0113] The depth sample may be retrieved from the stereo panoramic depth map (step 304). The depth sample may correspond to distance measurements stored in the panoramic image at the transformed coordinates from the step 302. In some cases, the depth data may be stored using D32_SFloat or D24_UNorm_S8_UInt formats queried from UNITY, a cross-platform game engine, developed by Unity Technologies, of San Francisco, California, depth pass descriptor, providing depth buffer storage with appropriate precision for mobile virtual reality applications. The UV coordinates may be snapped to exact pixel centers to match the original capture parameters (step 306). The snapping process may involve multiplying UV coordinates by pixel dimensions, taking a floor operation, adding 0.5, and dividing by the pixel dimensions. In some cases, a SnapToGrid shader function may implement the snapping operation to ensure that the reconstructed view parameters match exactly with the parameters used during the original panoramic capture process.

[0114] The method 300 may advance to a step 308 which may present a decision point determining whether the sample is from an above-horizon or below-horizon region. The decision may be based on the vertical coordinate position within the stereo panoramic image, where samples in the upper portion may correspond to above-horizon regions and samples in the lower portion may correspond to below-horizon regions.

[0115] If the decision in the step 308 indicates that the sample is from an above-horizon region, the method 300 may branch to a step 310 where original camera parameters for above-horizon TSZ-00325

[0116] capture may be calculated. The calculation may involve determining the camera position and orientation corresponding to the forty-five degree above-horizon projection used during the panoramic capture process. The getRays shader function may calculate the original values for the camera's view position, up direction vectors, and forward vectors based on the above-horizon designation, utilizing the intrinsic matrix K_above = [512.0, 0.0, 1024.0; 0.0, 512.0, 1024.0; 0.0, 0.0, 1.0] and extrinsic rotation matrix R_above = [1.0, 0.0, 0.0; 0.0, 0.707, -0.707; 0.0, 0.707, 0.707], The stereo eye offset may be applied using the baseline distance of 64mm, with vergence correction calculated as a_vergence = arctan(0.032 / 2.0) for the 2-meter target depth, resulting in vergence-adjusted forward vectors that maintain proper stereo geometry without introducing vertical disparity.

[0117] In an alternative embodiment, the above-horizon camera parameter calculation may be performed without vergence corrections. In this configuration, the getRays shader function may calculate identical forward vectors for both left and right eye positions, with the forward vector determined as F_forward = normalize([cos(0_col), sin(7r / 4), sin(0_col)]) for the 45-degree upward tilt. The stereo separation may be achieved solely through the baseline offset positioning, where the left and right cameras maintain parallel viewing directions while being separated by the 64mm inter-pupillary distance. This approach may simplify the camera parameter calculations while maintaining effective stereo imaging through positional disparity alone.

[0118] If the decision in the step 308 indicates that the sample is from a below-horizon region, the method 300 may branch to a step 312 where original camera parameters for below-horizon capture may be calculated. The calculation may determine the camera position and orientation corresponding to the forty-five degree below-horizon projection used during the panoramic capture process. The getRays function may calculate the camera parameters using the intrinsic matrix K below = [512.0, 0.0, 1024.0; 0.0, -512.0, 1024.0; 0.0, 0.0, 1.0] with inverted Y-axis focal length and extrinsic rotation matrix R_below = [1.0, 0.0, 0.0; 0.0, 0.707, 0.707; 0.0, -0.707, 0.707] for the downward tilt configuration. The stereo baseline calculations may mirror the TSZ-00325

[0119] above- horizon approach, with left and right eye positions offset by ±32mm along the horizontal right vector, and vergence angles applied to ensure the camera forward vectors converge at the target depth while maintaining identical vertical orientations to eliminate vertical disparity between the stereo pair.

[0120] In an alternative embodiment, the below-horizon camera parameter calculation may be performed without vergence adjustments. In this configuration, both left and right eye cameras may be oriented to look straight ahead with identical forward vectors calculated as F forward = normalize([cos(0_col), sin(-π / 4), sin(0_col)]) for the 45-degree downward tilt. The stereo separation may be achieved through baseline positioning alone, where the cameras maintain parallel viewing directions while being offset by the 64mm inter-pupillary distance. This approach may reduce computational complexity during the depth reconstruction process while relying on the natural stereo disparity created by the baseline separation to provide depth information.

[0121] With continued reference to FIG. 3, both the step 310 and the step 312 may converge at a step 314 where clip-space coordinates may be constructed with the extracted depth values.

[0122] The clip-space coordinate construction may set the X component to 0.0 since each column in the panoramic capture may be one pixel wide, making the horizontal center of the column the appropriate X coordinate for the clip-space transformation. This single-pixel width constraint arises from the column-wise rendering approach where the camera rotates in discrete steps of 360° / image\_width, capturing vertical strips that are exactly one pixel wide in the horizontal dimension. The Y component may be derived from the UV coordinates by converting from the normalized 0 to 1 texture coordinate range to the -1 to 1 clip-space range using the transformation Y = (correctedY×2.0) − 1.0, where correctedY represents the UV coordinate that has been processed through the equirectangular coordinate transformation and adjusted for the appropriate stereo eye view. The coordinate conversion accounts for the standard graphics pipeline convention where clip-space coordinates range from -1 to 1 in both horizontal and TSZ-00325

[0123] vertical dimensions, while texture coordinates typically range from 0 to 1. The Z component may be set directly to the extracted depth value from the step 304, representing the distance measurement sampled from the depth texture at the corresponding equirectangular projection coordinates. This depth value maintains the original precision and format from the panoramic capture process, whether stored as 32-bit floating point values in reversed Z-buffer format or alternative depth representations, ensuring accurate world-space position reconstruction when combined with the inverse projection and view matrix transformations.

[0124] An inverse projection matrix and an inverted view matrix transformation may be applied to convert the clip-space coordinates to world-space positions (step 316). The inverse projection matrices may be calculated as K_above-1= [1 / 512.0, 0.0, -2.0; 0.0, 1 / 512.0, -2.0; 0.0, 0.0, 1.0] for above-horizon samples and K_below-1= [1 / 512.0, 0.0, -2.0; 0.0, -1 / 512.0, 2.0; 0.0, 0.0, 1.0] for below-horizon samples, where the principal point offset of -2.0 accounts for the centered 1024-pixel principal point in the 2048-pixel capture resolution. The InvertedLookAt shader function may construct the inverted view matrix using the camera position, forward vector, up vector, and right vector calculated in the step 310 or the step 312, with the view matrix V = [R right.x, R up.x, -F forward.x, -dot(R_right, P camera); R right.y, R up.y, -F forward.y, -dot(R_up, P camera); R right.z, R up.z, -F forward.z, dot(F_forward,

[0125] P camera); 0, 0, 0, 1] and its inverse V-1obtained through matrix inversion. The matrices may be stored in row-major format following UNITY, a cross-platform game engine, developed by Unity Technologies, of San Francisco, California, and DIRECT3D, a graphics application programming interface, developed by Microsoft Corporation, of Redmond, Washington, convention as evidenced by Mat4 struct layout, enabling proper matrix operations during the transformation process.

[0126] The transformed depth value may be written to the runtime depth buffer (step 318). The depth buffer writing may enable proper occlusion relationships between static panoramic content and dynamic three-dimensional objects rendered subsequently. In some cases, the depth values may TSZ-00325

[0127] be provided as 32-bit floating point values, though graphics hardware may automatically convert the values to whatever format depth buffer is currently bound for the rendering operation.

[0128] The depth buffer writing process may involve transforming the reconstructed world-space positions back into the current camera's clip space using the runtime projection and view matrices. The transformed positions may undergo perspective division to obtain normalized device coordinates, where the resulting Z-component may be written directly to the depth buffer at the corresponding pixel locations. The depth buffer format may be queried from the system's depth pass descriptor, typically D32_SFloat or D24_UNorm_S8_UInt on mobile virtual reality platforms, ensuring compatibility with subsequent rendering operations.

[0129] The depth buffer seeding may respect the reversed Z-buffer convention where applicable, maintaining consistency with the system's depth testing configuration. In reversed Z-buffer systems, depth values of 1.0 may represent the near plane while values of 0.0 may represent the far plane, providing enhanced depth precision compared to traditional Z-buffer arrangements by distributing floating point precision more evenly across the depth range. The seeded depth values may be written using standard depth testing operations, where closer objects may overwrite farther objects according to the established depth comparison function.

[0130] The runtime depth buffer writing may establish a foundation for proper occlusion testing between the static panoramic content and dynamic three-dimensional objects rendered in subsequent passes. The seeded depth information may enable the graphics processing unit to perform standard depth testing operations during dynamic object rendering, where fragments from dynamic objects may be discarded if they fail the depth test against the seeded panoramic depth values. The depth buffer seeding may create the illusion that the panoramic content was rendered using traditional geometric primitives, allowing standard rasterization techniques to interact naturally with the pre- captured environmental content.

[0131] The depth writing operation may utilize the same coordinate system transformations as the dynamic object rendering, including UnityToHephPoint and UnityToHephQuaternion functions TSZ-00325

[0132] for alignment between different coordinate systems. The graphics processing unit may employ matrix operations stored in row-major format following UNITY, a cross-platform game engine, developed by Unity Technologies, of San Francisco, California, and DIRECT3D, a graphics application programming interface, developed by Microsoft Corporation, of Redmond, Washington, convention, ensuring proper transformation calculations during the depth buffer seeding process. In some cases, the depth buffer writing may target performance specifications of less than 1ms for depth buffer seeding on SNAPDRAGON XR2, a system-on-chip processor, manufactured by Qualcomm Incorporated, of San Diego, California, processors, enabling efficient real-time rendering within mobile virtual reality hardware constraints.

[0133] The method 300 may enable accurate reconstruction of world-space positions from panoramic depth data through systematic application of inverse transformations. In some cases, the method 300 may be implemented within virtual reality rendering systems, computer-implemented methods, and non-transitory computer-readable media that utilize the reconstructed depth information to provide correct occlusion relationships in hybrid rendering applications.

[0134] Referring now to FIG. 4, a method 400 for virtual reality rendering with stereo handling may be implemented to process stereo panoramic images with vertically stacked views for left and right eye perspectives. The method 400 represents an implementation of the process that takes panoramic imagery generated using standard rasterization pipelines and reconstructs world space positions of each pixel to enable occlusion with traditionally rasterized geometry. The method 400 may provide a systematic approach for handling stereo panoramic content where a left eye view and a right eye view may be captured with an inter-pupillary distance baseline and vertically stacked in a single texture.

[0135] The method 400 may begin by loading a stereo panoramic image with vertically stacked views (step 402). The stereo panoramic image may contain both left eye and right eye perspectives arranged vertically within a single texture, where the left eye view may occupy the upper portion corresponding to texture coordinates ranging from 0.0 to 0.5 in the vertical dimension, and the TSZ-00325

[0136] right eye view may occupy the lower portion corresponding to texture coordinates ranging from 0.5 to 1.0 in the vertical dimension. In some cases, the vertically stacked arrangement may enable efficient storage and processing of stereo content while maintaining separate perspective information for each eye, reducing memory bandwidth requirements compared to side-by-side stereo arrangements that would require wider texture formats.

[0137] The stereo panoramic image loading process may involve accessing the stereo panoramic image storage 512 within the memory subsystem 508, where both the color data 514 and depth data 516 are stored in the vertically stacked format. The loading operation may utilize texture streaming techniques to efficiently transfer the panoramic data from system memory to graphics memory, where the 8192x8192 pixel stereo panoramic image may consume approximately 268MB uncompressed or approximately 65MB with ASTC 4x4 block compression applied to the color data. The depth data 516 may remain uncompressed at approximately 268MB for full 32-bit floating point precision, ensuring accurate depth reconstruction during subsequent processing steps.

[0138] The vertically stacked stereo format may provide several advantages over alternative stereo arrangements including reduced texture binding operations during rendering, simplified coordinate transformations for stereo eye separation, and improved cache coherency when accessing corresponding pixels between left and right eye views. The vertical arrangement may align with the column-wise capture methodology used during panoramic image generation, where each column represents a discrete azimuthal slice captured with both above-horizon and below-horizon projections for each stereo eye perspective. In some cases, the loading process may include validation checks to ensure the stereo panoramic image contains the expected resolution, format, and stereo eye arrangement before proceeding with subsequent rendering operations.

[0139] The stereo panoramic image may be captured with an inter-pupillary distance baseline of 64mm, where the left eye camera position is offset by -32mm along the horizontal right vector and the TSZ-00325

[0140] right eye camera position is offset by +32mm along the horizontal right vector from the central capture position. The vergence adjustments may be applied during capture to ensure proper stereo convergence at a target depth of 2.0 meters, with vergence angles calculated as Mergence = arctan(0.032 / 2.0) « 0.92° to provide comfortable stereo viewing without vertical disparity artifacts. The vertically stacked arrangement may preserve these stereo relationships while enabling efficient processing through the graphics pipeline using unity StereoEyelndex to determine which portion of the texture should be accessed for the current rendering pass.

[0141] The method 400 may proceed to a step 404 which may present a decision point determining whether rendering is for the left eye or the right eye. The decision may be based on a stereo eye index that indicates which eye perspective should be processed during the current rendering pass. In some cases, the graphics processing unit may support StereoRenderMode. Multiview for ANDROID, a mobile operating system, developed by Google LLC, of Mountain View, California, using VULKAN, a low-level graphics application programming interface, developed by the Khronos Group, of Beaverton, Oregon, multiview extension and StereoRenderMode. Instancing for desktop, enabling efficient stereo rendering through different rendering modes depending on the target platform.

[0142] If the decision in the step 404 indicates that rendering is for the left eye, texture coordinates may be converted from a zero to half range (step 406).

[0143] The coordinate conversion may map the upper portion of the vertically stacked stereo panoramic image to a normalized coordinate range for left eye processing. The upper portion may correspond to texture coordinates ranging from 0.0 to 0.5 in the vertical dimension, where 0.0 represents the topmost edge of the stereo panoramic image and 0.5 represents the horizontal dividing line between the left and right eye views. In some cases, the conversion may involve scaling texture coordinates from a range of 0 to 0.5 to a range of 0 to 1.0 to access the left eye view data within the stereo panoramic image. TSZ-00325

[0144] The coordinate scaling operation may be implemented using the transformation vnormalized= vtexturex 2.0, where vtexturerepresents the original vertical texture coordinate in the 0 to 0.5 range and vnormalizedrepresents the scaled coordinate in the 0 to 1.0 range. This scaling transformation may enable the fragment shader to sample the left eye view data using standard normalized texture coordinates, simplifying the texture sampling operations and maintaining consistency with conventional graphics pipeline expectations. The coordinate conversion may be performed in conjunction with the unity StereoEyeindex parameter, which may indicate whether the current rendering pass corresponds to the left eye (index 0) or right eye (index 1) perspective. The left eye coordinate conversion may preserve the spatial relationships captured during the panoramic image generation process, where the left eye camera position was offset by -32mm along the horizontal right vector from the central capture position. The coordinate mapping may maintain the inter-pupillary distance baseline of 64mm established during capture, ensuring that the left eye view data accessed through the normalized coordinates corresponds to the appropriate stereo perspective for comfortable virtual reality viewing. In some cases, the coordinate conversion may be combined with vergence corrections that were applied during capture to ensure proper stereo convergence at the target depth of 2.0 meters, preventing vertical disparity artifacts that could cause viewer discomfort.

[0145] If the decision in the step 404 indicates that rendering is for the right eye, texture coordinates may be converted from a half to one range (step 408).

[0146] The coordinate conversion may map the lower portion of the vertically stacked stereo panoramic image to a normalized coordinate range for right eye processing. The lower portion may correspond to texture coordinates ranging from 0.5 to 1.0 in the vertical dimension, where 0.5 represents the horizontal dividing line between the left and right eye views and 1.0 represents the bottommost edge of the stereo panoramic image. The conversion may involve scaling texture coordinates from a range of 0.5 to 1.0 to a range of 0 to 1.0 to access the right eye view data within the stereo panoramic image. TSZ-00325

[0147] The coordinate scaling operation may be implemented using the transformation vnormalized= (vtexture-0.5) x 2.0, where vtexturerepresents the original vertical texture coordinate in the 0.5 to 1.0 range and vnormalizedrepresents the scaled coordinate in the 0 to 1.0 range. This scaling transformation may enable the fragment shader to sample the right eye view data using standard normalized texture coordinates, maintaining consistency with the left eye coordinate conversion process while accessing the appropriate stereo perspective. The coordinate conversion may be performed in conjunction with the unity StereoEyeindex parameter, which may indicate that the current rendering pass corresponds to the right eye (index 1) perspective.

[0148] The right eye coordinate conversion may preserve the spatial relationships captured during the panoramic image generation process, where the right eye camera position was offset by +32mm along the horizontal right vector from the central capture position. The coordinate mapping may maintain the inter-pupillary distance baseline of 64mm established during capture, ensuring that the right eye view data accessed through the normalized coordinates corresponds to the appropriate stereo perspective for comfortable virtual reality viewing. In some cases, the coordinate conversion may be combined with vergence corrections that were applied during capture to ensure proper stereo convergence at the target depth of 2.0 meters, preventing vertical disparity artifacts that could cause viewer discomfort during virtual reality presentation. quirectangular coordinate transformation may be performed using a specialized shader function (step 410).

[0149] The equirectangular coordinate transformation may convert the normalized texture coordinates to spherical coordinates suitable for sampling the panoramic image data through a systematic mathematical process that accounts for the spherical nature of panoramic imagery. The transformation may map planar UV coordinates ranging from 0 to 1 in both horizontal and vertical dimensions to angular coordinates that correspond to azimuth and elevation angles within the captured panoramic environment. In some cases, the transformation may utilize the EquirectangularCoords shader function to perform coordinate conversion based on above- TSZ-00325

[0150] horizon and below-horizon regions within the stereo panoramic image, where the function calculates phi values using modulo operations and tangent functions to determine the appropriate spherical sampling coordinates.

[0151] The EquirectangularCoords shader function may implement the coordinate transformation using the formula phi = fmod(coord.y,0.5) x 2 x - - -, where the modulo operation separates the above-horizon and below-horizon regions based on the vertical texture coordinate. The function may then calculate the transformed Y coordinate using y = tan(phi) x 0.25 + 0.25 + step(0.5,coord.y), where the tangent function converts the angular phi value to a linear coordinate suitable for texture sampling, and the step function determines whether the coordinate corresponds to the upper or lower half of the stereo panoramic image. The horizontal coordinate may remain unchanged during the transformation, preserving the azimuthal relationship between the UV coordinate and the corresponding angular position within the panoramic environment. The coordinate transformation may account for the dual-projection capture methodology where the upper portion of the panoramic image corresponds to the 45-degree above-horizon projection and the lower portion corresponds to the 45-degree below-horizon projection. The transformation may ensure seamless integration between these two projection regions by applying appropriate angular corrections that maintain geometric consistency across the horizon boundary. In some cases, the EquirectangularCoords function may incorporate stereo eye separation by adjusting the coordinate calculations based on the unity StereoEyelndex parameter, ensuring that left and right eye views are properly accessed from their respective portions of the vertically stacked stereo panoramic texture.

[0152] Color data may be sampled with bilinear filtering enabled (step 412). The bilinear filtering may provide smooth interpolation between adjacent pixels in the color data, reducing visual artifacts and providing higher quality color sampling from the stereo panoramic image. The bilinear filtering process may utilize weighted averaging of the four nearest pixel values surrounding the sampling coordinate, where the weights are determined by the fractional portions of the texture TSZ-00325

[0153] coordinate relative to the pixel grid. This interpolation technique may eliminate pixelation artifacts that could occur with nearest-neighbor sampling, particularly when the panoramic content is viewed at varying distances or when the equirectangular coordinate transformation results in non-integer pixel coordinates.

[0154] In some cases, the color sampling may access the RGB texture data corresponding to the eyespecific coordinate range established in the step 406 or the step 408. The RGB texture data may be stored in various formats including 8-bit per channel (RGB888), 10-bit per channel (RGB 101010), or 16-bit per channel (RGB161616) depending on the color precision requirements and hardware capabilities of the target virtual reality platform. The bilinear filtering may be implemented using hardware texture sampling units within the graphics processing unit, providing efficient interpolation without additional computational overhead during the fragment shader execution.

[0155] The color sampling process may respect the ASTC 4x4 block compression format applied to the color data 514, where the graphics hardware may automatically decompress the compressed texture blocks during the sampling operation. The bilinear filtering may operate on the decompressed color values, ensuring that the interpolation calculations are performed on the actual RGB color data rather than the compressed representation. In some cases, the color sampling may incorporate gamma correction to ensure proper color space handling, where sRGB encoded color values may be converted to linear color space before interpolation and then converted back to the appropriate display color space for final presentation.

[0156] The bilinear filtering may maintain consistency with the point filtering used for depth sampling in the step 414, where the different filtering approaches serve distinct purposes within the hybrid rendering pipeline. While depth values require exact precision for accurate world-space position reconstruction, color values benefit from smooth interpolation to provide visually pleasing results when the panoramic content is displayed through the virtual reality headset. The color sampling may also incorporate mipmap filtering for distant viewing angles, where lower TSZ-00325

[0157] resolution versions of the panoramic texture may be sampled to reduce aliasing artifacts and improve rendering performance when fine detail is not perceptually necessary.

[0158] With continued reference to FIG. 4, depth data may be sampled using a point filtering method (step 414). The point filtering may provide exact depth values without interpolation, maintaining precise depth information needed for accurate world-space position reconstruction. The point filtering approach may utilize nearest-neighbor sampling where the depth value at the exact pixel location corresponding to the transformed equirectangular coordinates is retrieved without any averaging or blending with adjacent pixels. This sampling method may be critical for maintaining the geometric accuracy required for proper depth buffer seeding, as any interpolation of depth values could introduce errors in the subsequent world-space position calculations that would compromise the occlusion relationships between static panoramic content and dynamic three-dimensional objects.

[0159] In some cases, the depth sampling may access depth buffer values stored alongside the color data in the stereo panoramic image, where the depth values may be stored as 32-bit floating point values for maximum precision. The 32-bit floating point storage format may provide sufficient numerical precision to represent depth values across the entire range from near plane to far plane without quantization artifacts that could occur with lower precision formats such as 16-bit or 24-bit depth representations. The point filtering may be implemented using specialized texture sampling operations that bypass the graphics hardware's bilinear filtering units, ensuring that the exact depth value stored at each pixel location is retrieved without modification.

[0160] The point filtering method may contrast with the bilinear filtering applied to color data sampling in step 412, where the different filtering approaches serve distinct purposes within the hybrid rendering pipeline. While color values benefit from smooth interpolation to provide visually pleasing results when displayed through the virtual reality headset, depth values require exact precision to ensure accurate reconstruction of the original three-dimensional geometry captured during the panoramic image generation process. The point filtering may utilize the TSZ-00325

[0161] point repeat sampler defined in the shader code, which may be configured to sample textures using nearest-neighbor filtering with repeat addressing mode for handling texture coordinate wrapping at panoramic image boundaries.

[0162] The depth sampling process may respect the reversed Z-buffer format where 1.0 represents the near plane and 0.0 represents the far plane, maintaining consistency with the depth encoding used during the panoramic capture process. The point filtering may preserve the original depth precision established during capture, where the depth values were generated using the same projection matrices that will be inverted during the world-space position reconstruction process. In some cases, the depth sampling may incorporate bounds checking to ensure that sampled coordinates fall within valid texture regions, preventing sampling artifacts that could occur at panoramic image boundaries or in regions where depth data may not be available.

[0163] The exact depth values retrieved through point filtering may be essential for the subsequent depth reconstruction operations performed in step 416, where the sampled depth values will be combined with the snapped UV coordinates and inverse transformation matrices to calculate precise world-space positions. The point filtering approach may ensure that the depth reconstruction process operates on the exact depth measurements captured during the original panoramic image generation, maintaining the geometric fidelity needed for proper integration between static panoramic content and dynamically rendered three-dimensional objects in the virtual reality environment.

[0164] Depth reconstruction and buffer seeding processes may be executed (step 416). The depth reconstruction may involve applying inverse projection matrices and inverted view matrices to transform the sampled depth values into world-space positions suitable for seeding the runtime depth buffer. The inverse projection matrices may be calculated as K~^ove=

[0165] [1 / 512.0,0.0,— 2.0;0.0, 1 / 512.0,-2.0;0.0, 0.0, 1.0] for above-horizon samples and K^low=

[0166] [1 / 512.0,0.0,— 2.0;0.0,— 1 / 512.0,2.0;0.0, 0.0, 1.0] for below-horizon samples, where the principal point offset of -2.0 accounts for the centered 1024-pixel principal point in the 2048-pixel capture TSZ-00325

[0167] resolution. The InvertedLookAt shader function may construct the inverted view matrix using the camera position, forward vector, up vector, and right vector calculated based on the column position and above-horizon or below-horizon designation, with the view matrix V =

[0168] \

[0169]

[0170] Rright-X> Rup-%>~ ^forward- x, dot(RrLgllt’^camera)’ bright- y > ^up- V > ^forward- Y’ ^-^{j^up’^camera and its inverse k-1obtained through matrix inversion operations.

[0171] The depth reconstruction process may utilize the TransformDepth shader function that combines the snapped UV coordinates from step 306, the sampled depth values from step 414, and the calculated inverse matrices to produce accurate world-space positions. The transformation may involve constructing clip-space coordinates where the X component is set to 0.0 due to the single-pixel column width, the Y component is derived from the corrected UV coordinates using Y = (correctedYx. O) - 1.0, and the Z component is set to the exact depth value retrieved through point filtering. The clip-space coordinates may then be multiplied by the inverse projection matrix followed by the inverted view matrix to obtain the final world-space positions that represent the original three-dimensional geometry captured in the stereo panoramic image. In some cases, the buffer seeding may write the reconstructed depth values to enable proper occlusion relationships between the static panoramic content and subsequently rendered dynamic objects. The seeding process may transform the reconstructed world-space positions back into the current camera's clip space using the runtime projection and view matrices, where the transformed positions undergo perspective division to obtain normalized device coordinates suitable for depth buffer storage. The resulting Z-component may be written directly to the depth buffer at the corresponding pixel locations using the same depth buffer format queried from Unity's depth pass descriptor, typically D32_SFloat or D24_UNorm_S8_UInt on mobile virtual reality platforms.

[0172] The depth buffer seeding may respect the reversed Z-buffer convention where 1.0 represents the near plane and 0.0 represents the far plane, maintaining consistency with the depth encoding used during the panoramic capture process. The seeded depth values may be written as 32-bit TSZ-00325

[0173] floating point values, though the graphics hardware may automatically convert these values to match the currently bound depth buffer format. The seeding operation may utilize standard depth testing operations where closer objects overwrite farther objects according to the established depth comparison function, creating the foundation for proper occlusion testing between the static panoramic content and dynamic three-dimensional objects rendered in subsequent passes. The depth buffer seeding may target performance specifications of less than 1ms on Snapdragon XR2 processors, enabling efficient real-time rendering within mobile virtual reality hardware constraints while establishing depth relationships that allow standard rasterization techniques to interact naturally with the pre- captured environmental content.

[0174] Dynamic objects may be rendered with proper occlusion relationships (step 418). The dynamic object rendering may utilize standard rasterization techniques while benefiting from the depth information established in the step 416. The standard rasterization techniques may include vertex processing, primitive assembly, rasterization, fragment shading, and per-fragment operations that are commonly employed in traditional three-dimensional graphics pipelines. The graphics processing unit may execute these operations using the same projection and view parameters established during the panoramic content rendering, ensuring consistent coordinate systems and depth relationships throughout the hybrid rendering process.

[0175] The occlusion relationships may be established through standard depth testing operations, where fragments from dynamic objects are compared against the seeded depth values in the runtime depth buffer. Dynamic object fragments that are positioned behind static panoramic surfaces may be discarded during the depth test, while fragments positioned in front of the panoramic content may overwrite the corresponding depth buffer values and contribute to the final rendered image. The depth testing may respect the reversed Z-buffer convention where applicable, maintaining consistency with the depth format established during the panoramic depth buffer seeding process in step 416. TSZ-00325

[0176] In some cases, the occlusion relationships may enable dynamic three-dimensional objects to naturally interact with the stereo panoramic image content, providing correct depth ordering and visual integration between static and dynamic elements in the virtual reality scene. The depthbased occlusion may allow dynamic objects such as animated characters, moving vehicles, or interactive elements to appear behind static environmental features like walls, trees, or architectural structures captured in the panoramic imagery. Conversely, dynamic objects positioned closer to the viewer than the static panoramic surfaces may properly occlude portions of the background content, creating realistic spatial relationships that enhance the immersive quality of the virtual reality experience.

[0177] The dynamic object rendering may incorporate advanced lighting models, shadow casting, and material properties that interact with the static panoramic environment through the established depth relationships. Dynamic objects may cast shadows onto static panoramic surfaces through standard shadow mapping techniques, where the shadow calculations utilize the reconstructed world-space positions from the panoramic depth data. The lighting calculations may treat the panoramic content as if it were composed of traditional geometric surfaces, enabling dynamic objects to receive lighting contributions from the static environment and vice versa, creating cohesive illumination throughout the hybrid scene.

[0178] The performance characteristics of the dynamic object rendering may benefit from the reduced geometric complexity compared to traditional fully three-dimensional environments, as the static environmental details are represented through the panoramic imagery rather than geometric primitives. The computational savings from the hybrid approach may enable higher quality dynamic object rendering, including enhanced particle effects, more detailed geometric models, or sophisticated material shaders for the dynamic elements. The rendering system may achieve frame rate targets of 90fps on mobile virtual reality hardware by balancing the computational load between the efficient panoramic display established in previous steps and the focused dynamic object processing in this step. TSZ-00325

[0179] The integration of dynamic objects with panoramic content may support complex visual effects including reflections, refractions, and volumetric lighting that span both static and dynamic elements. Dynamic objects may interact with atmospheric effects or environmental conditions represented in the panoramic imagery, such as fog, rain, or lighting conditions that were captured during the original panoramic generation process. The depth-based occlusion system may enable these interactions to occur naturally through the graphics pipeline without requiring specialized handling for the boundary between panoramic and dynamic content.

[0180] The method 400 may enable efficient processing of stereo panoramic content through systematic separation of left and right eye views and appropriate coordinate transformations. In some cases, the method 400 may be implemented within virtual reality rendering systems that utilize the stereo handling capabilities to provide immersive virtual reality experiences with proper depth perception and occlusion relationships between panoramic and dynamic content.

[0181] Referring now to FIG. 5, a virtual reality rendering system 500 may be implemented to provide hybrid rendering capabilities that combine stereo panoramic imagery with dynamic three-dimensional objects through coordinated hardware and software components. The virtual reality rendering system 500 shown in FIG. 5 may implement both the first process for capturing panoramic imagery using standard rasterization pipelines and the second process that reconstructs world space positions of each pixel within the captured panorama to enable occlusion calculations with traditionally rasterized geometry. The virtual reality rendering system 500 may include a processing unit 502, a memory subsystem 508, a rendering pipeline 518, and mobile VR hardware 526 that work together to enable efficient virtual reality content rendering on mobile platforms. The system architecture may be specifically designed to address the computational limitations of mobile virtual reality hardware while maintaining high visual fidelity through the strategic combination of pre-captured panoramic content and real-time dynamic object rendering. TSZ-00325

[0182] The processing unit 502 may comprise a processor 504 and a graphics processing unit 506 that provide computational resources for the hybrid rendering operations. The processor 504 may be configured to execute instructions for general computational tasks and system coordination, including memory management, input processing, and application logic execution. The processor 504 may implement ARM-based architectures commonly found in mobile virtual reality platforms, providing sufficient computational power for system-level operations while maintaining power efficiency requirements for portable devices. In some cases, the graphics processing unit 506 may handle specialized graphics operations including rendering, depth reconstruction, and dynamic object processing through dedicated shader execution units and texture sampling hardware. The graphics processing unit 506 may utilize tile-based rendering architectures typical of mobile GPUs, where rendering operations are divided into screen-space tiles to optimize memory bandwidth and power consumption. The mobile VR hardware 526 may target PICO 4 ULTRA, a virtual reality headset, manufactured by ByteDance Ltd., of Singapore, Singapore, with SNAPDRAGON XR2 GEN2, a system-on-chip processor, manufactured by Qualcomm Incorporated, of San Diego, California, chipset, providing enhanced performance capabilities compared to previous generation mobile virtual reality platforms, including improved GPU performance, increased memory bandwidth, and enhanced power efficiency for extended virtual reality sessions.

[0183] As shown in FIG. 5, the memory subsystem 508 may include system memory 510 and stereo panoramic image storage 512 for storing various types of data needed during rendering operations. The system memory 510 may store general application data including shader programs, vertex buffers, uniform variables, and temporary rendering data, and may use coordinate system transformations including UnityToHephPoint and UnityToHephQuaternion functions for alignment between different coordinate systems used by the UNITY, a crossplatform game engine, developed by Unity Technologies, of San Francisco, California, rendering engine and the virtual reality hardware platform. The system memory 510 may be implemented TSZ-00325

[0184] using LPDDR5, a low power double data rate synchronous dynamic random-access memory standard, developed by JEDEC Solid State Technology Association, of Arlington, Virginia, or similar low-power memory technologies that provide high bandwidth while maintaining power efficiency suitable for mobile applications. The stereo panoramic image storage 512 may contain color data 514 and depth data 516 that represent static components of virtual environments captured using above-horizon and below-horizon projections, where the storage may be organized to enable efficient streaming and access patterns during real-time rendering operations. The color data 514 may use ASTC, an adaptive scalable texture compression format, developed by ARM Limited, of Cambridge, England, 4x4 block compression reducing bandwidth by approximately 75% with memory footprint of approximately 65MB for 8192x8192 stereo images, where the ASTC compression format provides variable bit rate encoding that can adapt to different texture complexity levels while maintaining visual quality. In some cases, the ASTC compression may enable efficient storage and transmission of high-resolution panoramic imagery while maintaining visual quality suitable for virtual reality applications, with compression ratios that can be adjusted based on the visual importance of different regions within the panoramic image. The ASTC format may support both low dynamic range and high dynamic range color encoding, enabling proper representation of lighting conditions captured during the panoramic image generation process. The depth data 516 may be stored as 32-bit floating point values using a reversed Z-buffer format where 1.0 may represent near plane and 0.0 may represent far plane, providing precise depth information for world-space position reconstruction with enhanced precision distribution across the depth range compared to traditional Z-buffer arrangements.

[0185] With continued reference to FIG. 5, the rendering pipeline 518 may comprise a full-screen quad renderer 520, a depth buffer manager 522, and a dynamic object renderer 524 that coordinate to perform the hybrid rendering process through a systematic sequence of rendering passes that combine panoramic content with dynamic elements. The full-screen quad Tenderer 520 may TSZ-00325

[0186] render the stereo panoramic image as a full-screen quad using equirectangular projection coordinates, transforming UV coordinates to spherical coordinates based on above-horizon and below-horizon regions through the EquirectangularCoords shader function that handles the mathematical conversion between planar texture coordinates and spherical panoramic coordinates. In some cases, the full-screen quad renderer 520 may utilize a fragment shader that samples both the color data 514 and the depth data 516 from the stereo panoramic image during a single full-screen quad pass, optimizing GPU utilization by combining color and depth processing operations within a unified shader execution context. The full-screen quad renderer 520 may implement stereo eye separation by converting texture coordinates from ranges of 0 to 0.5 or 0.5 to 1.0 to normalized 0 to 1.0 ranges based on the unity StereoEyelndex parameter, enabling proper access to left and right eye perspectives within the vertically stacked stereo panoramic texture format.

[0187] The depth buffer manager 522 may extract depth values from the depth data 516 and reconstruct original world-space positions by applying an inverse projection matrix and an inverted view matrix corresponding to capture parameters used during panoramic image generation, where the reconstruction process involves snapping UV coordinates to pixel centers, calculating original camera parameters based on column position and above-horizon or below-horizon designation, and constructing clip-space coordinates for transformation to world-space positions. The depth buffer manager 522 may seed a runtime depth buffer with the reconstructed world-space positions to enable depth-based occlusion relationships between the static panoramic content and subsequently rendered dynamic objects, creating the foundation for proper spatial integration within the hybrid rendering approach. In some cases, the depth buffer manager 522 may use UNITY, a cross-platform game engine, developed by Unity Technologies, of San Francisco, California, layer system with cullingMask to determine what geometry gets rendered during the depth reconstruction process, enabling selective processing of different object categories within the virtual environment. The depth buffer manager 522 may implement the TransformDepth TSZ-00325

[0188] shader function that combines snapped UV coordinates, sampled depth values, and calculated inverse matrices to produce accurate world-space positions suitable for depth buffer seeding operations.

[0189] As further shown in FIG. 5, the dynamic object renderer 524 may render dynamic three-dimensional objects using standard rasterization techniques, where the dynamic objects may interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships that enable realistic spatial integration between static and dynamic content elements. The dynamic object renderer 524 may utilize UNITY, a cross-platform game engine, developed by Unity Technologies, of San Francisco, California, standard rendering pipeline with vertex processing, primitive assembly, rasterization, fragment shading, and perfragment operations that benefit from the depth information established by the depth buffer manager 522. The rendering pipeline 518 may include video playback integration with hardware-accelerated decoding on ANDROID, a mobile operating system, developed by Google LLC, of Mountain View, California, for dynamic content, enabling efficient processing of video elements within the virtual reality environment through dedicated video decoding hardware present in mobile virtual reality platforms. In some cases, the dynamic three-dimensional objects may be constrained to move within predetermined regions of the virtual environment while a viewer position remains fixed to rotational movement, aligning with the three degrees of freedom limitation inherent in the panoramic capture approach while enabling focused computational resource allocation to areas requiring dynamic content.

[0190] The graphics processing unit 506 may coordinate the operations of the full-screen quad Tenderer 520, the depth buffer manager 522, and the dynamic object renderer 524 to achieve performance targets including less than 2ms for full-screen quad pass and less than 1ms for depth buffer seeding on SNAPDRAGON XR2, a system- on- chip processor, manufactured by Qualcomm Incorporated, of San Diego, California, enabling the system to maintain 90fps frame rates required for comfortable virtual reality experiences. The virtual reality rendering system 500 TSZ-00325

[0191] may support VULKAN, a low-level graphics application programming interface, developed by the Khronos Group, of Beaverton, Oregon, to enable efficient graphics operations on mobile virtual reality platforms through low-level graphics API access that provides enhanced control over GPU resources, reduced driver overhead, and improved multi-threading capabilities compared to higher-level graphics APIs. In some cases, the system may achieve bandwidth requirements of approximately 3 GB / s read bandwidth at 90fps for 33MB panorama textures, operating within the memory bandwidth capabilities of mobile virtual reality hardware while providing high-resolution panoramic content display.

[0192] The mobile VR hardware 526 may interface with the graphics processing unit 506 and may include a display interface 530 for presenting the rendered content to users through high-resolution OLED, organic light-emitting diode displays, manufactured by various companies including Samsung Display Co., Ltd., of Yongin, South Korea, or LCD, liquid crystal displays, manufactured by various companies including LG Display Co., Ltd., of Seoul, South Korea, displays with refresh rates of 90Hz or higher to minimize motion blur and provide smooth visual experiences. The display interface 530 may receive the final rendered frames from the dynamic object renderer 524 and present the combined panoramic and dynamic content through the virtual reality headset display system, implementing lens distortion correction, chromatic aberration compensation, and other optical corrections required for proper virtual reality presentation. In some cases, transparent objects may be rendered back-to-front after the stereo panoramic image and dynamic three-dimensional objects using standard alpha blending operations, while transparency may not be supported within the stereo panoramic image itself due to the nature of the panoramic capture and storage process that requires opaque surface representation for accurate depth reconstruction. The display interface 530 may support various display resolutions including 1832x1920 per eye for QUEST 2, a virtual reality headset, manufactured by Meta Platforms, Inc., of Menlo Park, California, class devices or higher TSZ-00325

[0193] resolutions for premium virtual reality platforms, with the panoramic content resolution scaled appropriately to match or exceed the display capabilities.

[0194] The virtual reality rendering system 500 may enable efficient hybrid rendering by leveraging the specialized capabilities of each component within the processing unit 502, memory subsystem 508, and rendering pipeline 518 through coordinated data flow and processing operations that optimize both computational efficiency and visual quality. The coordinated operation of these components may provide high-quality virtual reality experiences while operating within the computational and bandwidth constraints of mobile virtual reality hardware platforms, achieving the balance between visual fidelity and performance necessary for practical virtual reality applications on portable devices. The system architecture may support scalable performance characteristics where rendering quality and computational load can be adjusted based on the capabilities of the target hardware platform, enabling deployment across a range of mobile virtual reality devices with varying performance specifications.

[0195] Referring to FIG. 6, a stereo panoramic capture system 600 may be implemented to generate the stereo panoramic images used by the virtual reality rendering system 500. The stereo panoramic capture system 600 may implement the process that utilizes standard rasterization pipelines to capture high-resolution panoramic imagery with associated depth information through coordinated camera control, projection processing, image generation, and data storage operations. The captured panoramic data from this process may subsequently be processed by the virtual reality rendering system 500 to reconstruct world space positions of each pixel and enable occlusion relationships with dynamic three-dimensional content. The system architecture may be designed to implement the column-wise capture methodology described in method 200, where each vertical strip of the panoramic image is rendered systematically through rotation of the camera about the vertical axis while maintaining fixed spatial positioning of the capture origin. The stereo panoramic capture system 600 may include a camera control module 602 that manages the positioning and orientation of capture equipment during the panoramic image TSZ-00325

[0196] generation process. The camera control module 602 may comprise a position controller 604 and an orientation controller 606 that work together to ensure precise camera placement and directional control throughout the capture sequence. In some cases, the position controller 604 may maintain the camera at a fixed spatial location while enabling rotational movement around the capture point to generate the complete 360-degree panoramic coverage. The position controller 604 may implement high-precision positioning mechanisms including stepper motors, servo systems, or robotic arms that provide accurate spatial control with positioning accuracy of 0.1mm or better to ensure consistent capture geometry across all column positions. The position controller 604 may utilize encoder feedback systems that verify the camera position matches the desired capture location before initiating each column rendering operation, preventing positioning errors that could introduce geometric inconsistencies in the final panoramic image. In some cases, the position controller 604 may implement vibration damping mechanisms to minimize camera shake during the capture process, utilizing passive damping materials, active stabilization systems, or weighted mounting platforms that reduce high-frequency vibrations that could blur the captured imagery. The position controller 604 may coordinate with external positioning systems including optical tracking, laser positioning, or mechanical alignment fixtures that provide reference points for verifying the camera's spatial location throughout the capture sequence.

[0197] As shown in FIG. 6, the orientation controller 606 may receive control signals from the position controller 604 and may manage the camera's directional positioning for both above-horizon and below-horizon capture operations. The orientation controller 606 may coordinate the camera rotation in discrete steps corresponding to the target image width, where each step may represent 360 degrees divided by the desired panoramic resolution. In some cases, the orientation controller 606 may implement the discrete rotation steps of 0.044 degrees per column for 8192-pixel wide panoramic images, ensuring systematic coverage of the complete horizontal field of view. The orientation controller 606 may utilize precision rotation stages including motorized TSZ-00325

[0198] pan-tilt heads, gimbal systems, or multi-axis rotation platforms that provide angular positioning accuracy of 0.01 degrees or better, ensuring that each column is captured at the exact azimuthal angle required for seamless panoramic assembly. The rotation mechanisms may incorporate angular encoders, resolvers, or optical angle sensors that provide real-time feedback on the camera's orientation, enabling closed-loop control systems that compensate for mechanical backlash, thermal drift, or other factors that could introduce angular positioning errors. In some cases, the orientation controller 606 may implement acceleration and deceleration profiles for the rotation movements, utilizing smooth motion curves that minimize mechanical stress on the camera mounting hardware while reducing settling time between successive column captures. The orientation controller 606 may coordinate the timing of rotation movements with the image capture operations, ensuring that the camera has reached the desired angular position and any residual vibrations have dissipated before initiating the rendering process for each column. The orientation controller 606 may support variable rotation increment modes where the angular step size can be adjusted dynamically based on the target panoramic resolution, enabling capture of panoramic images at resolutions ranging from 1024 pixels to 16384 pixels or higher by modifying the rotation increment from 0.352 degrees down to 0.022 degrees or smaller.

[0199] The stereo panoramic capture system 600 may include a projection system 608 that handles the specialized projection techniques needed for panoramic image capture. The projection system 608 may comprise an above-horizon projector 610 and a below-horizon projector 612 that implement the dual-projection approach for capturing the complete vertical field of view. The above-horizon projector 610 may capture imagery with the camera pointed 45 degrees above the horizon, covering the upper 90 degrees of the vertical field of view. The below-horizon projector 612 may capture imagery with the camera pointed 45 degrees below the horizon, covering the lower 90 degrees of the vertical field of view. The dual-projection approach may be derived from the omnidirectional stereo rendering methodology described in the Google VR Jump paper, which establishes the mathematical foundation for capturing spherical panoramic content using TSZ-00325

[0200] perspective projections with tilted camera orientations. The above-horizon projector 610 may 512.0 0.0 1024.0

[0201] implement the intrinsic camera matrix Kabove= 0.0 512.0 1024.0 with focal length

[0202]

[0203] 0.0 0.0

[0204]

[0205] components of 512.0 pixels corresponding to the 90-degree field of view coverage, and extrinsic 1.0 0.0 0.0

[0206] rotation matrix Rabove = 0.0 0.707 -0.707 to achieve the upward tilt orientation. The below- 0.0 0.707 0.707

[0207] horizon projector 612 may implement the intrinsic camera matrix Kbelow=

[0208] 512.0 0.0 1024.0

[0209] 0.0 -512.0 1024.0 with inverted Y-axis focal length to account for the downward camera. 0.0 0.0 1.0

[0210]

[0211] F1.0

[0212]

[0213] orientation, and extrinsic rotation matrix Rbelow= 0.0 0.707 0.707 for the downward tilt LO. O -0.707 0.707

[0214] configuration. The projection system 608 may utilize standard pinhole camera models without lens distortion correction, simplifying the coordinate transformation calculations during both capture and runtime rendering operations while maintaining geometric accuracy sufficient for virtual reality applications.

[0215] With continued reference to FIG. 6, the orientation controller 606 may provide control signals to both the above-horizon projector 610 and the below-horizon projector 612, enabling coordinated switching between the two projection modes during the column-by-column capture process. In some cases, the projection system 608 may utilize standard pinhole camera models with standard rasterization camera parameters, maintaining consistent projection matrices across all capture operations while varying the camera orientation between above-horizon and below-horizon configurations. The projection system 608 may implement near and far plane values that are user-configurable, enabling adjustment of the depth capture range based on the specific characteristics of the virtual environment being captured, where typical near plane values may range from 0.1 meters to 1.0 meters and far plane values may range from 100 meters to 10000 meters depending on the scale of the environment. The projection system 608 may utilize nonlinear depth encoding with reversed Z-buffer format, where depth values are stored as 1.0 for the near plane and 0.0 for the far plane, providing enhanced depth precision distribution across the TSZ-00325

[0216] capture range compared to traditional linear depth encoding. The above-horizon projector 610 and below-horizon projector 612 may share common projection parameters including field of view, aspect ratio, and depth encoding format, ensuring consistency in the captured depth information across the horizon boundary where the two projection regions meet. In some cases, the projection system 608 may implement depth buffer capture alongside color capture, where both RGB texture data and depth buffer values are preserved for every pixel in each column, providing the complete information needed for depth-based occlusion during runtime rendering operations.

[0217] The stereo panoramic capture system 600 may include an image processing unit 614 that processes the captured imagery from the projection system 608 into the final stereo panoramic format. The image processing unit 614 may comprise a column renderer 616 and a stereo generator 618 that handle the systematic processing of captured image data. The column renderer 616 may receive input from both the above-horizon projector 610 and the below-horizon projector 612, combining the upper and lower portions of each column into complete vertical strips of the panoramic image. The column renderer 616 may implement pixel-perfect alignment operations that ensure seamless integration between the above-horizon and below-horizon captures at the horizon boundary, utilizing coordinate snapping techniques similar to the SnapToGrid function employed during runtime depth reconstruction to maintain geometric consistency. The column renderer 616 may process the captured image data using graphics processing unit acceleration, leveraging shader programs and parallel processing capabilities to efficiently handle the high-resolution image data generated during the panoramic capture process. In some cases, the column renderer 616 may implement quality validation checks that analyze each captured column for artifacts including motion blur, exposure inconsistencies, or geometric distortions, flagging problematic columns for recapture before proceeding with subsequent rotation steps. TSZ-00325

[0218] As further shown in FIG. 6, the column renderer 616 may process the captured image data in 1 2048 pixel column chunks, where each chunk may represent a complete vertical slice of the panoramic environment. The column renderer 616 may combine the above- horizon and below-horizon captures for each column position, creating seamless vertical strips that span the full 180-degree vertical field of view. In some cases, the column renderer 616 may apply appropriate coordinate transformations and pixel alignment operations to ensure proper integration between the upper and lower portions of each column. The column rendering process may involve concatenating the 1x1024 pixel above-horizon strip with the 1x1024 pixel below-horizon strip to form the complete 1 x2048 pixel column, where the vertical concatenation preserves the spatial relationships captured during the dual-projection process. The column renderer 616 may implement boundary smoothing operations at the horizon interface, utilizing blending techniques or feathering algorithms that minimize visible seams between the above-horizon and below-horizon regions while maintaining the geometric accuracy of the depth information. In some cases, the column renderer 616 may apply color correction operations that normalize exposure, white balance, or color temperature across the above-horizon and below-horizon captures, ensuring consistent appearance throughout the vertical field of view despite potential lighting variations between the upper and lower projection regions. The column renderer 616 may accumulate the processed column strips into a growing panoramic image buffer, systematically building the complete 360-degree panoramic representation through sequential addition of each column as the camera rotates through the full azimuthal range. The column rendering operations may be performed in real-time during the capture process or in post-processing after all columns have been captured, depending on the computational resources available and the desired workflow for the panoramic image generation system.

[0219] The stereo generator 618 may contain a left eye processor 620 and a right eye processor 622 that generate separate perspective views for stereo panoramic imaging. The left eye processor 620 and the right eye processor 622 may receive processed column data from the column renderer TSZ-00325

[0220] 616 and may apply inter-pupillary distance baseline adjustments to create the left and right eye perspectives. In some cases, the stereo generator 618 may move the camera for the right eye along the camera's right vector by half the inter-pupillary distance, while moving the camera for the left eye along the camera's left vector by half the inter-pupillary distance. The inter-pupillary distance baseline may be set to 64mm, representing the average human inter-pupillary distance and providing comfortable stereo viewing characteristics for most users, though the baseline value may be configurable to accommodate different user populations or to create exaggerated or compressed stereo effects for specific applications. The left eye processor 620 may calculate the left eye camera position using the formula Pleft= Pcenter ~base*inex Rright(9coi) Pcenter, where ^center represents the central capture position, baseline equals 0.064 meters, and Rright(®cod = [sin (9coi) is the normalized right vector for the current azimuth angle. The right eye processor 622 may calculate the right eye camera position using the formula Pright= Pcenter +

[0221] baselinexRright(9colmaintaining the symmetric baseline offset that creates the stereo disparity needed for depth perception. The stereo generator 618 may implement vergence corrections that adjust the camera forward vectors to ensure proper stereo convergence at a target depth, typically set to 2.0 meters for optimal viewing comfort, where the vergence angle may be calculated as “vergence=arctan(0.032 / tar get_depth) and applied to the left and right eye camera orientations. The vergence adjustments may modify the camera forward vectors to Fleft= normalize{ cos 9coi+avergence / 2), sin(tilt_angle)Fright=

[0222] normalize ([cos(0coi— avergence / 2>), sin(tilt_angle') and Fright=

[0223] n

[0224]

[0225] ormalize ([cos(0coi— avergence / £')> sm(tilt_angle), where tilt_angle equals ±?r / 4 for above / below-horizon projections respectively, ensuring that the left and right eye views converge at the target depth while maintaining identical vertical orientations to prevent vertical disparity artifacts. The stereo generator 618 may process both left and right eye perspectives simultaneously for each column position, generating stereo pairs that maintain temporal synchronization and geometric consistency throughout the panoramic capture sequence. TSZ-00325

[0226] In an alternative embodiment, the stereo generator 618 may operate without vergence corrections, implementing a parallel stereo configuration where both left and right eye cameras maintain identical forward orientations. In this configuration, the left eye processor 620 and right eye processor 622 may calculate forward vectors as Fleft= norma / ize([cos(0co(), sm(tilt_angle') and Fright= normalize [cos 9coi'),sm(tilt_angle'), where both eyes look straight ahead at each azimuth position while being separated by the 64mm baseline distance. This approach may simplify the stereo generation process by eliminating vergence calculations while relying on the natural stereo disparity created by the baseline separation to provide depth perception cues. The parallel stereo configuration may reduce computational overhead during the panoramic capture process while maintaining effective stereo imaging capabilities for virtual reality applications. The stereo panoramic capture system 600 may include a data storage system 624 that preserves the captured and processed panoramic data for subsequent use in virtual reality rendering operations. The data storage system 624 may comprise color texture storage 626, depth buffer storage 628, and projection matrix storage 630 that store different types of information generated during the capture process. The color texture storage 626 may receive color information from both the left eye processor 620 and the right eye processor 622, storing the RGB texture data for each eye's perspective in the vertically stacked format used by the virtual reality rendering system 500. The vertically stacked storage format may place the left eye view in the upper half of the texture corresponding to vertical texture coordinates from 0.0 to 0.5, and the right eye view in the lower half corresponding to vertical texture coordinates from 0.5 to 1.0, enabling efficient stereo processing during runtime rendering operations through simple coordinate range conversions based on the unity\_StereoEyeIndex parameter. The color texture storage 626 may implement ASTC 4x4 block compression for the RGB texture data, reducing the memory footprint from approximately 268MB uncompressed to approximately 65MB compressed for 8192x8192 pixel stereo panoramic images, achieving approximately 75% bandwidth reduction while maintaining visual quality suitable for virtual reality applications. The ASTC compression TSZ-00325

[0227] may be applied as a post-processing operation after the complete panoramic image has been assembled, utilizing compression quality settings that balance file size against visual fidelity based on the specific requirements of the target virtual reality application. In some cases, the color texture storage 626 may support high dynamic range color encoding, storing RGB values in formats including RGB16F or RGB32F that preserve extended luminance ranges captured during the panoramic image generation process, enabling proper representation of bright highlights, deep shadows, and wide color gamuts in the virtual environment.

[0228] With continued reference to FIG. 6, the depth buffer storage 628 may receive depth information from the stereo generator 618, preserving the distance measurements for each pixel in the panoramic image. The depth buffer storage 628 may store the depth values as 32-bit floating point values using the reversed Z-buffer format, maintaining the precision needed for accurate world-space position reconstruction during runtime rendering operations. In some cases, the depth buffer storage 628 may store depth information for both left and right eye perspectives, enabling proper stereo depth reconstruction in the virtual reality rendering system 500. The depth buffer storage 628 may maintain the depth values in uncompressed format at approximately 268MB for full 32-bit floating point precision at 8192x8192 pixel resolution, prioritizing numerical accuracy over storage efficiency to ensure that the depth reconstruction operations performed during runtime rendering can accurately recover the original three-dimensional geometry captured during the panoramic image generation process. The reversed Z-buffer format may store depth values where 1.0 represents the near plane and 0.0 represents the far plane, providing enhanced depth precision distribution across the depth range by leveraging the non-uniform distribution of floating point precision, where more precision bits are available near 1.0 than near 0.0, resulting in better depth resolution for nearby objects that are more critical for accurate occlusion relationships. The depth buffer storage 628 may implement validation checks that verify the depth values fall within the expected range defined by the near and far plane parameters used during capture, flagging any anomalous depth values that could indicate capture TSZ-00325

[0229] errors or geometric inconsistencies requiring correction. In some cases, the depth buffer storage 628 may store additional metadata including the near plane distance, far plane distance, and depth encoding format used during capture, providing the information needed by the virtual reality rendering system 500 to properly interpret and reconstruct world-space positions from the stored depth values.

[0230] The projection matrix storage 630 may receive and store the projection parameters used during the capture process from the stereo generator 618. The projection matrix storage 630 may preserve the inverse projection matrix and other capture parameters that may be needed during the depth reconstruction operations performed by the virtual reality rendering system 500. In some cases, the projection matrix storage 630 may store matrices in row-major format following UNITY, a cross-platform game engine, developed by Unity Technologies, of San Francisco, California, and DIRECT3D, a graphics application programming interface, developed by Microsoft Corporation, of Redmond, Washington, convention, ensuring compatibility with the matrix operations performed during runtime depth reconstruction and world-space position calculation. The projection matrix storage 630 may store the inverse projection matrices K_above-1= [1 / 512.0, 0.0, -2.0; 0.0, 1 / 512.0, -2.0; 0.0, 0.0, 1.0] for above-horizon samples and K_below-1= [1 / 512.0, 0.0, -2.0; 0.0, -1 / 512.0, 2.0; 0.0, 0.0, 1.0] for below-horizon samples, where these pre-computed inverse matrices enable efficient depth reconstruction during runtime rendering without requiring matrix inversion operations in the fragment shader. The projection matrix storage 630 may also preserve the original forward projection matrices, near and far plane values, field of view parameters, and other camera intrinsic properties that characterize the capture configuration, providing comprehensive documentation of the capture parameters that may be useful for debugging, validation, or alternative rendering approaches. In some cases, the projection matrix storage 630 may store per-column camera parameters including the azimuthal rotation angle, stereo baseline offset, and vergence correction for each column position, enabling advanced rendering techniques that account for the column-specific TSZ-00325

[0231] capture geometry during runtime depth reconstruction operations. The projection matrix storage 630 may implement data formats that are directly compatible with GPU uniform buffer objects or shader storage buffer objects, enabling efficient transfer of the projection parameters to the graphics processing unit during runtime rendering operations without requiring format conversions or data restructuring.

[0232] The stereo panoramic capture system 600 may enable systematic generation of high-quality stereo panoramic content through the coordinated operation of the camera control module 602, projection system 608, image processing unit 614, and data storage system 624. The captured stereo panoramic images may provide the static environmental content that forms the foundation for the hybrid rendering approach implemented by the virtual reality rendering system 500, enabling efficient display of complex virtual environments on mobile virtual reality hardware platforms. The system architecture may support various capture workflows including real-time capture where the panoramic image is assembled during the rotation sequence, offline capture where all columns are captured first and then processed in post-production, or hybrid workflows that combine real-time preview with offline high-quality processing. The stereo panoramic capture system 600 may be implemented using various hardware configurations including physical camera rigs with motorized rotation stages for capturing real-world environments, or virtual camera systems within three-dimensional rendering engines for capturing computergenerated environments, where the virtual camera approach may be particularly suitable for generating panoramic content from game engines, architectural visualization tools, or other three-dimensional content creation platforms. The capture system may support batch processing modes where multiple panoramic images are captured sequentially with different camera positions, lighting conditions, or environmental configurations, enabling creation of panoramic image libraries for virtual reality applications. The stereo panoramic capture system 600 may integrate with the virtual reality rendering system 500 through standardized file formats and data interfaces, where the color texture storage 626, depth buffer storage 628, and projection matrix TSZ-00325

[0233] storage 630 output data in formats that can be directly loaded into the stereo panoramic image storage 512 within the memory subsystem 508 of the rendering system, enabling seamless workflow integration between the capture and rendering stages of the hybrid virtual reality content pipeline.

[0234] Various techniques, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, a non-transitory computer readable storage medium, or any other machine-readable storage medium wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the various techniques. In the case of program code execution on programmable computers, the computing device may include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The volatile and non-volatile memory and / or storage elements may be a RAM, an EPROM, a flash drive, an optical drive, a magnetic hard drive, or another medium for storing electronic data. The eNB (or other base station) and UE (or other mobile station) may also include a transceiver component, a counter component, a processing component, and / or a clock component or timer component. One or more programs that may implement or utilize the various techniques described herein may use an application programming interface (API), reusable controls, and the like. Such programs may be implemented in a high-level procedural or an object-oriented programming language to communicate with a computer system. However, the program(s) may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or an interpreted language, and combined with hardware implementations.

[0235] It should be understood that many of the functional units described in this specification may be implemented as one or more components, which is a term used to more particularly emphasize their implementation independence. For example, a component may be implemented as a hardware circuit comprising custom very large scale integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A TSZ-00325

[0236] component may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like. Components may also be implemented in software for execution by various types of processors. An identified component of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions, which may, for instance, be organized as an object, a procedure, or a function. Nevertheless, the executables of an identified component need not be physically located together, but may comprise disparate instructions stored in different locations that, when joined logically together, comprise the component and achieve the stated purpose for the component.

[0237] Indeed, a component of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within components, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, and may exist, at least partially, merely as electronic signals on a system or network. The components may be passive or active, including agents operable to perform desired functions.

[0238] Reference throughout this specification to "an example" means that a particular feature, structure, or characteristic described in connection with the example is included in at least one embodiment of the present invention. Thus, appearances of the phrase "in an example" in various places throughout this specification are not necessarily all referring to the same embodiment. As used herein, a plurality of items, structural elements, compositional elements, and / or materials may be presented in a common list for convenience. However, these lists should be construed as though each member of the list is individually identified as a separate and unique member. Thus, no individual member of such list should be construed as a de facto equivalent of any other member of the same list solely based on its presentation in a common group without indications to the contrary. In addition, various embodiments and examples of the present invention may be referred to herein along with alternatives for the various components thereof. It TSZ-00325

[0239] is understood that such embodiments, examples, and alternatives are not to be construed as de facto equivalents of one another, but are to be considered as separate and autonomous representations of the present invention.

[0240] Although the foregoing has been described in some detail for purposes of clarity, it will be apparent that certain changes and modifications may be made without departing from the principles thereof. It should be noted that there are many alternative ways of implementing both the processes and apparatuses described herein. Accordingly, the present embodiments are to be considered illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims. Those having skill in the art will appreciate that many changes may be made to the details of the above-described embodiments without departing from the underlying principles of the invention. The scope of the present invention should, therefore, be determined only by the following claims.

Claims

TSZ-00325CLAIMS1. A virtual reality rendering system, comprising:a processor configured to execute instructions;a memory storing a stereo panoramic image comprising color data and depth data captured using an above- horizon projection and a below-horizon projection, wherein the stereo panoramic image represents static components of a virtual environment; anda graphics processing unit configured to:render the stereo panoramic image as a full-screen quad using equirectangular projection coordinates;extract depth values from the depth data and reconstruct original world-space positions by applying an inverse projection matrix and an inverted view matrix corresponding to capture parameters used during panoramic image generation;seed a runtime depth buffer with the reconstructed world-space positions to enable depth-based occlusion; andrender dynamic three-dimensional objects using standard rasterization techniques, wherein the dynamic objects interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships.

2. The virtual reality rendering system of claim 1, wherein the above-horizon projection captures imagery with a camera pointed 45 degrees above a horizon and the below-horizon projection captures imagery with a camera pointed 45 degrees below the horizon.

3. The virtual reality rendering system of claim 2, wherein the stereo panoramic image is captured by rotating the camera in discrete steps of 360 degrees divided by a target image width to generate column-wise image strips.TSZ-003254. The virtual reality rendering system of claim 1, wherein the stereo panoramic image comprises a left eye view and a right eye view captured with an inter-pupillary distance baseline, and the left eye view and right eye view are vertically stacked in a single texture.

5. The virtual reality rendering system of claim 4, wherein the graphics processing unit is configured to separate the left eye view and right eye view by converting texture coordinates from a range of 0 to 0.5 or 0.5 to 1.0 to a range of 0 to 1.0 based on a stereo eye index.

6. The virtual reality rendering system of claim 1, wherein reconstructing the original world-space positions comprises:snapping UV coordinates to pixel centers to match exact capture parameters; calculating original camera position and orientation based on column position and abovehorizon or below-horizon designation;constructing clip-space coordinates with X set to 0.0, Y derived from UV coordinates, and Z set to the extracted depth value; andapplying the inverse projection matrix and inverted view matrix to transform the clipspace coordinates to world-space positions.

7. The virtual reality rendering system of claim 6, wherein snapping UV coordinates to pixel centers comprises multiplying UV coordinates by pixel dimensions, taking a floor operation, adding 0.5, and dividing by the pixel dimensions.

8. The virtual reality rendering system of claim 1, wherein the depth data is stored as 32-bit floating point values using a reversed Z-buffer format where 1.0 represents near plane and 0.0 represents far plane.

9. The virtual reality rendering system of claim 1, wherein the graphics processing unit is configured to render the stereo panoramic image using a fragment shader that samplesTSZ-00325both color data and depth data from the stereo panoramic image during a single full-screen quad pass.

10. The virtual reality rendering system of claim 9, wherein the fragment shader performs equirectangular coordinate transformation using a function that converts UV coordinates to spherical coordinates based on above-horizon and below-horizon regions.

11. The virtual reality rendering system of claim 1, wherein the virtual reality rendering system is implemented on a mobile virtual reality headset comprising a SNAPDRAGON XR2 processor and supports VULKAN graphics API.

12. The virtual reality rendering system of claim 11, wherein the stereo panoramic image has a resolution of 8192×8192 pixels and uses ASTC 4×4 block compression for color data to reduce memory bandwidth requirements.

13. The virtual reality rendering system of claim 1, wherein the dynamic three-dimensional objects are constrained to move only within predetermined regions of the virtual environment while a viewer position remains fixed to rotational movement only.

14. The virtual reality rendering system of claim 13, wherein transparent objects are rendered back-to-front after the stereo panoramic image and dynamic three-dimensional objects, and transparency is not supported within the stereo panoramic image itself.

15. A method for hybrid rendering of virtual reality content, comprising:storing a stereo panoramic image comprising color data and depth data, wherein the stereo panoramic image represents static components of a virtual environment and is captured using an above-horizon projection and a below-horizon projection;rendering the stereo panoramic image as a full-screen quad using equirectangular projection coordinates;extracting depth values from the depth data;TSZ-00325reconstructing original world-space positions from the extracted depth values by applying an inverse projection matrix and an inverted view matrix corresponding to capture parameters used during panoramic image generation;seeding a runtime depth buffer with the reconstructed world-space positions; and rendering dynamic three-dimensional objects using standard rasterization techniques, wherein the dynamic objects interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships.

16. The method of claim 15, wherein the above-horizon projection captures imagery with a camera pointed 45 degrees above a horizon and the below-horizon projection captures imagery with a camera pointed 45 degrees below the horizon.

17. The method of claim 16, wherein the stereo panoramic image is captured by rotating the camera in discrete steps of 360 degrees divided by a target image width to generate column-wise image strips.

18. The method of claim 15, wherein reconstructing the original world-space positions comprises:snapping UV coordinates to pixel centers to match exact capture parameters; calculating original camera position and orientation based on column position and abovehorizon or below-horizon designation;constructing clip-space coordinates with X set to 0.0, Y derived from UV coordinates, and Z set to the extracted depth value; andapplying the inverse projection matrix and inverted view matrix to transform the clipspace coordinates to world-space positions.TSZ-0032519. The method of claim 18, wherein snapping UV coordinates to pixel centers comprises multiplying UV coordinates by pixel dimensions, taking a floor operation, adding 0.5, and dividing by the pixel dimensions.

20. The method of claim 15, wherein the stereo panoramic image comprises a left eye view and a right eye view captured with an inter-pupillary distance baseline, and the method further comprises separating the left eye view and right eye view by converting texture coordinates from a range of 0 to 0.5 or 0.5 to 1.0 to a range of 0 to 1.0 based on a stereo eye index.

21. The method of claim 15, wherein the depth data is stored as 32-bit floating point values using a reversed Z-buffer format where 1.0 represents near plane and 0.0 represents far plane, and the dynamic three-dimensional objects are constrained to move only within predetermined regions of the virtual environment while a viewer position remains fixed to rotational movement only.

22. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:access a stereo panoramic image comprising color data and depth data, wherein the stereo panoramic image represents static components of a virtual environment and is captured using an above-horizon projection and a below-horizon projection;render the stereo panoramic image as a full-screen quad using equirectangular projection coordinates;extract depth values from the depth data;reconstruct original world-space positions from the extracted depth values by applying an inverse projection matrix and an inverted view matrix corresponding to capture parameters used during panoramic image generation;seed a runtime depth buffer with the reconstructed world-space positions; andTSZ-00325render dynamic three-dimensional objects using standard rasterization techniques, wherein the dynamic objects interact with the stereo panoramic image through the seeded depth buffer to provide correct occlusion relationships.

23. The non-transitory computer-readable medium of claim 22, wherein the abovehorizon projection captures imagery with a camera pointed 45 degrees above a horizon and the below-horizon projection captures imagery with a camera pointed 45 degrees below the horizon.

24. The non-transitory computer-readable medium of claim 23, wherein the stereo panoramic image is captured by rotating the camera in discrete steps of 360 degrees divided by a target image width to generate column-wise image strips.

25. The non-transitory computer-readable medium of claim 24, wherein reconstructing the original world-space positions comprises:snapping UV coordinates to pixel centers by multiplying UV coordinates by pixel dimensions, taking a floor operation, adding 0.5, and dividing by the pixel dimensions;calculating original camera position and orientation based on column position and abovehorizon or below-horizon designation;constructing clip-space coordinates with X set to 0.0, Y derived from UV coordinates, and Z set to the extracted depth value; andapplying the inverse projection matrix and inverted view matrix to transform the clipspace coordinates to world-space positions.

Images