Generating a three-dimensional texture synthesis using generative artificial intelligence

Abstract

A method includes generating a three-dimensional (3D) mesh of the virtual environment from a set of images of physical terrain and depth information. The method further includes applying an initial texture to the 3D mesh based on the depth information. The method further includes generating a projection of a 360-degree panorama onto the 3D mesh. The method further includes identifying one or more areas in the projection that are missing the initial texture. The method further includes providing the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information to a machine-learning model. The method further includes outputting output texture for the one or more areas in the projection that are missing the initial texture and blending the output texture with the initial texture in the projection to obtain a blended texture in the projection.

Description

Attorney Docket No.: LE-3049-01-WOGENERATING A THREE-DIMENSIONAL TEXTURE SYNTHESIS USING GENERATIVE ARTIFICIAL INTELLIGENCEBACKGROUND

[0001] A three-dimensional (3D) virtual environment may be generated from images of a corresponding real-world scene. The images may be obtained from different angles and / or using different types of cameras and at different times of day, month, or year. If the real- world scene is outdoor terrain, capturing the images from different areas at different times leads to inconsistent lighting and textures.

[0002] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.SUMMARY

[0003] A method includes generating a three-dimensional (3D) mesh of a virtual environment from a set of images of physical terrain and depth information. The method further includes applying an initial texture to the 3D mesh based on the depth information. The method further includes generating a projection of a 360-degree panorama onto the 3D mesh, where the projection is generated from a perspective of a user in the virtual environment. The method further includes identifying one or more areas in the projection that are missing the initial texture based on the set of images having one or more portions that are occluded by one or more objects. The method further includes providing the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information to a machine-learning model. The method further includes outputting, with the machine- learning model, output texture for the one or more areas in the projection that are missing the initial texture. The method further includes blending the output texture with the initial texture in the projection to obtain a blended texture in the projection.

[0004] In some embodiments, the method further includes determining a field of view of the user based on a range of motion of the user; determining a subset of the one or more areas in the projection that are not visible to the user at different angles in the range of motion; and excluding the subset of the one or more areas from the identification of the one or more areasAttorney Docket No.: LE-3049-01-WO that are provided to the machine-learning model. In some embodiments, the method further includes determining a parallax correction for the range of motion by calculating a maximum horizontal angle and a maximum vertical angle of a triangle from a particular position in 3D space; and responsive to a degree of parallax being less than a predetermined number of degrees, replacing a portion of the 3D mesh that is beyond the predetermined number of degrees with a flat surface. In some embodiments, the method further includes determining, based on at least one factor, a range of visibility of the user, wherein the at least one factor is selected from a group of a vision capability of the user and prescription lenses used in a headmounted display worn by the user; and replacing a portion of the 3D mesh that is beyond the range of visibility with a flat surface.

[0005] In some embodiments, the set of images are captured at different times of day and generating the projection of the 360-degree panorama includes averaging the set of images. In some embodiments, the set of images are captured from one or more sources selected from a group of a digital Single-Lens Reflex (dSLR) camera, a 360-degree camera, a drone, and combinations thereof. In some embodiments, the machine-learning model is an in-painter model and the identification of the one or more areas in the projection is a mask. In some embodiments, the method further includes adding shadows to the blended texture in the projection. In some embodiments, the method further includes transmitting the projection with the blended texture to a head-mounted display associated with the user.

[0006] A computing device includes one or more processors; and one or more memories in communication with the one or more processors, with instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include generating a 3D mesh of a virtual environment from a set of images of physical terrain and depth information; applying an initial texture to the 3D mesh based on the depth information; generating a projection of a 360-degree panorama onto the 3D mesh, wherein the projection is generated from a perspective of a user in the virtual environment; identifying one or more areas in the projection that are missing the initial texture based on the set of images having one or more portions that are occluded by one or more objects; providing the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information to a machine-learning model; outputting, with the machine-learning model, output texture for the one or more areas in the projection that are missing the initial texture; and blending the output texture with the initial texture in the projection to obtain a blended texture in the projection.Attorney Docket No.: LE-3049-01-WO

[0007] In some embodiments, the operations further include determining a field of view of the user based on a range of motion of the user; determining a subset of the one or more areas in the projection that are not visible to the user at different angles in the range of motion; and excluding the subset of the one or more areas from the identification of the one or more areas that are provided to the machine-learning model. In some embodiments, the operations further include determining a parallax correction for the range of motion by calculating a maximum horizontal angle and a maximum vertical angle of a triangle from a particular position in 3D space; and responsive to a degree of parallax being less than a predetermined number of degrees, replacing a portion of the 3D mesh that is beyond the predetermined number of degrees with a flat surface. In some embodiments, the operations further include determining, based on at least one factor, a range of visibility of the user, wherein the at least one factor is selected from a group of a vision capability of the user and prescription lenses used in a head-mounted display worn by the user; and replacing a portion of the 3D mesh that is beyond the range of visibility with a flat surface. In some embodiments, the set of images are captured at different times of day and where generating the projection of the 360-degree panorama includes averaging the set of images. In some embodiments, the set of images are captured from one or more sources selected from a group of a digital Single- Lens Reflex (dSLR) camera, a 360-degree camera, a drone, and combinations thereof.

[0008] A computer-program product that includes one or more non- transitory computer- readable media with instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations. The operations include generating a 3D mesh of a virtual environment from a set of images of physical terrain and depth information; applying an initial texture to the 3D mesh based on the depth information; generating a projection of a 360-degree panorama onto the 3D mesh, wherein the projection is generated from a perspective of a user in the virtual environment; identifying one or more areas in the projection that are missing the initial texture based on the set of images having one or more portions that are occluded by one or more objects; providing the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information to a machine-learning model; outputting, with the machine-learning model, output texture for the one or more areas in the projection that are missing the initial texture; and blending the output texture with the initial texture in the projection to obtain a blended texture in the projection.

[0009] In some embodiments, the operations further include determining a field of view of the user based on a range of motion of the user; determining a subset of the one or more areasAttorney Docket No.: LE-3049-01-WO in the projection that are not visible to the user at different angles in the range of motion; and excluding the subset of the one or more areas from the identification of the one or more areas that are provided to the machine-learning model. In some embodiments, the operations further include determining a parallax correction for the range of motion by calculating a maximum horizontal angle and a maximum vertical angle of a triangle from a particular position in 3D space; and responsive to a degree of parallax being less than a predetermined number of degrees, replacing a portion of the 3D mesh that is beyond the predetermined number of degrees with a flat surface. Tn some embodiments, the operations further include determining, based on at least one factor, a range of visibility of the user, wherein the at least one factor is selected from a group of a vision capability of the user and prescription lenses used in a head-mounted display worn by the user; and replacing a portion of the 3D mesh that is beyond the range of visibility with a flat surface. In some embodiments, the set of images are captured at different times of day and generating the projection of the 360-degree panorama includes averaging the set of images.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of an example network environment, according to some embodiments described herein.

[0011] Figure 2 is a block diagram of an example computing device, according to some embodiments described herein.

[0012] Figure 3 is an example user interface for specifying one or more areas in the projection that are missing an initial texture, according to some embodiments described herein.

[0013] Figure 4 is an example camera setup for obtaining a 360-degree panorama or images that are used to generate the 360-degree panorama, according to some embodiments described herein.

[0014] Figure 5 illustrates an example scene with different types of terrain that corresponds to different image-capturing methods, according to some embodiments described herein.

[0015] Figure 6A illustrates an example projection of physical terrain with different types of occluded regions, according to some embodiments described herein.

[0016] Figure 6B illustrates an enlarged portion of the projection illustrated in Figure 6A, according to some embodiments described herein.

[0017] Figure 7 illustrates an example of calculation of parallax correction, according to some embodiments described herein.Attorney Docket No.: LE-3049-01-WO

[0018] Figure 8 illustrates an example process of using a machine-learning model to generate output texture, according to some embodiments described herein.

[0019] Figure 9 illustrates an example method to generate output texture that is blended with an initial texture in a projection, according to some embodiments described herein.DETAILED DESCRIPTIONOverview

[0020] Physical terrain may be imaged to create a three-dimensional (3D) virtual environment by capturing two-dimensional (2D) images of the physical terrain and using photogrammetry to stitch together the 2D images to form a 3D model. The images may be captured using aerial photography or videography, where drones or aircraft equipped with high-resolution cameras capture overlapping images of the physical terrain from multiple angles and altitudes. The 2D images are aligned and processed to generate a dense point cloud representing the physical terrain’s surface, which is converted into a 3D mesh. A 3D mesh is composed of triangles (or more generally, polygons) that are connected to each other to form a seamless surface. The 3D mesh is initially textured with a 360-degree panorama of the physical terrain to create a photo-realistic representation.

[0021] The 3D mesh may have areas that are missing the initial texture. For example, an outdoor scene of mountains may include mountains with rocky terrain that prevents areas behind the rocky terrain from being captured in the 2D images from certain camera angles. As a result, the projection on the 3D mesh may have areas that are unrealistic. The missing initial texture can possibly be determined by returning to the physical terrain and capturing more images; however, this is prohibitively expensive and in certain cases, may not be feasible owing to factors such as the nature of the terrain, environment, objects present at the location, etc.

[0022] The technology described herein advantageously solves the problem by using a machine-learning model to output texture for the one or more areas in the 3D mesh that are missing the initial texture and blending the output texture with the initial texture to obtain a blended texture. In some embodiments, the machine-learning model uses depth information to sample from the initial texture at locations with a similar depth as the one or more areas that are missing the initial texture. In some embodiments, the machine-learning model performs in-painting to output the output texture.

[0023] Generating output textures is computationally expensive. In some embodiments, a media application determines a range of motion of the user. For example, the user may haveAttorney Docket No.: LE-3049-01-WO a lens prescription that can be used to determine a range of visibility of the user, the projection may be used for an application with a limited range of motion (e.g., a background for working), the projection may be used for an application with a wider range of motion (e.g., a workout application), etc. The media application uses the range of motion to determine a subset of the one or more areas that are missing the initial texture that are not visible to the user at different angles in the range of motion. For example, the back of a rock may not be visible to the user regardless of how the user moves. The media application excludes a subset of the one or more areas from the identification of the one or more areas that are provided to the machine-learning model. Exclusion of the subset reduces the overall computational cost since output textures are not generated for the excluded areas.Environment

[0024] Figure 1 illustrates a block diagram of an example environment 100. In some embodiments, the environment 100 includes a media server 101, head-mounted displays 115a, 115n, camera devices 130, a Light Detection And Ranging (LiDAR) system 135, and one or more third-party servers 140 that are each coupled to a network 105. Users 125a, 125n may be associated with respective head-mounted displays 115a, 1 15n. In some embodiments, the environment 100 may include other servers or devices not shown in Figure 1. In Figure 1 and the remaining figures, a letter after a reference number, e.g., “115a,” represents a reference to the element having that particular reference number. A reference number in the text without a following letter, e.g., “115,” represents a general reference to embodiments of the element bearing that reference number.

[0025] The camera devices 130 may be computing devices that each include an image sensor and a memory coupled to a hardware processor. For example, the camera devices 130 may include a camera on a drone, a digital Single-Lens Reflex (dSLR) camera, a 360-degree camera, a 3D scanning camera, a camera in a smartphone, etc. The camera devices 130 are communicatively coupled to the network 105 via signal line 102. Signal line 131 may be a wired connection, such as Ethernet, coaxial cable, fiber-optic cable, etc., or a wireless connection, such as Wi-Fi®, Bluetooth®, or other wireless technology. In some embodiments, the camera device 130 includes a memory card that is used to transfer images.

[0026] The camera devices 130 capture a set of images. In some embodiments, different camera devices 130 are used for different functions. For example, a first camera device 130 may be a drone that is used to capture aerial images, a second camera device 130 may be a dSLR camera or a 360-degree camera that is mounted to a tripod to capture panoramic images. A third camera device 130 may be a 3D scanning camera that captures high-Attorney Docket No.: LE-3049-01-WO resolution images. The camera devices 130 capture a set of images and a 360-degree panorama of physical terrain.

[0027] The LiDAR system 135 is a ranging device that measures the distance to a target by sending a laser pulse and recording a time lapse between an outgoing light pulse and detection of the reflected light pulse. The LiDAR system 135 is communicatively coupled to the network 105 via signal line 136. Signal line 136 may be a wired connection or a wireless connection. In some embodiments, the LiDAR system 135 is part of the same device as the camera device 130. For example, a drone may incorporate both LiDAR and a camera to obtain information about physical terrain.

[0028] The third-party server 140 may include a processor, a memory, and network communication hardware. The third-party server 140 is communicatively coupled to the network 105 via signal line 141. Signal line 141 may be a wired connection or a wireless connection. In some embodiments, the third-party server 140 generates a point cloud, a 3D mesh, and / or an initial texture. The third-party server 140 may receive the set of images of physical terrain, a 360-degree panorama of the physical terrain, and / or depth information directly from the media server 101 or from other sources, such as the camera devices 130 and / or the LiDAR system 135.

[0029] The media server 101 may include a processor, a memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to the network 105 via signal line 102.Signal line 102 may be a wired connection or a wireless connection. In some embodiments, the media server 101 sends and receives data to and from one or more of the head- mounted displays 115a, 115n via the network 105. The media server 101 may include a media application 103 a and a database 199.

[0030] The database 199 may store machine-learning models, training data sets, images, etc. The database 199 may also store social network data associated with users 125, user preferences for the users 125, etc.

[0031] The head-mounted display 115 may be a computing device that includes (or can be coupled to) a display screen and a memory coupled to a hardware processor, where the display is used to display extended Reality (XR) content, where XR includes virtual reality (VR), augmented reality (AR), and / or mixed reality (MR). For example, the head-mounted display 115 may be a headset, smart glasses, or another electronic device capable of displaying XR content and accessing a network 105. In some embodiments, the head-Attorney Docket No.: LE-3049-01-WO mounted display 115 includes a media application 103 that receives a projection of physical terrain from the media server 101.

[0032] In the illustrated implementation, head- mounted display 1 15a is coupled to the network 105 via signal line 108 and head-mounted display 115n is coupled to the network 105 via signal line 110. The media application 103 may be stored as media application 103b on the head-mounted display 115a and / or media application 103c on the head-mounted display 115n. Signal lines 108 and 110 may be wired connections, such as Ethernet, coaxial cable, fiber-optic cable, etc., or wireless connections, such as Wi-Fi®, Bluetooth®, or other wireless technology. Head-mounted displays 115a, 115n are accessed by users 125a, 125n, respectively. The head- mounted displays 115a, 115n in Figure 1 are used by way of example. While Figure 1 illustrates two head-mounted displays 115a, 115n, the disclosure applies to a system architecture having one or more head- mounted displays 1 15.

[0033] The media application 103 may be stored and executed on one or more of the media server 101, the head-mounted display 115, or a user device (not shown) that transmits projections to the head-mounted display 115. In some embodiments, the operations described herein are performed on the media server 101. Performance of operations is in accordance with user settings. For example, the user 125a must authorize the use of any personalized settings for determining a field of view based on user behavior. Further, a user 125 a may specify that images and / or other data of the user are to be stored only locally on the headmounted display 115a and not on the media server 101. With such settings, no user data is transmitted to or stored on the media server 101 . Transmission of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and performance of operations on such data by the media server 101 are performed only if the user has agreed to transmission, storage, and performance of operations by the media server 101. Users are provided with options to change the settings at any time, e.g., such that they can enable or disable the use of the media server 101.

[0034] The media application 103 receives a set of images of physical terrain, a 360-degree panorama of the physical terrain from camera devices 130, and depth information. In some embodiments, the media application 103 receives a point cloud of the physical terrain and determines depth information from the point cloud. For example, the media application 103 may receive the point cloud or the depth information from the third-party server 140 or the LiDAR system 135. In some embodiments, the media application 103 generates the point cloud from the set of images and / or the depth information of the physical terrain using photogrammetry. In some embodiments, the point cloud is also generated based on publiclyAttorney Docket No.: LE-3049-01-WO available information about the physical terrain that is provided by a third-party server 140, such as from satellite images.

[0035] The media application 103 generates a 3D mesh of a virtual environment from the set of images of the physical terrain and depth information. The media application 103 applies an initial texture to the 3D mesh based on the depth information. The media application 103 generates a projection of the 360-degree panorama onto the 3D mesh based on a perspective of a user in the virtual environment.

[0036] The media application 103 identifies one or more areas in the 3D mesh that are missing the initial texture based on the set of images having one or more portions that are occluded by one or more objects. The media application 103 provides the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information to a machine-learning model. The machine-learning model outputs output texture for the one or more areas in the projection that are missing the initial texture. The media application 103 blends the output texture with the initial texture in the projection to obtain a blended texture in the projection. In some embodiments, the media application 103 transmits the projection to the head-mounted display 115.

[0037] In some embodiments, the media application 103 may be implemented using hardware including a central processing unit (CPU), a field-programmable gate array (FPGA), an application- specific integrated circuit (ASIC), machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, the media application 103a may be implemented using a combination of hardware and software. Computing Device

[0038] Figure 2 is a block diagram of an example computing device 200 that may be used to implement one or more features described herein. Computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 200 is media server 101 used to implement the media application 103a.

[0039] In some embodiments, computing device 200 includes a processor 235, a memory 237, an input / output (I / O) interface 239, a display 241, and a storage device 245 all coupled via a bus 218. The processor 235 may be coupled to the bus 218 via signal line 222, the memory 237 may be coupled to the bus 218 via signal line 224, the I / O interface 239 may be coupled to the bus 218 via signal line 226, the display 241 may be coupled to the bus 218 via signal line 228, the camera 243 may be coupled to the bus 218 via signal line 230, and the storage device 245 may be coupled to the bus 218 via signal line 232.Attorney Docket No.: LE-3049-01-WO

[0040] Processor 235 can be one or more processors and / or processing circuits to execute program code and control basic operations of the computing device 200. A “processor” includes any suitable hardware system, mechanism or component that processes data, signals or other information. A processor may include a system with a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for achieving functionality, a special-purpose processor to implement neural network modelbased processing, neural circuits, processors optimized for matrix computations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more co-processors that implement neural-network processing. In some embodiments, processor 235 may be a processor that processes data to produce probabilistic output, e.g., the output produced by processor 235 may be imprecise or may be accurate within a range from an expected output. Processing need not be limited to a particular geographic location or have temporal limitations. For example, a processor may perform its functions in real-time, offline, in a batch mode, etc. Portions of processing may be performed at different times and at different locations, by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0041] Memory 237 is typically provided in computing device 200 for access by the processor 235, and may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), Electrical Erasable Read-only Memory (EEPROM), Flash memory, etc., suitable for storing instructions for execution by the processor or sets of processors, and located separate from processor 235 and / or integrated therewith. Memory 237 can store software operating on the computing device 200 by the processor 235, including a media application 103.

[0042] The memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 can include, e.g., an image library application, an image management application, an image gallery application, communication applications, web hosting engines or applications, media sharing applications, an extended reality application, etc. One or more methods disclosed herein can operate in several environments and platforms, e.g., as a stand-alone computer program that can run on any type of computing device, as a web application having web pages, as a mobile application ("app") run on a mobile computing device, etc.Attorney Docket No.: LE-3049-01-WO

[0043] The application data 266 may be data generated by the other applications 264 or hardware of the computing device 200. For example, the application data 266 may include images used by the image library application and user actions identified by the other applications 264 (e.g., a social networking application), etc.

[0044] I / O interface 239 can provide functions to enable interfacing the computing device 200 with other systems and devices. Interfaced devices can be included as part of the computing device 200 or can be separate and communicate with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices can communicate via I / O interface 239. In some embodiments, the FO interface 239 can connect to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, scanner, sensors, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0045] Some examples of interfaced devices that can connect to VO interface 239 can include a display 241 that can be used to display content, e.g., images, projections, and / or a user interface as described herein, and to receive touch (or gesture) input from a user. For example, display 241 may be utilized to display a user interface that includes a graphical guide on a viewfinder. Display 241 can include any suitable display device such as a liquid crystal display (LCD), light emitting diode (LED), or plasma display screen, cathode ray tube (CRT), television, monitor, touchscreen, three-dimensional display screen, or other visual display device. For example, display 241 can be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or headset device, or a monitor screen for a computer device.

[0046] The storage device 245 stores data related to the media application 103. For example, the storage device 245 may store a training data set that includes labeled images, a machinelearning model, output from the machine-learning model, etc.Media Application

[0047] Figure 2 illustrates an example media application 103, stored in memory 237, that includes a user interface module 202, an image processing module 204, and a machinelearning module 206.

[0048] The user interface module 202 generates graphical data for displaying a user interface that is used to create XR content. The XR content may be used for a variety of purposes including a background that is displayed during an exercise application, a background that includes different locations that a user views during planning of a vacation, a background thatAttorney Docket No.: LE-3049-01-WO is used while the user displays a work screen and works at a standing desk, a virtual environment for a game, etc.

[0049] In some embodiments, the user interface includes functionality for importing a set of images of physical terrain and a 360-degree panorama, instructing the image processing module 204 to generate the 3D mesh, instructing the image processing module 204 to apply an initial texture to the 3D mesh, and instructing the image processing module 204 to generate a projection of the 360-panorama onto the 3D mesh. The set of images imported by the user interface module 202 may be received from the media server 101 via the I / O interface 239.

[0050] In some embodiments, the user interface module 202 generates a user interface that includes options for specifying user preferences. For example, a user may provide information such as the user’s age, gender, and height. The user provides user consent to use the information for customization of a virtual environment. If the user does not provide user consent to use the information, the virtual environment is not customized but still provides the remaining functionality.

[0051] Once the user interface module 202 displays the user interface of the projection, one or more areas in the projection may be missing the initial texture. The missing initial texture may result from one or more portions that are occluded by one or more objects. For example, a camera may capture images of slot canyons where the curve of the canyon walls prevents other parts of the walls from being visible through occlusion. The user interface module 202 identifies the one or more areas in the projection that are missing the initial texture. In some embodiments, the user interface module 202 automatically identifies the location of the one or more areas. For example, the user interface module 202 may perform ray tracing from a position of the camera that captured the 360-degree images to identify areas that are missing the initial texture. In some embodiments, the user interface module 202 includes an option for a user to select one or more areas for correction.

[0052] Figure 3 is an example user interface 300 for specifying one or more areas in the projection that are missing an initial texture, according to some embodiments described herein. The user interface 300 includes a display panel 302 and an instruction panel 305. The display panel 302 includes a projection of a mountain range from the perspective of a user on top on a particular mountain 310. The particular mountain 310 has an area 315 that is missing the initial texture in the captured images. The missing initial texture is discernable because the area 315 lacks texture and instead is a single shade of color and appears flat unlike the rest of the mountain 310, which is textured.Attorney Docket No.: LE-3049-01-WO

[0053] The user interface 300 provides an option for a user to select the area 315 for correction. For example, the user may drag a cursor 319 around the area 315 to specify a region 320 for correction. Once the user specifies the region 320, the user may select a “correct missing texture” button 322. The user interface 300 also includes an “identify missing initial texture” button 325. Responsive to the user selecting the “identify missing initial texture” button, the user interface module 202 automatically identifies one or more areas in the projection that are missing the initial texture.

[0054] In some embodiments, the image processing module 204 receives a 360-degree panorama of the physical terrain. The 360-degree panorama may be generated by a 360- degree camera, a dSLR camera attached to a tripod with a motorized head, etc. In some embodiments, the image processing module 204 generates the 360-degree panorama of the physical terrain by stitching together multiple individual images that are taken within a short time frame to ensure consistent lighting.

[0055] Figure 4 is an example camera setup 400 for obtaining a 360-degree panorama or images that are used to generate the 360-degree panorama, according to some embodiments described herein. In some embodiments, the camera setup 400 includes a camera 405 that is mounted on a tripod 410. The tripod 410 is configured to rotate 360 degrees. In some embodiments, the tripod 410 automatically rotates while the camera 405 captures images.

[0056] In some embodiments, the camera 405 is configured to have a height 415 about equal to the height of a person so that the 360-degree panorama is from an average user’s perspective. For example, the height may be between 160-200 centimeters (cm). In some embodiments, the images are captured at different heights to personalize the experience for users. For example, images captured at a height of 105 cm may be used for generating a projection for a user that is a child (and that is used upon user consent of the child’s parents). In some embodiments, the distance 420 between the legs of the tripod 410 may be around 120 cm. In some embodiments, the camera 405 is positioned on a flat ground with a radius 425 of at least 150 cm (where the radius 425 is measured from a center of the tripod to a circle 430).

[0057] The camera 405 may capture high-resolution images of the flat ground to be used for texturing. The camera 405 may capture images at predetermined periods of time, such as every five minutes during the day, every one minute during the golden hour, and every 30 seconds during sunrise and sunset. The camera 405 may also capture long exposure images after twilight. In some embodiments, before attaching the camera 405 to the tripod 410, the camera 405 captures one or more nadir images and zenith images. In some embodiments, theAttorney Docket No.: LE-3049-01-WO image processing module 204 generates a 360-degree panorama from the images by averaging the set of images to soften shadows and reduce glare from sunlight.

[0058] The image processing module 204 receives a set of images of physical terrain from the media server 101. The set of images may be captured from different types of cameras, such as a dSLR, a camera on a drone or other types of aerial photography, a 3D scanning camera, etc. The set of images are captured at different times of day.

[0059] Figure 5 illustrates an example scene 500 with different types of terrain that corresponds to different image-capturing methods, according to some embodiments described herein. The scene 500 is divided based on types of terrain: a sky region 505, distant terrain 510, nearby terrain 515, intermediate terrain 520, closest terrain 525, and ground 530.

[0060] The sky region 505 may be captured with a camera on a drone (not shown), a 360- degree camera 535 during the day, and a dSLR camera (not shown) at night. Images of distant terrain 510a, 510b may be received from a third-party server, such as a satellite service, and used by the image processing module 204 to generate a 3D mesh. Nearby terrain 515 may be captured by a camera on a drone. Intermediate terrain 520 may be captured with a medium level of detail by the 360-degree camera 535. The closest terrain 525 may be captured with a high level of detail by a 3D scanning camera. The ground 530 may be captured by a variety of sources including the 360-degree camera 535, a camera of a drone, etc. and may include lower-quality images than the images captured for the intermediate terrain 520. In some embodiments, the image processing module 204 uses markers for alignment of the different images, such as marker 540.

[0061] The image processing module 204 processes the set of images. In some embodiments, the image processing module 204 generates a 3D mesh of a virtual environment from the set of images and depth information. The depth information may be received from a LiDAR system, determined from a point cloud, determined from the set of images, etc. In some embodiments, the image processing module 204 generates a 360-degree panorama or receives the 360-degree panorama from a camera and generates the 3D mesh based on the 360-degree panorama as well.

[0062] In some embodiments, the image processing module 204 applies an initial texture to the 3D mesh based on the depth information. The initial texture may be a non-color- corrected texture that uses the set of images taken under different lighting conditions.

[0063] In some embodiments, the image processing module 204 performs image enhancement of the 360-degree panorama. For example, the image processing module 204 may perform color grading, detail adjustments, and blend multiple 360-degree panoramas.Attorney Docket No.: LE-3049-01-WOThe detail adjustments may include increasing or decreasing contrast to reduce flickering caused by display pixel aliasing, averaging pixels to soften shadows and increase an aesthetic appearance of the shadows, etc. In some embodiments, the image processing module 204 averages the images in the 360-degree panorama to decrease glare.

[0064] In some embodiments, the image processing module 204 may identify an alignment issue with pixels that occur as a result of environmental changes (e.g., wind may blow away sand or leaves). The image processing module 204 may provide unaligned pixels to the machine-learning module 206 with a request for a 360-degree panorama or images for the 360-degree panorama that modify the images such that they may be aligned.

[0065] The image processing module 204 generates a projection of the 360-degree panorama onto the 3D mesh. The projection is generated from a perspective of a user in the virtual environment. For example, the projection is based on the 360-degree panorama being captured at a particular height to be similar to a view of a person. If multiple 360-degree panoramas are available based on different heights, the image processing module 204 may select a 360-degree panorama that is used for generating the projection based on a height of the user.

[0066] The image processing module 204 identifies one or more areas in the projection that are missing the initial texture based on the set of images having one or more portions that are occluded by one or more objects. The image processing module 204 identifies parameters of the one or more areas that are missing the initial texture. For example, the image processing module 204 may receive locations of missing initial texture from the user interface module 202 as a result of a user circling the area or from the user interface module 202 determining the one or more areas. In some embodiments, the image processing module 204 generates a mask that encompasses the one or more areas with missing initial texture. For example, the mask may be a map that identifies whether pixels in the projection include initial texture or are missing initial texture.

[0067] In some embodiments, the image processing module 204 determines that a subset of the one or more areas in the projection are not visible to the user regardless of how the user turns their head or where the user moves. For example, if a user is on the top of a mountain and the virtual environment is designed for the user to walk no more than five feet in any direction, certain areas on the other side of the mountain are not visible to the user. As a result, the image processing module 204 may exclude the subset of the one or more areas of the projection from the identification of the one or more areas in the projection that are occluded. In embodiments where the machine-learning model is an in-painter model, theAttorney Docket No.: LE-3049-01-WO image processing module 204 may exclude the one or more areas of the projection from the mask.Determination of Areas that Are Not Visible to a User

[0068] In some embodiments, the image processing module 204 determines the subset of the one or more areas in the projection are not visible to the user by determining a field of view of the user based on a range of motion of the user. For example, the user may be able to walk up to five feet (one meter) in any direction starting from the origin. The image processing module 204 determines the subset of the one or more areas in the projection that are not visible to the user at different angles in the range of motion and excludes the subset of the one or more areas from the identification of the areas that are provided by the machine- learning module 206. In some embodiments where the machine-learning model is an in-painter model, the image processing module 204 may remove the subset of the one or more areas from the mask provided to the machine-learning module 206.

[0069] Figure 6A illustrates an example projection 600 of physical terrain with different types of occluded regions, according to some embodiments described herein. Figure 6B illustrates an enlarged portion 650 of the projection illustrated in Figure 6A, according to some embodiments described herein.

[0070] In Figure 6B, the largest circle 655 defines the maximum allowable head range of the user. The largest circle 655 may vary depending on the type of application that is using the XR content. For example, the largest circle 655 may be larger for a video game that is designed to let a user explore a virtual environment. The circle 655 may be smaller for a productivity application. The center circle 657 represents the center of projection, which is also the current head position. The two smallest circles 659a, 659b illustrate the head range of motion for the user.

[0071] The head-mounted display worn by the user configures a field of view for the user. For example, a longer display corresponds to a wider field of view. The field of view may vary between different models. Figure 6B illustrates the field of view 661a, 661b. A mountain 663 is within the user’s field of view 661. The front of the mountain 663 is illustrated with dashed lines, such as the one associated with reference character 665. The occluded areas are identified with solid lines, such as 667. Occluded area 667 is missing initial texture because the images of the mountain 663 were captured from the front of the mountain 663, such that the capturing device had no line of sight to the occluded area 667.Attorney Docket No.: LE-3049-01-WO

[0072] The image processing module 204 determines that a subset 669 of the occluded area 667 is not visible to the user regardless of how the user moves within the largest circle 655. The subset 669 is illustrated in Figure 6B as the region enclosed by the gray lines.

[0073] In some embodiments, the image processing module 204 determines a range of visibility of a user. The range of visibility may be based on the head-mounted display worn by the user, a vision capability of the user, and / or prescription lenses used in the headmounted display. The image processing module 204 may replace a portion of the 3D mesh that is beyond the range of visibility with a flat surface.Correction of Parallax

[0074] A problem arises in generating projections for users based on parallax. When a user moves within a virtual environment, the user’s line of sight changes, resulting in objects in the scene undergoing different displacements depending on their depth. This makes it appear to the user that objects are moving. In some embodiments, the image processing module 204 determines a parallax correction for the range of motion of a user by calculating a maximum horizontal angle and a maximum vertical angle of a triangle from a particular position in 3D space and responsive to a degree of parallax being less than a predetermined number of degrees, replacing a portion of the 3D mesh that is beyond the predetermined number of degrees with a flat surface, merging a first triangle with an adjacent triangle, and / or reducing a texture mapping for the first triangle.

[0075] Figure 7 illustrates an example 700 of calculation of parallax correction, according to some embodiments described herein. In this example 700, a user 705 is viewing a mountain range. The user is standing stationery and moves his head to two positions 710, 715 while looking at a point 730 in the mountains. The point 730 is associated with a triangle that is part of the 3D mesh. The image processing module 204 determines a parallax correction for the range of motion such that if a degree of parallax is less than a predetermined number of degrees, the triangle that encompasses the point 730 is replaced with a flat surface or merged with another triangle and / or the texture mapping for the triangle is reduced. This advantageously avoids the user experiencing movement of the mountains in response to moving their head and also advantageously reduces the storage demands for the projection since it reduces the amount of the 3D scene that includes a 3D mesh.

[0076] In some embodiments, the image processing module 204 determines the parallax correction by defining a position in 3D space and defining a triangle in the 3D space according to three vertices with x-, y-, and z-coordinates. The triangle is associated with a texture.Attorney Docket No.: LE-3049-01-WO

[0077] The image processing module 204 calculates a centered triangle by subtracting the position from each of the vertices. For example, if the position is defined as (2, 1, 1) and the triangle is defined as (1, 2, 3), (4, -1, 2), and (0, 5, 1), the centered triangle is (1-2, 2-1, 3-1), (4-2, -1-1, 2-1), and (0-2, 5-1, 1-1) for a result of (-1, 1, 2), (2, -2, 1), and (-2, 4, 0). The image processing module 204 calculates projected vertices onto the unit sphere based on a magnitude of the vector where the magnitude is calculated using the following equation:

[0079] where v[0] is the first vertex, v

[0001] is the second vertex, and v[2] is the third vertex.

[0080] The image processing module 204 calculates the azimuthal angles based on the projected vertices for the first and second vertices and a maximum horizontal angle as a difference between the maximum and minimum azimuthal angles. The image processing module 204 calculates the maximum vertical angle as a difference between maximum and minimum polar angles for the third vertex.

[0081] The image processing module 204 uses the maximum horizontal angle and the maximum vertical angle to determine if the triangle should be flattened / merged with an adjacent triangle or the texture mapping for the triangle should be reduced based on the overall Pixel Per Degree (PPD) resolution requirements, where the PPD resolution depends on a minimum resolution and a field of view of the head mounted display worn by the user. For example, although a ground plane has a texture with a large surface area, based on the field of view calculations the user sees a compacted version of the ground plane. Because the extra details are not visible to the user, the image processing module 204 may flatten the triangle to give it the same (or similar) look as the surrounding triangles and reduce the texture size for the triangles.

[0082] The machine-learning module 206 receives the projection, the identification of the one or more areas in the projection, and the depth information from the image processing module 204. The machine- learning module 206 includes a machine-learning model that is trained to output texture for the one or more areas in the projection that are missing the initial texture. The machine-learning module 206 blends the output texture with the initial texture in the projection to obtain a blended texture in the projection. In some embodiments, the machine-learning module 206 may implement a machine-learning model that is an in-painter model, e.g., a model that performs in-painting to fill in pixels that are missing. For example, the one or more areas in the projection that are missing the initial texture are adjacent to otherAttorney Docket No.: LE-3049-01-WO areas that have the initial texture and the in-painter model predicts values for the pixels that are missing based at least in part on pixel values of pixels in the adjacent areas.

[0083] In some embodiments, the in-painter model uses the depth information to identify a depth of the one or more areas that are missing initial texture and to generate output texture that matches initial textures that have a similar depth. In some embodiments, the in-painter model uses a gradient of neighborhood textures to determine properties of the one or more areas that are missing the initial texture. For example, the machine- learning model may be trained to identify types of geographical features (e.g., rock, ground, sand, foliage, etc.) and to apply textures to the missing initial texture based on the geographical features identified for the neighboring areas.

[0084] Figure 8 illustrates an example process 800 of using a machine-learning model to generate output texture, according to some embodiments described herein. A projection 805 with an area 806 that is missing initial texture, depth information 810, and a mask 815 is provided to an in-painter model 820. The mask 815 may indicate the region to be filled in, e.g., a portion of area 806. The in-painter model 820 generates output texture and blends the output texture with the projection 805 to form a projection 825 with blended texture.

[0085] In some embodiments, the machine- learning module 206 may specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) enabling processor 235 to apply a machine-learning model. In some embodiments, the machine-learning module 206 may include software instructions, hardware instructions, or a combination. In some embodiments, the machine-learning module 206 may offer an application programming interface (API) that can be used by the operating system 262 and / or other applications 264 to invoke the machine-learning module 206 e.g., to apply the machine-learning model to application data 266 to output the preserving mask.

[0086] The machine-learning module 206 uses training data to generate a trained machinelearning model. For example, training data may include pairs of input projections with a ground truth projection that does not include missing initial texture and a corresponding projection with one or more areas that are missing initial textures, as well as depth information. In this case, supervised learning can be utilized to train a machine-learning model to inpaint (or otherwise generate) texture that matches the ground truth projection by adjusting model parameters.

[0087] In some embodiments, the machine- learning model is an in-painter model that is trained to generate output texture based on the initial texture and the depth information. In some embodiments, the in-painter model is trained with pairs that each include a ground truthAttorney Docket No.: LE-3049-01-WO image that represents a full image and a masked version of the ground truth image, along with depth information. The masked images in the training data may include masks in random areas (arbitrary masks) in order to train the in-painter model to generate output texture for different areas.

[0088] Training data may be obtained from any source, e.g., a data repository specifically marked for training, data for which permission is provided for use as training data for machine learning, etc. In some embodiments, the training may be performed on the media server 101 that provides the training data directly to the head-mounted display 1 15, the training may be performed locally on the head-mounted display 115, or a combination of both.

[0089] In some embodiments, the machine- learning module 206 uses weights that are taken from another application and are unedited / transferred. For example, in these embodiments, the trained model may be generated, e.g., on a different device, and be provided as part of the machine-learning module 206. In various embodiments, the trained model may be provided as a data file that includes a model structure or form (e.g., that defines a number and type of neural network nodes, connectivity between nodes and organization of the nodes into a plurality of layers), and associated weights. The machine-learning module 206 may read the data file for the trained model and implement neural networks with node connectivity, layers, and weights based on the model structure or form specified in the trained model.

[0090] The trained machine-learning model may include one or more model forms or structures. For example, model forms or structures can include any type of neural-network, such as a linear network, a deep-learning neural network that implements a plurality of layers (e.g., “hidden layers” between an input layer and an output layer, with each layer being a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, processes each tile separately using one or more neural- network layers, and aggregates the results from the processing of each tile), a sequence-to- sequence neural network (e.g., a network that receives as input sequential data, such as 3D meshes with initial texture in a projection, etc. and produces as output a result sequence), etc. In some embodiments, the machine- learning model is an in-painter model that includes a Generative Adversarial Network (GAN) or a diffusion model.

[0091] The model form or structure may specify connectivity between various nodes and organization of nodes into layers. For example, nodes of a first layer (e.g., an input layer) may receive data as input data or application data. Such data can include, for example, one or more pixels per node, e.g., when the trained model is used for analysis, e.g., of an inputAttorney Docket No.: LE-3049-01-WO image. Subsequent intermediate layers may receive as input, output of nodes of a previous layer per the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. For example, a first layer may output an first layer of texture. A final layer (e.g., output layer) produces an output of the machine-learning model. For example, the output layer may output the output texture. In some embodiments, model form or structure also specifies a number and / or type of nodes in each layer.

[0092] In different embodiments, the trained model can include one or more models. One or more of the models may include a plurality of nodes, arranged into layers per the model structure or form. In some embodiments, the nodes may be computational nodes with no memory, e.g., configured to process one unit of input to produce one unit of output. Computation performed by a node may include, for example, multiplying each of a plurality of node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to produce the node output. In some embodiments, the computation performed by a node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such computation may include operations such as matrix multiplication. In some embodiments, computations by the plurality of nodes may be performed in parallel, e.g., using multiple processors cores of a multicore processor, using individual processing units of a graphics processing unit (GPU), or special-purpose neural circuitry. In some embodiments, nodes may include memory, e.g., may be able to store and use one or more earlier inputs in processing a subsequent input. For example, nodes with memory may include long short-term memory (LSTM) nodes. LSTM nodes may use the memory to maintain “state” that permits the node to act like a finite state machine (FSM).

[0093] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, a model may be initiated as a plurality of nodes organized into layers as specified by the model form or structure. At initialization, a respective weight may be applied to a connection between each pair of nodes that are connected per the model form, e.g., nodes in successive layers of the neural network. For example, the respective weights may be randomly assigned, or initialized to default values. The model may then be trained, e.g., using training data, to produce a result.

[0094] Training may include applying supervised learning techniques. In supervised learning, the training data can include a plurality of inputs (e.g., projections, depth information, masks, ground truth texture, etc.) and a corresponding ground truth output for each input (e.g., an output texture). Based on a comparison of the output of the model withAttorney Docket No.: LE-3049-01-WO the ground truth output, values of the weights are automatically adjusted, e.g., in a manner that increases a probability that the model produces the ground truth output for the image.

[0095] In various embodiments, a trained model includes a set of weights, or embeddings, corresponding to the model structure. In some embodiments, the trained model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights. In various embodiments, a trained model includes a set of weights, or embeddings, corresponding to the model structure. In embodiments where data is omitted, the machinelearning module 206 may generate a trained model that is based on prior training, e.g., by a developer of the machine-learning module 206, by a third-party, etc. In some embodiments, the trained model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights.

[0096] In some embodiments, the machine- learning module 206 receives feedback, such as a rating of the ground truth images from one or more users. The rating may include numbers on a scale. The machine-learning module 206 may use the ratings as metadata associated with the ground truth images. For example, the machine-learning module 206 may train an in-painter model to generate output texture with threshold quality score.

[0097] The machine-learning module 206 blends the output texture with the initial texture in the projection to obtain a blended texture in the projection. In some embodiments, the machine-learning module 206 transmits the projection with the blended texture to a headmounted display for downloading and viewing by the user.

[0098] In some embodiments, after the projection with the blended texture is generated, the image processing module 204 receives the projection and adds shadows to the projection to imrpove the realistic nature of the projection.Method

[0099] Figure 9 illustrates an example method 900 to generate output texture that is blended with a projection, according to some embodiments described herein. The method 900 may be performed by the computing device 200 in Figure 2. In some embodiments, the method 900 is performed by the media server 101 in Figure 1.

[0100] The method 900 of Figure 9 may begin at block 902. At block 902, a 3D mesh of a virtual environment is generated from a set of images of physical terrain and depth information. In some embodiments, the set of images are captured from one or more sources selected from a group of a digital Single-Lens Reflex (dSLR) camera, a 360-degree camera, a drone, and combinations thereof. Block 902 may be followed by block 904.Attorney Docket No.: LE-3049-01-WO

[0101] At block 904, an initial texture is applied to the 3D mesh based on the depth information. Block 904 may be followed by block 906.

[0102] At block 906, a projection of a 360-degree panorama onto the 3D mesh is generated, where the projection is generated from a perspective of a user in the virtual environment. Some embodiments include determining, based on at least one factor, a range of visibility of the user, wherein the at least one factor is selected from a group of a vision capability of the user and prescription lenses used in a head-mounted display worn by the user; and replacing a portion of the 3D mesh that is beyond the range of visibility with a flat surface. In some embodiments, the set of images are captured at different times of day and generating the projection of the 360-degree panorama includes averaging the set of images. Block 906 may be followed by block 908.

[0103] At block 908, one or more areas in the projection that are missing the initial texture are identified based on the set of images having one or more portions that are occluded by one or more objects. Some embodiments include determining a field of view of the user based on a range of motion of the user; determining a subset of the one or more areas in the projection that are not visible to the user at different angles in the range of motion; and excluding the subset of the one or more areas from the identification of the one or more areas that are provided to the machine-learning model. Some embodiments include determining a parallax correction for the range of motion by calculating a maximum horizontal angle and a maximum vertical angle of a triangle from a particular position in 3D space; and responsive to a degree of parallax being less than a predetermined number of degrees, replacing a portion of the 3D mesh that is beyond the predetermined number of degrees with a flat surface. Block 908 may be followed by block 910.

[0104] At block 910, the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information are provided to a machine-learning model. In some embodiments, the machine-learning model is an in-painter model and the identification of the one or more areas in the projection is a mask. Block 910 may be followed by block 912.

[0105] At block 912, the machine-learning model outputs output texture for the one or more areas in the projection that are missing the initial texture. Block 912 may be followed by block 914.

[0106] At block 914, the output texture is blended with the initial texture in the projection to obtain a blended texture in the projection. Some embodiments include adding shadows to the blended texture in the projection. Some embodiments include transmitting theAttorney Docket No.: LE-3049-01-WO projection with the blended texture to an HMD associated with the user. The projection and the blended texture can be used to display the virtual environment to the user via the HMD, to allow the user to view different areas by moving about within a specified range while in the virtual environment and to enable the user to view the virtual environment from different perspectives, e.g., by turning their head while wearing the HMD.

[0107] Further to the descriptions above, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user’s social network, social actions, or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’ s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0108] In the above description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the specification. It will be apparent, however, to one skilled in the art that the disclosure can be practiced without these specific details. In some instances, structures and devices are shown in block diagram form in order to avoid obscuring the description. For example, the embodiments can be described above primarily with reference to user interfaces and particular hardware. However, the embodiments can apply to any type of computing device that can receive data and commands, and any peripheral devices providing services.

[0109] Reference in the specification to “some embodiments” or “some instances” means that a particular feature, structure, or characteristic described in connection with the embodiments or instances can be included in at least one implementation of the description. The appearances of the phrase “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiments.

[0110] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizingAttorney Docket No.: LE-3049-01-WO terms including “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.

[0111] The embodiments of the specification can also relate to a processor for performing one or more steps of the methods described above. The processor may be a special-purpose processor selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory computer- readable storage medium, including, but not limited to, any type of disk including optical disks, ROMs, CD-ROMs, magnetic disks, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memories including USB keys with non-volatile memory, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0112] The specification can take the form of some entirely hardware embodiments, some entirely software embodiments or some embodiments containing both hardware and software elements. In some embodiments, the specification is implemented in software, which includes, but is not limited to, firmware, resident software, microcode, etc.

[0113] Furthermore, the description can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0114] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

Claims

Attorney Docket No.: LE-3049-01-WOCLAIMSWhat is claimed is:

1. A computer-implemented method, the method comprising: generating a three-dimensional (3D) mesh of a virtual environment from a set of images of physical terrain and depth information; applying an initial texture to the 3D mesh based on the depth information; generating a projection of a 360-degree panorama onto the 3D mesh, wherein the projection is generated from a perspective of a user in the virtual environment; identifying one or more areas in the projection that are missing the initial texture based on the set of images having one or more portions that are occluded by one or more objects; providing the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information to a machine-learning model; outputting, with the machine-learning model, output texture for the one or more areas in the projection that are missing the initial texture; and blending the output texture with the initial texture in the projection to obtain a blended texture in the projection.

2. The method of claim 1, further comprising: determining a field of view of the user based on a range of motion of the user; determining a subset of the one or more areas in the projection that are not visible to the user at different angles in the range of motion; and excluding the subset of the one or more areas from the identification of the one or more areas that are provided to the machine-learning model.

3. The method of claim 2, further comprising: determining a parallax correction for the range of motion by calculating a maximum horizontal angle and a maximum vertical angle of a triangle from a particular position in 3D space; andAttorney Docket No.: LE-3049-01-WO responsive to a degree of parallax being less than a predetermined number of degrees, replacing a portion of the 3D mesh that is beyond the predetermined number of degrees with a flat surface.

4. The method of claim 1, further comprising: determining, based on at least one factor, a range of visibility of the user, wherein the at least one factor is selected from a group of a vision capability of the user and prescription lenses used in a head-mounted display worn by the user; and replacing a portion of the 3D mesh that is beyond the range of visibility with a flat surface.

5. The method of claim 1 , wherein the set of images are captured at different times of day and wherein generating the projection of the 360-degree panorama includes averaging the set of images.

6. The method of claim 1, wherein the set of images are captured from one or more sources selected from a group of a digital Single- Lens Reflex (dSLR) camera, a 360-degree camera, a drone, and combinations thereof.

7. The method of claim 1, wherein the machine-learning model is an in-painter model and the identification of the one or more areas in the projection is a mask.

8. The method of claim 1, further comprising: adding shadows to the blended texture in the projection.

9. The method of claim 1 , further comprising: transmitting the projection with the blended texture to a head-mounted display associated with the user.Attorney Docket No.: LE-3049-01-WO10. A computing device comprising: one or more processors; and one or more memories in communication with the one or more processors, with instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: generating a three-dimensional (3D) mesh of a virtual environment from a set of images of physical terrain and depth information; applying an initial texture to the 3D mesh based on the depth information; generating a projection of a 360-degree panorama onto the 3D mesh, wherein the projection is generated from a perspective of a user in the virtual environment; identifying one or more areas in the projection that are missing the initial texture based on the set of images having one or more portions that are occluded by one or more objects; providing the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information to a machine- learning model; outputting, with the machine- learning model, output texture for the one or more areas in the projection that are missing the initial texture; and blending the output texture with the initial texture in the projection to obtain a blended texture in the projection.

11. The computing device of claim 10, wherein the operations further include: determining a field of view of the user based on a range of motion of the user; determining a subset of the one or more areas in the projection that are not visible to the user at different angles in the range of motion; and excluding the subset of the one or more areas from the identification of the one or more areas that are provided to the machine-learning model.

12. The computing device of claim 11, wherein the operations further include: determining a parallax correction for the range of motion by calculating a maximum horizontal angle and a maximum vertical angle of a triangle from a particular position in 3D space; andAttorney Docket No.: LE-3049-01-WO responsive to a degree of parallax being less than a predetermined number of degrees, replacing a portion of the 3D mesh that is beyond the predetermined number of degrees with a flat surface.

13. The computing device of claim 10, wherein the operations further include: determining, based on at least one factor, a range of visibility of the user, wherein the at least one factor is selected from a group of a vision capability of the user and prescription lenses used in a head-mounted display worn by the user; and replacing a portion of the 3D mesh that is beyond the range of visibility with a flat surface.

14. The computing device of claim 10, wherein the set of images are captured at different times of day and wherein generating the projection of the 360-degree panorama includes averaging the set of images.

15. The computing device of claim 10, wherein the set of images are captured from one or more sources selected from a group of a digital Single- Lens Reflex (dSLR) camera, a 360- degree camera, a drone, and combinations thereof.

16. A computer-program product that includes one or more non-transitory computer- readable media with instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations comprising: generating a three-dimensional (3D) mesh of a virtual environment from a set of images of physical terrain and depth information; applying an initial texture to the 3D mesh based on the depth information; generating a projection of a 360-degree panorama onto the 3D mesh, wherein the projection is generated from a perspective of a user in the virtual environment; identifying one or more areas in the projection that are missing the initial texture based on the set of images having one or more portions that are occluded by one or more objects; providing the projection, identification of the one or more areas in the projection that are missing the initial texture, and the depth information to a machine-learning model;Attorney Docket No.: LE-3049-01-WO outputting, with the machine- learning model, output texture for the one or more areas in the projection that are missing the initial texture; and blending the output texture with the initial texture in the projection to obtain a blended texture in the projection.

17. The computer-program product of claim 16, wherein the operations further include: determining a field of view of the user based on a range of motion of the user; determining a subset of the one or more areas in the projection that are not visible to the user at different angles in the range of motion; and excluding the subset of the one or more areas from the identification of the one or more areas that are provided to the machine-learning model.

18. The computer-program product of claim 17, wherein the operations further include: determining a parallax correction for the range of motion by calculating a maximum horizontal angle and a maximum vertical angle of a triangle from a particular position in 3D space; and responsive to a degree of parallax being less than a predetermined number of degrees, replacing a portion of the 3D mesh that is beyond the predetermined number of degrees with a flat surface.

19. The computer-program product of claim 16, wherein the operations further include: determining, based on at least one factor, a range of visibility of the user, wherein the at least one factor is selected from a group of a vision capability of the user and prescription lenses used in a head-mounted display worn by the user; and replacing a portion of the 3D mesh that is beyond the range of visibility with a flat surface.

20. The computer-program product of claim 16, wherein the set of images are captured at different times of day and wherein generating the projection of the 360-degree panorama includes averaging the set of images.

Images