Display mode selection for digitized scene rendering

Abstract

There is provided techniques for selecting a display mode for an application visualizing a digitized scene. The method comprises selecting the display mode based on whether the location data of the user device satisfies a proximity condition with respect to the location data of the digitized scene. A remote inspection mode is selected when the proximity condition is not satisfied, and an onsite inspection mode is selected when the proximity condition is satisfied. The method comprises rendering the digitized scene on the display according to the selected display mode.

Description

[0001] DISPLAY MODE SELECTION FOR DIGITIZED SCENE RENDERING

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to a method, a user device, a computer program, and a computer program product for selecting a display mode for an application visualizing a digitized scene.

[0004] BACKGROUND

[0005] Advancements in imaging technology have enhanced the ability to create detailed three-dimensional (3D) representations of physical environments. Such representations are widely utilized in industrial and end-user applications to capture and interact with the spatial geometry of various scenes. The process of acquiring 3D scene geometry typically involves the use of scanning technologies such as visual cameras and LiDAR sensors. These technologies facilitate the generation of diverse forms of 3D data, including pointclouds, 3D meshes, and continuous volumetric representations, such as Neural Radiance Fields (NeRF) and Gaussian Splatting.

[0006] Presenting digitized 3D models of, for example, indoor environments, can be achieved by constructing 360-degree panoramic views in terms of Skybox views. These panoramic views rely on 360-degree panoramic images to provide an immersive experience that simulates being physically present within the environment. This allows users to navigate through digitized scenes, perform spatial measurements, and engage with the 3D space in an intuitive manner. Interactions with such virtual representations are made accessible through standard two-dimensional screens, such as those on smartphones and tablet computers, making these technologies versatile and user-friendly.

[0007] For enhanced functionality, connected user devices, such as smartphones and tablet computers, can be positioned within indoor environments using radio positioning techniques. Technologies such as 5G-based positioning and Wi-Fi-based positioning utilize indicators like Received Signal Strength (RSS) and Time of Flight (ToF) to determine the location of the device relative to some radio equipment. These positioning capabilities prove particularly valuable in large and complex indoor spaces, such as factories and shopping malls, where users can benefit from precise navigation aids.Applications that visualize digitized scenes have been developed to support both onsite and remote usage scenarios. For instance, technicians inspecting cell sites or factory installations often rely on the same applications as experts conducting remote inspections of the same locations. While these applications are designed to enable remote inspections efficiently, they may not cater to the unique requirements of onsite visits. Specifically, during onsite inspections, positioning within the physical environment is a critical feature, as it assists users in locating and navigating to specific sections of an installation. Conversely, this functionality holds little relevance in remote scenarios, where the user device used for visualization is not physically present within the environment being inspected.

[0008] The dual-purpose nature of scene visualization applications highlights the need to balance usability across diverse operational contexts. Hence, there is still a need for improved scene visualization applications.

[0009] SUMMARY

[0010] An object of embodiments herein is to address the above issues.

[0011] A particular object is to enable the proper display mode to be used for the given operational context.

[0012] According to a first aspect there is presented a method for selecting a display mode for an application visualizing a digitized scene, performed by a user device that comprises a display. The method comprises obtaining a 3D representation of the digitized scene, location data of the digitized scene, and location data of the user device. The method comprises selecting the display mode based on whether the location data of the user device satisfies a proximity condition with respect to the location data of the digitized scene. A remote inspection mode is selected when the proximity condition is not satisfied, and an onsite inspection mode is selected when the proximity condition is satisfied. The method comprises rendering the digitized scene on the display according to the selected display mode. The remote inspection mode renders a full 3D model, and the onsite inspection mode renders a lightweight representation of the digitized scene. A relative position of the user device in relation to the digitized scene is marked in the lightweight representation. The lightweight representation is derived from the full 3D display model. The relative position of theuser device is given by the location data of the user device and the location data of the digitized scene.

[0013] According to a second aspect there is presented a user device for selecting a display mode for an application visualizing a digitized scene. The user device comprises a display and processing circuitry. The processing circuitry is configured to cause the user device to obtain a 3D representation of the digitized scene, location data of the digitized scene, and location data of the user device. The processing circuitry is configured to cause the user device to select the display mode based on whether the location data of the user device satisfies a proximity condition with respect to the location data of the digitized scene. A remote inspection mode is selected when the proximity condition is not satisfied, and an onsite inspection mode is selected when the proximity condition is satisfied. The processing circuitry is configured to cause the user device to render the digitized scene on the display according to the selected display mode. The remote inspection mode renders a full 3D model, and the onsite inspection mode renders a lightweight representation of the digitized scene. A relative position of the user device in relation to the digitized scene is marked in the lightweight representation. The lightweight representation is derived from the full 3D display model. The relative position of the user device is given by the location data of the user device and the location data of the digitized scene.

[0014] According to a third aspect there is presented a computer program for selecting a display mode for an application visualizing a digitized scene. The computer program comprises computer code which, when run on processing circuitry of a user device that comprises a display, causes the user device to perform actions. One action comprises the user device to obtain a 3D representation of the digitized scene, location data of the digitized scene, and location data of the user device. One action comprises the user device to select the display mode based on whether the location data of the user device satisfies a proximity condition with respect to the location data of the digitized scene. A remote inspection mode is selected when the proximity condition is not satisfied, and an onsite inspection mode is selected when the proximity condition is satisfied. One action comprises the user device to render the digitized scene on the display according to the selected display mode. The remote inspection mode renders a full 3D model, and the onsite inspection mode renders a lightweight representation of the digitized scene. A relative position of the user devicein relation to the digitized scene is marked in the lightweight representation. The lightweight representation is derived from the full 3D display model. The relative position of the user device is given by the location data of the user device and the location data of the digitized scene.

[0015] According to a fourth aspect there is presented a computer program product comprising a computer program according to the third aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium.

[0016] Advantageously, these aspects enable the proper display mode to be used for the given operational context (as defined by whether the proximity condition is satisfied or not).

[0017] Advantageously, these aspects thereby improve the user experience when using applications that visualize a digitized scene, both in terms of remote inspection and onsite inspection.

[0018] Advantageously, these aspects improve the operational efficiency for both the remote inspection mode and the onsite inspection mode.

[0019] Advantageously, the rendering of a lightweight representation in the onsite inspection mode enables the computational load on the user device to be reduced compared to rendering the full 3D model. Hence, the computational efficiency can be improved. This is particularly advantageous for user devices with limited processing power or battery capacity.

[0020] Advantageously, the marking of the relative position of the user device within the lightweight representation facilitates accurate spatial orientation and navigation, leveraging the integration of location data for precise positioning.

[0021] Advantageously, these aspects enable the appropriate display mode to be dynamically selected based on a proximity condition, ensuring that device resources and rendering processes are aligned with the operational context, whether onsite or remote.Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0022] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0025] Fig. 1 schematically illustrates a digitized scene according to an embodiment;

[0026] Figs. 2 and 3 schematically illustrate lightweight representations of a digitized scene according to embodiments;

[0027] Fig. 4 is a block diagram of a system according to an embodiment;

[0028] Fig. 5 is a block diagram of a user device according to an embodiment;

[0029] Fig. 6 is a flowchart of methods according to embodiments;

[0030] Fig. 7 schematically illustrates a lightweight representation of a digitized scene according to an embodiment;

[0031] Fig. 7 schematically illustrates a set-up for determining optimal discrete-valued position and optimal heading according to an embodiment;

[0032] Fig. 9 is a schematic diagram showing structural units of a user device according to an embodiment; and

[0033] Fig. 10 shows one example of a computer program product comprising computer readable storage medium according to an embodiment.DETAILED DESCRIPTION

[0034] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0035] Fig. 1 is a schematic illustration of a digitized scene 100. The digitized scene 100 corresponds to physical scene. In the illustrative example of Fig. 1, the physical scene is a corridor in an indoor industrial environment, where computer equipment 130a, 130b is placed.

[0036] The 3D scene geometry of the digitized scene can be acquired by employing scanning technologies that capture the underlying physical scene in detail. The process of scanning the physical scene to acquire 3D scene geometry may involve systematically moving a scanning device through various positions within the space to ensure comprehensive coverage. Scanning devices such as visual cameras or LiDAR sensors can be mounted on mobile platforms, handheld rigs, or tripods and be positioned at multiple locations. Visual cameras record images that can be processed to extract depth information and reconstruct 3D structures. LiDAR, on the other hand, is based on using laser pulses to measure distances to objects in the physical scene, creating a detailed 3D pointcloud that represents the spatial geometry of the scene. At each scanning position, the scanning device captures data by recording images or emitting laser pulses, collecting detailed spatial information about the surroundings. To achieve a complete representation of the physical scene, the scanning device is moved incrementally, with overlapping fields of view between successive positions. This overlap ensures that gaps in data are minimized and that the individual scans can be seamlessly stitched together during post -processing. In some cases, automated systems or robotic platforms are used to navigate pre-defined paths, while in others, manual operation guides the device to critical areas. The captured data is subsequently aligned and merged to create a unified 3D model of the environment.Computational methods can be used to process the data collected by the scanning devices to generate comprehensive 3D representations, such as 3D pointclouds, 3D meshes, or continuous volumetric formats like NeRFs.

[0037] As noted above, there is still a need for improved scene visualization applications. In more detail, as disclose above, there are different needs for the scene visualization applications, depending on whether the scene visualization application is to be used for onsite inspection or remote inspection.

[0038] The embodiments disclosed herein in particular relate to techniques for selecting a display mode for an application that is visualizing a digitized scene 100. In order to obtain such techniques, there is provided a user device, a method performed by the user device, a computer program product comprising code, for example in the form of a computer program, that when run on a user device, causes the user device to perform the method.

[0039] According to at least some of the herein disclosed embodiments, the display mode of the application that is visualizing the digitized scene 100 is switched between a remote inspection mode and an onsite inspection mode, depending on whether some proximity condition is fulfilled or not. As will be further disclosed below, the proximity condition pertains to whether the user device is located in the physical scene corresponding to the digitized scene or not.

[0040] As will be further disclosed below, in the remote inspection mode, a full 3D model of the digitized scene is rendered, for example to provide a 360-degree panoramic view of the scene, with Skybox type navigation between positions at which the scanning of the physical scene was performed. In general terms, the remote inspection mode is configured to facilitate navigation in the digitized scene, as well as to provide close inspection, measurement, and labeling functionality. In this display mode, the user device requires access only to the full 3D model. An example of a digitized scene as rendered on the display of the user device according to the remote inspection mode is provided in Fig. 1. Accordingly, the digitized scene is exposed for user interaction in terms of navigation, measurement, and labeling. The underlying 3D pointcloud is not directly displayed on the display but is a foundation for all user interaction. A position 120 is indicated on an inlay map no in the lower right-hand corner of the digitized scene. This position 120 shows the current scanning position (based onwhich the current view of the digitized scene is rendered) and not the true user device position.

[0041] As will be further disclosed below, in the onsite inspection mode, a lightweight representation of the digitized scene is rendered, for example in terms of an area map, or a floor plan, with the current user device position indicated in the digitized scene. In general terms, the onsite inspection mode is configured to facilitate navigation in the underlying physical scene. A map capturing the entirety of the physical scene could here be used, to support user’s need for global positioning. A first example of a lightweight representation 200 of the digitized scene as rendered on the display of the user device according to the onsite inspection mode is provided in Fig. 2. In Fig. 2 a lightweight representation 200 of the digitized scene is provided where the digitized scene is represented in its entirety. The lightweight representation 200 in Fig. 2 is a less detailed 3D and zoomed-out 3D display model compared to the full 3D model rendered in the remote inspection mode in Fig. 1. The location 210 of the user device in the physical scene is indicated in the lightweight representation 200. A compass rose indicating the cardinal directions north (N), south (S), east (E), and west (W) is also provided in the lightweight representation 200. A second example of a lightweight representation 300 of the digitized scene as rendered on the display of the user device according to the onsite inspection mode is provided in Fig. 3. Also in the lightweight representation 300 the location 310 of the user device in the physical scene is indicated. Doors have been indicated as openings 320a, 320b, and the computer equipment as rectangular boxes 330a, 330b. The lightweight representation 300 is a two-dimensional (2D) map of the digitized scene, as viewed from above. As in the lightweight representations 200, a compass rose indicating the cardinal directions N, S, E, and W is also provided in the lightweight representation 300. Both lightweight representations 200, 300 are derived from the full 3D display model.

[0042] Reference is next made to Fig. 4. Fig. 4 is a block diagram of a system 400. The system 400 comprises a user device 410, a server 420 and a radio equipment 430. The user device 410 comprises a display 415 and is located in a physical scene 440. The server 420 holds (pre-built) 3D representations of digitized scenes. The radio equipment 430 represents a device by means of which the absolute location of the user device 410 in the physical scene 440 can be established. Examples of such radioequipment 430 and how the absolute location of the user device 410 in the physical scene 440 can be established will be disclosed below.

[0043] In Fig. 5 is provided a block diagram of a user device 500 according to an embodiment. The user device 500 comprises a number of operational blocks 510, 520, 530. The operational blocks 510, 520, 530 maybe implemented in hardware or software, or using a combination of software and hardware. An obtainer block 510 is configured to obtain information, such as a 3D representation of the digitized scene 100, location data of the digitized scene 100, and location data of the user device 410, 500. A display mode selector block 520 is configured to selects the display mode for the display 415. A Tenderer block 530 is configured to render the digitized scene 100 on the display 415 according to the selected display mode. Further aspects of the operational blocks 510, 520, 530 will be disclosed below.

[0044] Fig. 6 is a flowchart illustrating embodiments of methods for selecting a display mode for an application visualizing a digitized scene 100. The methods are performed by the user device 410, 500. The user device comprises a display 415. The methods are advantageously provided as computer programs 1020.

[0045] S102: The user device 410, 500 obtains a 3D representation of the digitized scene 100, location data of the digitized scene 100, and location data of the user device 410, 500. Step S102 maybe implemented by the obtainer block 510.

[0046] In some aspects, the remote inspection mode represents the default display mode. The onsite inspection mode is selected when it can be determined that the user device has entered the physical scene corresponding digitized area. As will be disclosed below, it can be possible for the user of the user device to override the selection of display mode.

[0047] S104: The user device 410, 500 selects the display mode based on whether the location data of the user device 410, 500 satisfies a proximity condition with respect to the location data of the digitized scene 100. A remote inspection mode is selected when the proximity condition is not satisfied. An onsite inspection mode is selected when the proximity condition is satisfied. Step S104 maybe implemented by the display mode selector block 520.S106: The user device 410, 500 renders the digitized scene 100 on the display 415 according to the selected display mode. Step S106 may be implemented by the renderer block 530.

[0048] The remote inspection mode renders a full 3D model, and the onsite inspection mode renders a lightweight representation 200, 300 of the digitized scene 100. A relative position of the user device 410, 500 in relation to the digitized scene is marked in the lightweight representation 200, 300. The lightweight representation 200, 300 is derived from the full 3D display model. The relative position of the user device 410, 500 is given by the location data of the user device 410, 500 and the location data of the digitized scene 100.

[0049] Embodiments relating to further details of selecting a display mode for an application visualizing a digitized scene 100 as performed by the user device 410, 500 will now be disclosed with continued reference to Fig. 6.

[0050] Further aspects of switching between the different display modes will be disclosed next.

[0051] As disclosed above, an onsite inspection mode is selected when the proximity condition is satisfied. However, it is envisioned that there may be situations where the remote inspection mode is to be used for rendering the digitized scene 100 on the display 415 also when the proximity condition is satisfied. Hence, in some embodiments, the user device 410, 500 is configured to perform (optional) step S106-2 as part of rendering the digitized scene 100 on the display 415 in step S106.

[0052] S106-2: The user device 410, 500 switches from rendering the digitized scene 100 on the display 415 in the onsite inspection mode to rendering the digitized scene 100 on the display 415 in the remote inspection mode. Step S106-2 may be implemented by the renderer block 530.

[0053] This switching, could, for example, be based on user input as received from a user of the user device 410, 500 as the user is using the user device 410, 500. One trigger for the user device 410, 500 to perform step S106-2 could thus be the reception of user input from the user of the user device 410, 500. In this way, the user of the user device 410, 500 is enabled to first be presented with the onsite inspection mode whenthe user device 410, 500 is at the location of the digitized scene 100 and then presented with the remote inspection mode.

[0054] Further, in this respect, the user device 410, 500 maybe configured to, upon having made the switch from rendering the digitized scene 100 on the display 415 in the onsite inspection mode to rendering the digitized scene 100 on the display 415 in the remote inspection mode, select the proper viewport / heading for the remote inspection mode that is the best match to the location of the user in the physical scene 440. In particular, in some embodiments, the user device 410, 500 is configured to perform (optional) step S106-2-2 as part of step S106-2.

[0055] S106-2-2: The user device 410, 500 maps an absolute position of the user device 410, 500 in the physical scene 440 corresponding to the digitized scene 100 to a scanner position and heading in the 3D representation. The full 3D display model is rendered from the scanner position and heading. Step S106-2-2 may be implemented by the renderer block 530.

[0056] In general terms, scanner positions are the 3D positions at which the scanner was placed to create the 3D model of the scene. Therefore, the scanner positions represent centers of cubemaps (or SkyBoxes) that can be used to visualize the digitized scene 100 and for navigation in the digitized scene 100 in the remote inspection mode. In some embodiments, the scanner position and heading are therefore selected from a plurality of scanner positions and headings in the 3D representation. The absolute position of the user device 410, 500 is mapped to the scanner position and heading for which a scene geometry condition with respect to the 3D representation of the digitized scene 100 to be rendered on the display 415.

[0057] An automatic switch back to the onsite inspection mode can then be made if user device 410, 500 is still present in the physical scene 440 and changes its location by more than some threshold distance. This could help the user to regain orientation in the physical scene 440 with respect to the digitized scene 100, and vice versa. Hence, in some embodiments, the user device 410, 500 is configured to perform (optional) step S106-4 responsive to detecting that the user device 410, 500 has moved more than a predefined threshold distance since the switching in step S106-2 was performed.S106-4: The user device 410, 500 switches from rendering the digitized scene 100 on the display 415 in the remote inspection mode to rendering the digitized scene 100 on the display 415 in the onsite inspection mode. Step S106-4 maybe implemented by the Tenderer block 530.

[0058] As disclosed above, the display mode for rendering the 3D representation of the digitized scene 100 is selected based on whether the location data of the user device 410, 500 satisfies a proximity condition with respect to the location data of the digitized scene 100. Further aspects of the proximity condition will be disclosed next.

[0059] In some aspects, the proximity condition is fulfilled when the user device 410, 500 is at the location of the digitized scene 100. In particular, in some embodiments, the proximity condition is fulfilled when the user device 410, 500 is within a predefined proximity threshold distance to the digitized scene 100. Likewise, the proximity condition is not fulfilled when the user device 410, 500 is outside the predefined proximity threshold distance to the digitized scene 100.

[0060] For this purpose, the location of the user device 410, 500 might be established in relation to the digitized scene 100. In particular, determining whether the user device 410, 500 is within the predefined proximity threshold distance to the digitized scene 100 or not may comprises detecting whether or not the user device 410, 500 is present in a physical scene 440 corresponding to the digitized scene 100. Whether or not the user device 410, 500 is present in this physical scene 440 can be determined based on the location data of the user device 410, 500 relative to the location data of the digitized scene 100. The user device 410, 500 is then regarded to be within the predefined proximity threshold distance to the digitized scene 100 when the user device 410, 500 is present in the physical scene 440.

[0061] Further aspects of how the location of the user device 410, 500 can be mapped to the digitized scene 100 will be disclosed next.

[0062] In some embodiments, the location data of the digitized scene 100 is associated with any, or any combination of: a cell site identifier, a base station identifier, a radio equipment identifier, an access point identifier, Global Navigation Satellite System (GNNS) information, associated with at least one radio equipment having a known location in a physical scene 440 corresponding to the digitized scene 100. In this way,the absolute location of the digitized scene 100 can be established. As will be further disclosed below, this knowledge can be used in combination with the absolute location of the user device 410, 500 to establish a relation between the user device 410, 500 and the digitized scene 100, e.g., in terms of where in the digitized scene 100 the user device 410, 500 is located.

[0063] In some embodiments, the location data of the user device 410, 500 is derived from any, or any combination of: cell site information, base station information, radio equipment information, access point information, as received from at least one radio equipment, and / or from GNNS information. This information could be extracted from measurements made by the user device 410, 500 on signals transmitted from a cell site, and / or from a satellite navigation system, etc. In this way, the absolute location of the user device 410, 500 can be established. By comparing the absolute location of the user device 410, 500 to the absolute location of the digitized scene 100, the relative position of the user device 410, 500 in the digitized scene 100 can be established.

[0064] In Fig. 7 is provided an example of a lightweight representation 700 of the digitized scene in terms of a 2D map as viewed from above. In this example, a 2D map of the digitized scene is thus rendered on the display 415 of the user device 410, 500 in the onsite inspection mode. Additionally, the relative location 710 of the user device is also shown together with locations 720a, 720b, 720c of radio equipment with known locations in the physical scene. A compass rose indicating the cardinal directions N, S, E, and W is also provided.

[0065] The location of the user device can be established in relation to the radio equipment. Exploiting that underlying full 3D model (from which the 2D map is derived) and the locations of radio equipment are in the same coordinate system, the user device can be correctly placed in the full 3D model and thus also in the view (i.e., the 2D map) rendered on the display 415 and presented to the user.

[0066] Still further aspects of switching between the different display modes and how the location of the user device 410, 500 can be mapped to the digitized scene 100 will be disclosed next.Aspects of detecting whether the user device is present or not present in the physical scene that corresponds to the digital model will be disclosed next.

[0067] In an offline stage, a mapping, denoted T, between the coordinate system, denoted U, of the full 3D model and the coordinate system, denoted V, in which installed radio equipment is recorded can be created. Further, the full 3D model is associated with location data of the digitized scene 100. In this way a lookup table with a unique mapping between location data of the digitized scene 100 and an identifier of the full 3D model can be established. If the user device determines that the user device is operatively connected to a radio equipment for which the location data is associated with a full 3D model, radio positioning of the user device is initiated. This radio positioning generates the position of the user device in the coordinate system V and then in the coordinate system U, via the mapping T. If the distance between the user device in the coordinate system U and the closest point of the full 3D model is below some threshold distance (e.g., in the order of 2-10 meters, such as 5 meters), then the user device is determined to be in proximity of the digitized scene 100.

[0068] Aspects of positioning the user device with respect to the digitized scene, e.g., selecting viewport / heading for displaying the full 3D mode that is the best match to the location of the user device in the physical scene will be disclosed next.

[0069] When the user device enters the physical scene, its location in 3D space is determined in the coordinate system V and then transformed to the coordinate system U. The coordinate system U is not continuous-valued but refers to a discrete set of points in 3D space. This discrete set of points are the positions where the scanning device was placed. Reference is here made to Fig. 8 which schematically illustrates a set-up for determining the optimal discrete-valued position (i.e., either position 810 or position 820) and optimal heading 850, from the continuous-valued position 830 of the user device provided by radio positioning. The specific continuous -valued position 830 thus needs to be translated to a discrete-valued position that will provide best heading and visualize the correct part of the digital model. For simplicity, the illustration in Fig. 8 is in the form of top-down view 800 of the digitized scene. Each point 810, 820 captures a respective 360-degree view of the digitized scene in the form of a cubemap (i.e., 6 perspective images). For all selected points 810, 820, key points (e.g., SIFT, SURF, ORB, AKAZE, etc.) can be extracted from theircorresponding perspective images. The same type of key points can be extracted from the perspective image captured by the camera of the user device. RAN SAC based key points matching can then be performed to determine the best matching cubemap image. In the setup in Fig. 8, this would be scanner position 810 and a cubemap image with a heading identified by the up arrow 850. The closest discrete position 810 is behind a corner. From this point it will not be possible to render all relevant parts of the full 3D model. Thus, because of the scene geometry (e.g., occlusions), the Euclidean distance alone is not a good criterion. Instead, the M closest scanner positions 810, 820 (where M=2 according to the illustrative example of Fig. 8) can be tested for visual similarity with perspective image captured by the camera of user device. In this way, position 820 and heading 840 maybe selected. This process can be used to generate views of the digitized scene 100 to be rendered on the display 415 according to the remote inspection mode.

[0070] Aspects of switching between the different display modes for enabling user interaction in the remote inspection mode for providing a lightweight representation for onsite navigation in the onsite inspection mode will be disclosed next in terms of two examples.

[0071] In a first example it is assumed that the user device is used for performing measurements in the full 3D model (and hence that the remote inspection mode is used) and that the user device is moved in the physical scene. Views facilitating positioning are generated from the underlying 3D pointcloud but are collapsed into a lightweight representation. The relative position of the thus moved user device is visualized in this representation, as in Fig, 3, but the viewport is determined based on 3D pointcloud analysis. This prevents the position of the user device to be occluded by the scene geometry, e.g., behind a wall. This can be achieved e.g., by checking for existence of 3D points between the 3D position of the user device and the 3D viewpoint from which the lightweight representation is to be rendered on the display 415- In a second example it is assumed that the user device is moved in the physical scene whilst in the onsite inspection mode, and that a switch is made to the remote inspection mode for enabling the user to interact with the full 3D model. In this case the position (x) of the suer device, tracked by radio positioning, needs to be mappedto one of scanner locations (•) and heading, from which the full 3D model could be inspected. This can be done using the procedure outlined above for positioning the user device with respect to the digitized scene.

[0072] Since all operations are performed based on the same full 3D model, returning to remote inspection mode enables the user to perform operations such as path measurement, e.g. measuring the length of a cable captured in digitize scene (by selecting respective end points of the measurement). In other words, the described mechanism for switching visual representations of the full 3D model does not terminate operations with non-persistent variables.

[0073] Further aspects of the onsite inspection mode will be disclosed next.

[0074] In general terms, the onsite inspection mode is configured to facilitate physical world navigation in the physical scene 440 corresponding to the digitized scene 100 using real-time positioning information. Therefore, in some embodiments, the indication of the relative position of the user device 410, 500 is updated as the user device 410, 500 obtains updated location data of the user device 410, 500.

[0075] As disclosed above, the onsite inspection mode renders a lightweight representation 200, 300 of the digitized scene 100. There could be different such lightweight representations 200, 300 of the digitized scene 100. For example, the lightweight representation 200, 300 of the digitized scene 100 could be a 2D map, a less detailed 3D display model, a zoomed-out 3D display model, etc. That is, in some embodiments, the lightweight representation 200, 300 of the digitized scene 100 is any of: an area map of the digitized scene 100, a floor plan of the digitized scene 100, a 3D display model of the digitized scene 100 that is less detailed than the full 3D display model and / or that is a zoomed-out version of the full 3D display model. As already disclosed, the lightweight representation 200, 300 is derived from the full 3D display model, and thus the area map of the digitized scene 100, the floor plan of the digitized scene 100, the 3D display model of the digitized scene 100 that is less detailed than the full 3D display model and / or that is a zoomed-out version of the full 3D display model is / are derived from the full 3D display model.

[0076] In some aspects, in the onsite inspection mode, an area map, floor plan, etc. is provided with an indication of the current position of the user device 410, 500. Thatis, in some embodiments, the relative position of the user device 410, 500 is marked on the area map, or on the floor plan, or in the 3D display model.

[0077] Further, in some aspects, annotations, or labels, are provided in the onsite inspection mode. These annotations, or labels, are associated with objects in the digitized scene 100 and are manually inserted in the full 3D display model based on user input. That is, in some embodiments, the onsite inspection mode comprises manually annotated labels associated with objects in the digitized scene 100.

[0078] Further aspects of the remote inspection mode will be disclosed next.

[0079] In some aspects, the remote inspection mode provides, or comprises, a 360-degree panoramic view of the digitized scene 100. As disclosed above, the remote inspection mode renders a full 3D model, and thus the 360-degree panoramic view of the digitized scene 100 can, for example, be derived from high-fidelity (dense) 3D pointcloud representing the full 3D model.

[0080] There could be different ways to provide the 360-degree panoramic view of the digitized scene 100. In some embodiments, the 360-degree panoramic view is comprised in a skybox image rendering environment of the digitized scene 100. This enables Skybox-type navigation between positions at which the scanning of the digitized scene 100 was performed. In this way, the remote inspection mode facilitates navigation, and / or other type of user interaction, in the full 3D model. Hence, in some embodiments, the remote inspection mode is configured for navigation, measurement taking, and label insertion (i.e., manual insertion of labels, or annotations) in the full 3D model.

[0081] Fig. 9 schematically illustrates, in terms of a number of structural units, the components of a user device 900 according to an embodiment. The user device 900 may be a user equipment, such as a smartphone, a tablet computer, or the like.

[0082] Processing circuitry 910 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 1010 (as in Fig. 9), e.g. in the form of a storage medium 930. The processing circuitry 910 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).Particularly, the processing circuitry 910 is configured to cause the user device 900 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 930 may store the set of operations, and the processing circuitry 910 maybe configured to retrieve the set of operations from the storage medium 930 to cause the user device 900 to perform the set of operations. The set of operations maybe provided as a set of executable instructions.

[0083] Thus, the processing circuitry 910 is thereby arranged to execute methods as herein disclosed. The storage medium 930 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The user device 900 may further comprise a communications (comm.) interface 920 at least configured for communications with other entities, functions, nodes, and devices, as in Fig. 4. As such the communications interface 920 may comprise one or more transmitters and receivers, comprising analogue and digital components. The communications interface 920 may also implement a user interface, and thus comprise a display for rendering the digitized scene according to the selected display mode.

[0084] The processing circuitry 910 controls the general operation of the user device 900 e.g. by sending data and control signals to the communications interface 920 and the storage medium 930, by receiving data and reports from the communications interface 920, and by retrieving data and instructions from the storage medium 930. Other components, as well as the related functionality, of the user device 900 are omitted in order not to obscure the concepts presented herein.

[0085] The user device 900 maybe provided as a standalone device or as a part of at least one further device. Thus, a first portion of the instructions performed by the user device 900 maybe executed in a first device, and a second portion of the of the instructions performed by the user device 900 may be executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the user device 900 may be executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by a user device 900 residing in a cloud computational environment.

[0086] Therefore, although a single processing circuitry 910 is illustrated in Fig. 9 theprocessing circuitry 910 maybe distributed among a plurality of devices, or nodes. The same applies to the computer program 1020 of Fig. 10.

[0087] Fig. 10 shows one example of a computer program product 1010 comprising computer readable storage medium 1030. On this computer readable storage medium 1030, a computer program 1020 can be stored, which computer program 1020 can cause the processing circuitry 910 and thereto operatively coupled entities and devices, such as the communications interface 920 and the storage medium 930, to execute methods according to embodiments described herein. The computer program 1020 and / or computer program product 1010 may thus provide means for performing any steps as herein disclosed.

[0088] In the example of Fig. 10, the computer program product 1010 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 1010 could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 1020 is here schematically shown as a track on the depicted optical disk, the computer program 1020 can be stored in any way which is suitable for the computer program product 1010.

[0089] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

CLAIMS1. A method for selecting a display mode for an application visualizing a digitized scene (100), performed by a user device (410, 500, 900) comprising a display (415), the method comprising:obtaining (S102) a three-dimensional, 3D, representation of the digitized scene (100), location data of the digitized scene (100), and location data of the user device (410, 500, 900);selecting (S104) the display mode based on whether the location data of the user device (410, 500, 900) satisfies a proximity condition with respect to the location data of the digitized scene (100), wherein a remote inspection mode is selected when the proximity condition is not satisfied, and an onsite inspection mode is selected when the proximity condition is satisfied; andrendering (S106) the digitized scene (100) on the display (415) according to the selected display mode, wherein the remote inspection mode renders a full 3D model, and the onsite inspection mode renders a lightweight representation (200, 300) of the digitized scene (100), where a relative position of the user device (410, 500, 900) in relation to the digitized scene is marked in the lightweight representation (200, 300), where the lightweight representation (200, 300) is derived from the full 3D display model, and where the relative position of the user device (410, 500, 900) is given by the location data of the user device (410, 500, 900) and the location data of the digitized scene (100).

2. The method according to claim 1, wherein the proximity condition is fulfilled when the user device (410, 500, 900) is within a predefined proximity threshold distance to the digitized scene (100), and wherein the proximity condition is not fulfilled when the user device (410, 500, 900) is outside the predefined proximity threshold distance to the digitized scene (100).

3. The method according to claim 1 or 2, wherein determining whether the user device (410, 500, 900) is within the predefined proximity threshold distance to the digitized scene (100) or not comprises detecting whether or not the user device (410, 500, 900) is present in a physical scene (440) corresponding to the digitized scene(IOO), based on the location data of the user device (410, 500, 900) relative to the location data of the digitized scene (100), and wherein the user device (410, 500, 900) is within the predefined proximity threshold distance to the digitized scene (100) when the user device (410, 500, 900) is present in the physical scene (440).

4. The method according to any preceding claim, wherein the location data of the digitized scene (100) is associated with any, or any combination of: a cell site identifier, a base station identifier, a radio equipment identifier, an access point identifier, Global Navigation Satellite System information, associated with at least one radio equipment having a known location in a physical scene (440) corresponding to the digitized scene (100).

5. The method according to any preceding claim, wherein the location data of the user device (410, 500, 900) is derived from any, or any combination of: cell site information, base station information, radio equipment information, access point information, as received from at least one radio equipment, and / or from Global Navigation Satellite System information.

6. The method according to any preceding claim, wherein the indication of the relative position of the user device (410, 500, 900) is updated as the user device (410, 500, 900) obtains updated location data of the user device (410, 500, 900).

7. The method according to any preceding claim, wherein the lightweight representation (200, 300) of the digitized scene (100) is any of: an area map of the digitized scene (100), a floor plan of the digitized scene (100), a 3D display model of the digitized scene (100) that is less detailed than the full 3D display model and / or that is a zoomed-out version of the full 3D display model.

8. The method according to claim 7, wherein the relative position of the user device (410, 500, 900) is marked on the area map, or on the floor plan, or in the 3D display model.

9. The method according to any preceding claim, wherein the onsite inspection mode comprises manually annotated labels associated with objects in the digitized scene (100).io. The method according to any preceding claim, wherein the remote inspection mode comprises a 360-degree panoramic view of the digitized scene (100).

11. The method according to claim 10, wherein the 360-degree panoramic view is comprised in a skybox image rendering environment of the digitized scene (100).

12. The method according to any preceding claim, wherein the remote inspection mode is configured for navigation, measurement taking, and label insertion in the full 3D model.

13. The method according to any preceding claim, wherein the method further comprises:switching (S106-2) from rendering the digitized scene (100) on the display (415) in the onsite inspection mode to rendering the digitized scene (100) on the display (415) in the remote inspection mode.

14. The method according to claim 13, wherein said switching comprises:mapping (S106-2-2) an absolute position of the user device (410, 500, 900) in a physical scene (440) corresponding to the digitized scene (100) to a scanner position and heading in the 3D representation, and wherein the full 3D display model is rendered from the scanner position and heading.

15. The method according to claim 14, wherein the scanner position and heading are selected from a plurality of scanner positions and headings in the 3D representation, and wherein the absolute position of the user device (410, 500, 900) is mapped to the scanner position and heading for which a scene geometry condition with respect to the 3D representation of the digitized scene (100) to be rendered on the display (415).

16. The method according to claim 13, 14, or 15, wherein the method further comprises, responsive to detecting that the user device (410, 500, 900) has moved more than a predefined threshold distance since said switching was performed:switching (S106-4) from rendering the digitized scene (100) on the display (415) in the remote inspection mode to rendering the digitized scene (100) on the display (415) in the onsite inspection mode.17- A user device (410, 500, 900) for selecting a display mode for an application visualizing a digitized scene (100), the user device (410, 500, 900) comprising a display (415) and processing circuitry (910), the processing circuitry being configured to cause the user device (410, 500, 900) to:obtain a three-dimensional, 3D, representation of the digitized scene (100), location data of the digitized scene (100), and location data of the user device (410, 500, 900);select the display mode based on whether the location data of the user device (410, 500, 900) satisfies a proximity condition with respect to the location data of the digitized scene (100), wherein a remote inspection mode is selected when the proximity condition is not satisfied, and an onsite inspection mode is selected when the proximity condition is satisfied; andrender the digitized scene (100) on the display (415) according to the selected display mode, wherein the remote inspection mode renders a full 3D model, and the onsite inspection mode renders a lightweight representation (200, 300) of the digitized scene (100), where a relative position of the user device (410, 500, 900) in relation to the digitized scene is marked in the lightweight representation (200, 300), where the lightweight representation (200, 300) is derived from the full 3D display model, and where the relative position of the user device (410, 500, 900) is given by the location data of the user device (410, 500, 900) and the location data of the digitized scene (100).

18. The user device (410, 500, 900) according to claim 17, further being configured to perform the method according to any of claims 2 to 16.

19. A computer program (1020) for selecting a display mode for an application visualizing a digitized scene (100), the computer program comprising computer code which, when run on processing circuitry (910) of a user device (410, 500, 900) comprising a display (415), causes the user device (410, 500, 900) to:obtain (S102) a three-dimensional, 3D, representation of the digitized scene (100), location data of the digitized scene (100), and location data of the user device (410, 500, 900);select (S104) the display mode based on whether the location data of the user device (410, 500, 900) satisfies a proximity condition with respect to the location data of the digitized scene (100), wherein a remote inspection mode is selected when the proximity condition is not satisfied, and an onsite inspection mode is selected when the proximity condition is satisfied; andrender (S106) the digitized scene (100) on the display (415) according to the selected display mode, wherein the remote inspection mode renders a full 3D model, and the onsite inspection mode renders a lightweight representation (200, 300) of the digitized scene (100), where a relative position of the user device (410, 500, 900) in relation to the digitized scene is marked in the lightweight representation (200, 300), where the lightweight representation (200, 300) is derived from the full 3D display model, and where the relative position of the user device (410, 500, 900) is given by the location data of the user device (410, 500, 900) and the location data of the digitized scene (100).

20. A computer program product (1010) comprising a computer program (1020) according to claim 19, and a computer readable storage medium (1030) on which the computer program is stored.

Images