Information processing system, information processing method, and program

Abstract

An information processing system according to the present invention acquires segmentation information and depth information on the basis of external sensing information. On the basis of the segmentation information, the information processing system performs shielding processing for overlap regions between virtual objects rendered using CG and recognition target objects that have become detection targets for segmentation. On the basis of the depth information, the information processing system performs shielding processing for overlap regions between virtual objects and non–recognition target objects that have not become detection targets for segmentation.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present invention relates to an information processing system, an information processing method, and a program.

[0002] Video see-through (VST) type displays such as head-mounted displays (HMDs) can fuse the real world and the virtual world by synthesizing an image of the real world (VST image) and a CG image. This type of technology is called XR (Cross Reality) and is becoming popular in the entertainment field and the like.

[0003] International Publication No. 2019 / 031015

[0004] In XR, an object in the real world (real object) and an object in the virtual world (virtual object) may interact. For example, this applies to a case where a user extends a hand (real object) toward a virtual object. Since the real object that is the target of interaction is pre-learned, it can be well identified by segmentation. However, for an unknown real object that has not been pre-learned, appropriate extraction cannot be performed. If a CG image is simply drawn in that state, an image in which the front-back relationship between the real object and the virtual object is contradictory is obtained, and the quality of the XR experience deteriorates.

[0005] Therefore, the present disclosure proposes an information processing system, an information processing method, and a program that can provide a high-quality XR experience.

[0006] According to the present disclosure, segmentation information and depth information are acquired based on sensing information of the external world, and based on the segmentation information, masking processing is performed on an overlapping region between a recognition target object that is a detection target of segmentation and a virtual object drawn in CG, and based on the depth information, masking processing is performed on an overlapping region between a non-recognition target object that is not a detection target of the segmentation and the virtual object. An information processing system is provided. Further, according to the present disclosure, an information processing method in which the information processing of the information processing system is executed by a computer, and a program for causing a computer to realize the information processing of the information processing system are provided.

[0007] This figure shows an example of the configuration of an information processing system. This figure shows an example of the configuration of an information processing system. This figure shows an example of occlusion processing. This figure shows an example of occlusion processing. This figure shows an example of setting the neighboring area to be processed. This figure shows variations of the neighboring area. This figure shows variations of the neighboring area. This figure shows an example of the processing flow related to occlusion processing. This figure shows an example of the processing flow related to occlusion processing. This figure shows blurring of the contour of the depth detection area due to noise in depth measurement. This figure shows an example of setting the neighboring area in a scene with multiple objects to be recognized. This figure shows depth-based occlusion processing in the area where interaction takes place. This figure shows an example of the hardware configuration of an information processing system.

[0008] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0009] The explanation will proceed in the following order: [1. Example System Configuration] [2. Occlusion Processing During Image Synthesis] [3. Scope of Depth-Based Occlusion Processing] [4. Processing Flow of Occlusion Processing] [5. Determination of Occlusion Feasibility Considering Depth Measurement Error] [6. Setting Neighborhood Regions in Scenes with Multiple Recognition Targets] [7. Depth-Based Occlusion Processing for Interaction Areas] [8. Example Hardware Configuration]

[0010] [1. Example of System Configuration] Figures 1 and 2 show an example of the configuration of the information processing system 1.

[0011] Information processing system 1 is an XR presentation system that merges the real world and the virtual world by combining VST image RI and CG image VI. For example, information processing system 1 has an information processing device 10 and an image presentation unit 20.

[0012] The video display unit 20 is a VST-type display device worn on the user's head. Examples of this type of display device include HMDs and AR glasses. Figure 1 shows an HMD as an example of the video display unit 20. The video display unit 20 acquires measurement information and video information necessary for processing the recognition of the external world as sensing information. For example, the video display unit 20 has a sensor unit 21 and a display unit 22.

[0013] The sensor unit 21 includes various sensors necessary for acquiring images of the external environment, depth information, and user US posture information. Examples of sensors include an RGB camera, an IMU (Internal Measurement Unit), and a depth sensor. The display unit 22 presents the user US with a composite image SI, which is a composite of images of the external environment (VST image RI) and a CG image VI.

[0014] The information processing device 10 performs external environment recognition processing, CG image VI generation processing, and VST image RI and CG image VI synthesis processing based on sensing information acquired from the sensor unit 21. The information processing device 10 outputs the synthesized image SI obtained by the synthesis processing to the display unit 22. For example, the information processing device 10 has a recognition processing unit 11, a CG drawing unit 12, and a synthesis unit 13.

[0015] The recognition processing unit 11 recognizes the external environment based on sensing information. The recognition processing can be performed using known techniques such as SLAM (Simultaneous Localization and Mapping). Based on the recognition results of the external environment, the recognition processing unit 11 acquires information regarding the position and shape of real objects (physical objects RO) present in the external environment as object information. In the example in Figure 2, information on the user US's hand HD and the physical object RO behind it is acquired. Based on the recognition results of the external environment, the recognition processing unit 11 acquires information regarding the relative position and orientation between the physical object RO and the user US as orientation information.

[0016] The CG rendering unit 12 renders a virtual object VO based on the user US's posture information. The synthesis unit 13 superimposes the CG image VI of the virtual object VO obtained by rendering onto the external image (VST image RI) acquired by the sensor unit 21. The synthesis unit 13 outputs the synthesized image SI, which is a composite of the CG image VI and the VST image RI, to the display unit 22.

[0017] The video display unit 20 and the information processing device 10 may be configured as a single integrated device, or as separate devices. An example of the latter is to provide the video display unit 20 as an HMD and the information processing device 10 as a server.

[0018] [2. Occlusion Processing During Image Synthesis] In XR experiences using HMDs, an image is presented to the user (US) that is a composite of an image of the real world (VST image RI) and an image of the virtual world (CG image VI). In this case, simply superimposing a virtual object VO onto the VST image RI may result in occlusion not being correctly represented. "Correct representation of occlusion" means that the spatial relationship between the real object RO and the virtual object VO from the user's (US) perspective is correctly presented.

[0019] For example, in the example in Figure 2, it is correct to display the virtual object VO behind the hand HD. However, if the virtual object VO is simply superimposed on the VST image RI, the front-to-back relationship between the hand HD and the virtual object VO will be reversed. If the front-to-back relationship between objects is incorrect, the quality of the XR experience will decrease. Therefore, it is necessary to use the hand HD as a mask to obscure a portion of the virtual object VO.

[0020] Figures 3 and 4 show examples of occlusion processing. "Segmentation-based" means occlusion processing based on segmentation. "Depth-based" means occlusion processing based on depth information. Segmentation is a type of image recognition processing using deep learning, which means labeling object classes (types of objects) on a pixel-by-pixel basis. Examples of segmentation include semantic segmentation, instance segmentation, and panoptic segmentation.

[0021] In segmentation-based occlusion methods, color or monochrome images (2D image CI) are used as input to estimate the area where a pre-trained object exists (object area). In the example in Figure 3, a human hand is the pre-trained object and is the target of detection by segmentation. The front-to-back relationship between the object area and the virtual object VO is determined by some method. For example, the real object RO may be placed in front, or the front-to-back relationship with the virtual object VO may be determined using a representative depth value of the object area.

[0022] Based on the determined spatial relationship, occlusion is applied to the real object RO or virtual object VO. For example, if the virtual object VO is behind the real object RO, the object region of the real object RO is used as a mask MK to occlude the virtual object VO. The virtual object VO is then drawn in a position excluding the real object RO. This ensures that the spatial relationship between the real object RO and the virtual object VO is correctly represented.

[0023] This method allows for high-precision acquisition of object regions, enabling clean occlusion where virtual objects VO and real objects RO are clearly separated. However, the real objects RO (recognized objects RT) that are targeted for segmentation detection are limited to those learned through machine learning. Real objects RO that are not targeted for segmentation detection (unrecognized objects NT) cannot be subjected to the same occlusion processing as the recognized objects RT.

[0024] For example, in the example in Figure 4, objects "A", "B", "C", and "D" are arranged from the side closest to user US as viewed from user US's viewpoint VP. The symbol "LS" indicates user US's line of sight. Object "A" represents user US's hand (real object RO). Object "B" represents the real-world camera (real object RO). Object "C" represents the virtual world camera (virtual object VO). Object "D" represents the desk (real object RO).

[0025] The hand is a pre-trained object. Therefore, object "A" is a recognized object RT that is subject to detection by segmentation. The camera and desk are not pre-trained. Therefore, objects "B" and "D" are unrecognized objects NT that are not subject to detection by segmentation.

[0026] In segmentation-based occlusion processing, only the recognized object RT has its spatial relationship with the virtual object VO determined. Therefore, the spatial relationship between object "A" and object "C" is correctly represented. For the unrecognized object NT, the spatial relationship with the virtual object VO is not determined, so the virtual object VO is simply superimposed on the unrecognized object NT. Therefore, the spatial relationship between object "B" and object "C" is not correctly represented (see the upper right diagram in Figure 4).

[0027] In depth-based occlusion methods, the relative positions of real objects (RO) and virtual objects (VO) are determined based on their depth information. Depth information is obtained from depth images (DI) acquired by depth sensors. The object region that will become the mask (MK) is determined based on the depth information. Unlike segmentation-based occlusion methods, this method has no limitations on the target object. However, because the accuracy of depth measurement is not very high, there is a possibility that parts of the relative positions may not be properly represented. Such areas become artifacts and may reduce the quality of the XR experience (see the lower right diagram in Figure 4).

[0028] Both of the occlusion methods described above have their own advantages and disadvantages. Therefore, this disclosure proposes to adaptively switch between these occlusion methods depending on the region. For example, the recognition processing unit 11 acquires segmentation information and depth information based on sensing information from the outside world. The segmentation information includes information on the object class of each pixel that has been labeled based on the segmentation. The depth information includes information on the depth of each pixel.

[0029] Based on the segmentation information, the synthesis unit 13 performs occlusion processing on the overlapping region between the recognized target object RT, which was the subject of segmentation (pre-training), and the virtual object VO, which is rendered by CG. Based on the depth information, the synthesis unit 13 performs occlusion processing on the overlapping region between the unrecognized target object NT, which was not the subject of segmentation, and the virtual object VO.

[0030] In this configuration, highly attention-grabbing real-world objects RO (recognized objects RT) that are the target of interaction are subjected to good segmentation-based occlusion processing, while other real-world objects RO (unrecognized objects NT) are subjected to depth-based occlusion processing. Even unknown real-world objects RO that are not detected by segmentation are subjected to appropriate occlusion processing, thus providing a high-quality XR experience.

[0031] As mentioned earlier, depth-based occlusion processing suffers from artifacts based on the accuracy of depth measurements. However, since the occlusion processing is limited to unrecognized objects NT of low importance, the visual degradation due to artifacts is minimized.

[0032] [3. Scope of Depth-Based Occlusion Processing] Depth-based occlusion processing does not necessarily need to be performed on the entire screen. It is also possible to target only areas where the user US is likely to focus their attention with depth-based occlusion processing. For example, if the object to be recognized RT is an object that interacts with other objects (e.g., a hand HD), the user US is likely to focus their attention on the object to be recognized RT and its vicinity. In such cases, the synthesis unit 13 can selectively perform depth-based occlusion processing on non-recognized objects NT that exist in the vicinity of the object to be recognized RT.

[0033] This configuration limits the scope of depth-based occlusion processing to a narrow range. Generally, the detection accuracy of real-world objects (ROs) based on depth information is lower than that of real-world objects (ROs) based on segmentation. Occluding processing is performed by masking the overlapping portion of real-world objects (ROs) and virtual objects (VOs). If the depth measurement accuracy is low, the contours of real-world objects (ROs) become unclear. As a result, artifacts are more likely to occur in the contour areas. If the processing range based on depth information is narrow, artifacts become less noticeable, and the quality of the XR experience is less likely to be compromised.

[0034] The range of the "neighborhood region" to be processed can be arbitrarily set by the system developer. The "neighborhood region" may be a part of the VST image RI (a planar region) or a three-dimensional region that takes depth into account. Figure 5 shows an example of setting the neighborhood region NA to be processed. In the example in Figure 5, the neighborhood region NA is set as a planar region with a rectangular shape. However, the neighborhood region NA is not limited to this. For example, the neighborhood region NA may be a rectangular, cubic, or rectangular prism region.

[0035] In the example in Figure 5, object "B," which is a real object RO, and object "C," which is a virtual object VO, are placed on a desk. Objects "B" and "C" represent cameras of different designs. User US is holding object "B" in their hand (object "A") and comparing it with object "C." The hand is a trained object (target object RT). The vicinity of object "A," which is the target object RT, is the nearby region NA that will be subjected to depth-based occlusion processing.

[0036] The arrangement of objects on the front side of the desk (the user's viewpoint, VP side) differs depending on the location. At position "K", objects "B" and "C" are placed in order from the viewpoint VP side. At position "L", only object "C" is placed. At position "M", objects "A" and "C" are placed in order from the viewpoint VP side.

[0037] At position "K", depth-based occlusion correctly displays the relative positions of objects "B" and "C". At position "L", object "C" is rendered by CG without considering depth. At position "M", segmentation-based occlusion correctly displays the relative positions of objects "A" and "C".

[0038] The shape of the neighboring region NA is not limited to a rectangle. Furthermore, the neighboring region NA can be a three-dimensional region extending in the depth direction. By making the neighboring region NA a three-dimensional region, the range of depth-based occlusion processing is limited in the depth direction. A narrower occlusion processing range reduces computational load and improves real-time performance. Additionally, if occlusion processing is performed on objects far behind the area of ​​interest of the user (US), which have a low contribution to the XR experience, artifacts may spread unnecessarily. Limiting the occlusion processing range in the depth direction reduces the likelihood of such problems occurring.

[0039] Figures 6 to 8 show variations in the nearby region NA.

[0040] In the example in Figure 6, the neighboring region NA is a circular region. However, the neighboring region NA may also be a spherical region. Making the neighboring region NA a circular or spherical region simplifies its calculation. The neighboring region NA only needs to be a region that encloses the object to be recognized RT. The center of the neighboring region NA can be a position on the object to be recognized RT, or a position in the vicinity of the object to be recognized RT. In the example in Figure 6, the center of the neighboring region NA is set to the position of the centroid of the object to be recognized RT.

[0041] The synthesis unit 13 obtains the object class of each pixel PX from the segmentation information. The synthesis unit 13 extracts the pixel PX that has the object class of the recognized object RT. The synthesis unit 13 uses the region where the extracted pixel PX exists as the mask MK. The synthesis unit 13 performs segmentation-based occlusion processing on the pixel PX of the virtual object VO that overlaps with the mask MK.

[0042] In the example of FIG. 7, the neighborhood area NA is a cubic area. In the example of FIG. 8, the neighborhood area NA is a rectangular parallelepiped area. By setting the neighborhood area NA as a rectangular, cubic or rectangular parallelepiped area, it becomes easier to calculate the neighborhood area NA. The neighborhood area NA is set as an area that encloses the recognition target object RT. Also in the examples of FIGS. 7 and 8, the center of the neighborhood area NA is set at the position of the center of gravity of the recognition target object RT.

[0043] The composition unit 13 can adjust the size of the neighborhood area NA according to the distance from the viewpoint of the user US to the recognition target object RT. According to this configuration, the calculation amount of the occlusion processing is adjusted according to the distance to the recognition target object RT.

[0044] For example, since the peripheral part of the recognition target object RT is likely to attract the user US's line of sight, if the occlusion processing is not correctly performed, it will give the user US a sense of discomfort. When the recognition target object RT is close, the range that the user US pays attention to also becomes wider, so it is preferable to widen the target range (neighborhood area NA) of the occlusion processing. When the recognition target object RT is far away, artifacts are less noticeable, so even if the target range of the occlusion processing is narrowed, the image quality is not significantly impaired. Thus, by adaptively adjusting the target range of the occlusion processing, the calculation load can be reduced while maintaining the quality of the XR experience.

[0045] [4. Processing Flow of Occlusion Processing] FIGS. 9 and 10 are diagrams showing an example of a processing flow related to occlusion processing.

[0046] The composition unit 13 extracts the pixel PX to be the drawing target of the virtual object VO as the target pixel (step S1). If the target pixel is not the drawing target of the virtual object VO (step S1: No), the composition unit 13 does not perform the occlusion processing on the target pixel (step S4) and ends the processing as it is.

[0047] If the target pixel is a rendering target of the virtual object VO (step S1: Yes), the compositing unit 13 determines whether the target pixel is the pixel PX on which the recognized target object RT is displayed (step S2). If the target pixel is the pixel PX on which the recognized target object RT is displayed (step S2: Yes), the compositing unit 13 performs segmentation-based occlusion processing on the target pixel (step S3) and ends the process.

[0048] If the target pixel is not the pixel PX on which the recognized target object RT is displayed (step S2: No), the compositing unit 13 determines whether there is a depth value of another object in the vicinity area NA of the recognized target object RT (step S11).

[0049] If there is a depth value of another object in the target pixel within the vicinity area NA (step S11: Yes), the compositing unit 13 performs depth-based occlusion processing on the target pixel (step S12) and ends the process. If there is no depth value of another object in the target pixel within the vicinity area NA (step S11: No), the compositing unit 13 draws the CG image of the virtual object VO on the target pixel (step S13) and ends the process.

[0050] [5. Determination of Occlusion Feasibility Considering Depth Measurement Error] The measurement result of the depth sensor includes a certain degree of error. Originally, the object area detected by segmentation (segmentation detection area) and the object area detected by the depth sensor (depth detection area) should indicate the existence area of the same object, so it is desirable for them to match. However, if the measurement accuracy of the depth is low, the depth may be measured at a position slightly deviated from the position where the object exists. The depth value measured at the wrong position becomes noise and blurs the contour of the depth detection area.

[0051] Figure 11 shows blurring of the contour of the depth detection area due to noise in the depth measurement. Depth detected outside the segmentation detection area may be recognized as the depth of another object. In this case, depth-based occlusion processing is performed on the pixels PX outside the segmentation detection area. Since depth-based occlusion processing can produce artifacts, it is desirable for the synthesis unit 13 to set some kind of judgment criteria to avoid such unnecessary occlusion processing.

[0052] For example, the synthesis unit 13 obtains the position of the contour of the object RT to be recognized based on the segmentation information. The synthesis unit 13 determines the depth value of the effective depth region as noise if the region in which the effective depth value that can be used to determine the presence or absence of the real object RO exists (effective depth region) is limited to the region very close to the contour, and the depth value of the effective depth region is similar to the depth value of the contour of the object RT to be recognized.

[0053] Effective depth value refers to a minute signal caused by measurement error, unrelated to the presence or absence of a physical RO (Resonance Observation) object. The presence or absence of effective depth value can be determined based on thresholds set by the system developer.

[0054] "The depth value of the effective depth region is similar to the depth value of the contour of the object being recognized" means that there is a continuity between the depth value of the effective depth region and the depth value of the contour of the object being recognized RT to the extent that it can be inferred that the object indicated by the depth value of the effective depth region and the object being recognized RT are the same object. The analogy determination of depth values ​​can be performed based on thresholds set by the system developer.

[0055] According to the above configuration, unnecessary depth-based occlusion processing on the contour of the object RT to be recognized is avoided. As a result, images with fewer artifacts are provided. In addition, since unnecessary occlusion processing is omitted, the processing load is reduced. As a method for suppressing artifacts on the contour of an object, the method described in Patent Document 1 can also be used in combination. In the method of Patent Document 1, visual degradation is suppressed by expanding the contour or changing the transparency.

[0056] [6. Setting the Neighborhood Region in Scenes with Multiple Recognition Targets] Depth-based occlusion processing may be limited to areas of high importance in the XR experience. For example, even if multiple hand HDs exist in the same scene, occlusion processing can be applied only to the vicinity of a specific hand HD that is of high interest to the user (e.g., the user's hand HD). The high-importance area can be determined based on the user's distance and line of sight.

[0057] Figure 12 shows an example of setting the neighboring region NA in a scene where multiple recognition target objects RT exist. In the example in Figure 12, multiple hands are detected as recognition target objects RT. Of these, the hand shown on the right is the user US's hand. User US grasps the real-world camera (real object RO) with their hand and compares it with the virtual-world camera (virtual object VO). Therefore, the vicinity of user US's hand becomes user US's area of ​​interest GA. The synthesis unit 13 can make the area of ​​interest GA the target range for depth-based occlusion processing.

[0058] In the example shown in Figure 12, the user US's attention is focused on one of the user US's hands. However, there are also cases where the user US's attention is focused on multiple objects, such as when the user US is working with both hands. In this case, the vicinity of each object can also be set as the target area for depth-based occlusion processing. For example, the synthesis unit 13 identifies one or more recognized target objects RT from a plurality of simultaneously detected recognized target objects RT based on the distance to the user US or the user US's line of sight. The synthesis unit 13 selectively performs occlusion processing on the identified one or more recognized target objects RT.

[0059] This configuration limits the scope of depth-based occlusion processing to areas where the user's gaze tends to focus. Therefore, it reduces computational load without significantly compromising the quality of the XR experience. This method is particularly effective in scenes where there are many objects to be recognized (RT), leading to increased processing load. The reduced processing load allows for occlusion processing to be performed while maintaining real-time performance.

[0060] [7. Depth-based occlusion processing for areas where interaction takes place] When interaction between objects is an important element of the XR experience, depth-based occlusion processing may be implemented near the objects that will be interacting. Figure 13 shows depth-based occlusion processing for areas where interaction takes place.

[0061] For example, the synthesis unit 13 selectively performs shielding on non-recognized object NT that is interacting with the recognized object RT. The interaction may be in contact with the objects or in a non-contact state where the objects are in close proximity. Examples of the former include grasping and button operation. Examples of the latter include electrical or magnetic interaction.

[0062] In the example shown in Figure 13, the hand (recognized object RT) interacts with the virtual object VO by grasping it with the hand. Known methods such as HOI (Hand Object Interaction) can be used to detect the grasping.

[0063] With this configuration, the scope of occlusion processing is limited to areas where the user's (US) gaze tends to focus due to interaction. Therefore, it is possible to reduce the computational load while maintaining the quality of the XR experience.

[0064] [8. Hardware Configuration Example] Figure 14 shows an example of the hardware configuration of the information processing system 1.

[0065] The information processing system 1 can be implemented by a computer 1000 as shown in Figure 14. The computer 1000 includes a processing circuit 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of the computer 1000 are connected by a bus 1050.

[0066] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0067] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0068] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing system 1 according to the embodiment of this disclosure, which is an example of program data 1450.

[0069] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the processing circuit 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0070] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, or semiconductor memory.

[0071] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (OLED display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device. The display unit 1700 corresponds to the display element of the image presentation unit 20 of this disclosure.

[0072] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.

[0073] For example, when computer 1000 functions as an information processing system 1 according to an embodiment of this disclosure, the processing circuit 1100 of computer 1000 functions as various detection and control units included in the information processing system 1 by executing a program loaded onto RAM 1200. The secondary storage device 1400 stores the information processing program and various data according to this disclosure. The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside computer 1000, but may be located outside computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0074] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0075] [Note] This technology can also be configured as follows: (1) An information processing system that acquires segmentation information and depth information based on sensing information from the outside world, performs occlusion processing on the overlapping region of the recognized object that was detected for segmentation and the virtual object drawn with CG based on the segmentation information, and performs occlusion processing on the overlapping region of the non-recognized object that was not detected for segmentation and the virtual object based on the depth information. (2) The information processing system according to (1) above, wherein the occlusion processing based on the depth information is selectively performed on the non-recognized object that is in the vicinity of the recognized object. (3) The information processing system according to (2) above, wherein the vicinity region is a three-dimensional region that extends in the depth direction. (4) The information processing system according to (2) above, wherein the vicinity region is a circular or spherical region. (5) The information processing system according to (2) above, wherein the vicinity region is a rectangular, cubic, or rectangular prism region. (6) An information processing system according to any one of (2) to (5) above, wherein the size of the neighborhood area is adjusted according to the distance from the user's viewpoint to the object to be recognized. (7) An information processing system according to (2) above, wherein the occlusion processing is selectively performed on the non-recognized object that is interacting with the object to be recognized. (8) An information processing system according to any one of (1) to (7) above, wherein the position of the contour of the object to be recognized is obtained based on the segmentation information, and the region in which an effective depth value that can be used to determine whether or not a real object exists is limited to the region very close to the contour, and the depth value of the region in which(10) A computer-based information processing method comprising: acquiring segmentation information and depth information based on sensing information from the outside world; performing occlusion processing on the overlapping region of a recognized object that has been detected for segmentation and a virtual object drawn by computer graphics based on the segmentation information; and performing occlusion processing on the overlapping region of a non-recognized object that has not been detected for segmentation and the virtual object based on the depth information. (11) The information processing method according to (10) above, further comprising selectively performing the occlusion processing based on the depth information on a non-recognized object that exists in the vicinity of the recognized object. (12) The information processing method according to (11) above, wherein the vicinity region is a three-dimensional region that extends in the depth direction. (13) The information processing method according to (11) above, wherein the vicinity region is a circular or spherical region. (14) The information processing method according to (11) above, wherein the vicinity region is a rectangular, cubic, or rectangular prism region. (15) The information processing method according to any one of (11) to (14) above, wherein the size of the neighborhood area is adjusted according to the distance from the user's viewpoint to the object to be recognized. (16) The information processing method according to (11) above, wherein the occlusion processing is selectively performed on the non-recognized object that is interacting with the object to be recognized. (17) The information processing method according to any one of (10) to (16) above, wherein the position of the contour of the object to be recognized is obtained based on the segmentation information, and the region in which an effective depth value that can be used to determine whether or not there is a real object exists is limited to the region very close to the contour, and the depth value of the region in which (18) The information processing method according to any one of (10) to (17) above, comprising: identifying one or more objects to be recognized from a plurality of objects to be recognized detected simultaneously, based on the distance from the user or the user's line of sight; and selectively performing the occlusion processing on the identified one or more objects to be recognized.(19) A program that enables a computer to perform the following actions: acquire segmentation information and depth information based on sensing information from the outside world; perform occlusion processing of the overlapping region between the recognized object that was detected as a segmentation target and a virtual object drawn with computer graphics based on the segmentation information; and perform occlusion processing of the overlapping region between the non-recognized object that was not detected as a segmentation target and the virtual object based on the depth information.

[0076] 1 Information Processing System 11 Recognition Processing Unit 13 Synthesis Unit NA Nearby Region NT Unrecognized Object RT Recognized Object VO Virtual Object

Claims

1. An information processing system that acquires segmentation information and depth information based on sensing information from the outside world, performs occlusion processing of the overlapping region between recognized objects detected for segmentation and virtual objects rendered by computer graphics based on the segmentation information, and performs occlusion processing of the overlapping region between unrecognized objects that were not detected for segmentation and the virtual objects based on the depth information.

2. The information processing system according to claim 1, wherein the occlusion processing based on the depth information is selectively performed on non-recognized objects that exist in the vicinity of the recognized object.

3. The information processing system according to claim 2, wherein the neighboring region is a three-dimensional region that extends in the depth direction.

4. The information processing system according to claim 2, wherein the neighboring region is a circular or spherical region.

5. The information processing system according to claim 2, wherein the neighboring region is a rectangular, cubic, or rectangular prism region.

6. The information processing system according to claim 2, wherein the size of the nearby area is adjusted according to the distance from the user's viewpoint to the object to be recognized.

7. The information processing system according to claim 2, wherein the shielding process is selectively performed on the non-recognized object that is interacting with the recognized object.

8. An information processing system according to claim 1, wherein the position of the contour of the object to be recognized is obtained based on the segmentation information, and the region in which an effective depth value that can be used to determine whether or not a real object exists is limited to the region very close to the contour, and the depth value of the region in which the value exists is similar to the depth value of the contour, the depth value of the region in which the value exists is determined to be noise.

9. The information processing system according to claim 1, which identifies one or more objects to be recognized from a plurality of objects to be recognized detected simultaneously, based on the distance from the user or the user's line of sight, and selectively performs the occlusion process on the identified one or more objects to be recognized.

10. A computer-based information processing method comprising: acquiring segmentation information and depth information based on sensing information from the outside world; performing occlusion processing of the overlapping region between a recognized object that has been detected for segmentation and a virtual object drawn with computer graphics based on the segmentation information; and performing occlusion processing of the overlapping region between a non-recognized object that was not detected for segmentation and the virtual object based on the depth information.

11. The information processing method according to claim 10, further comprising selectively performing the occlusion processing based on the depth information on non-recognized objects that exist in the vicinity of the recognized object.

12. The information processing method according to claim 11, wherein the neighboring region is a three-dimensional region that extends in the depth direction.

13. The information processing method according to claim 11, wherein the neighboring region is a circular or spherical region.

14. The information processing method according to claim 11, wherein the neighboring region is a rectangular, cubic, or rectangular prism region.

15. The information processing method according to claim 11, further comprising adjusting the size of the nearby area according to the distance from the user's viewpoint to the object to be recognized.

16. The information processing method according to claim 11, further comprising selectively performing the shielding process on a non-recognized object that is interacting with the recognized object.

17. The information processing method according to claim 10, comprising: obtaining the position of the contour of the object to be recognized based on the segmentation information; determining that the region in which an effective depth value that can be used to determine the presence or absence of a real object exists is limited to the region very close to the contour, and determining that the depth value of the region in which the value exists is similar to the depth value of the contour, the depth value of the region in which the value exists is noise.

18. The information processing method according to claim 10, comprising: identifying one or more objects to be recognized from a plurality of objects to be recognized detected simultaneously, based on the distance from the user or the user's line of sight; and selectively performing the occlusion process on the identified one or more objects to be recognized.

19. A program that enables a computer to perform the following actions: acquire segmentation information and depth information based on sensing information from the outside world; perform occlusion processing of the overlapping region between the recognized object detected for segmentation and a virtual object drawn with computer graphics based on the segmentation information; and perform occlusion processing of the overlapping region between the unrecognized object that was not detected for segmentation and the virtual object based on the depth information.

Images