Information processing system, information processing method, and program

Abstract

This information processing system comprises a stereo screen control unit. The stereo screen control unit selects, as appropriate data, stereo video data corresponding to a camera angle closest to a viewing attitude of a viewer, from among a plurality of pieces of stereo video data having different camera angles. The stereo screen control unit generates a billboard by projecting the appropriate data onto a stereo screen.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present invention relates to an information processing system, an information processing method, and a program.

[0002] In virtual live creation and the like, production using 2.5D characters is mainstream and is already in the realm of commoditization. Therefore, as a new production method, it has been proposed to 3D model real-world people and incorporate them into a virtual space. As a 3D modeling method, stereoscopic video technology, which is superior in terms of image quality and production period, can be used.

[0003] Japanese Patent Application Laid-Open No. 2021-177580

[0004] The character video is provided as a billboard. A billboard means a plate with a texture having no thickness. A billboard is generated by projecting stereoscopic video data of a character onto a stereoscopic screen, which is a virtual plane. The stereoscopic video data is video data including a right-eye image and a left-eye image. The right-eye image and the left-eye image mean 2D images visible from the position of the right eye and 2D images visible from the position of the left eye for the same scene.

[0005] The viewer views the right-eye image displayed on the stereoscopic screen with the right eye. The viewer views the left-eye image displayed on the stereoscopic screen with the left eye. Thereby, binocular parallax occurs, and the viewer can view the billboard stereoscopically. The viewer enjoys the virtual world while changing the posture of the head and body. However, when the viewing posture is changed significantly, the character stereoscopically viewed due to binocular parallax becomes a distorted video.

[0006] Therefore, the present disclosure proposes an information processing system, an information processing method, and a program in which distortion is unlikely to occur in the video of a stereoscopically viewed character.

[0007] According to this disclosure, an information processing system is provided, which includes a stereo screen control unit that selects stereo video data corresponding to the camera angle closest to the viewer's viewing posture from among multiple stereo video data with different camera angles as appropriate data, and generates a billboard by projecting the appropriate data onto a stereo screen. Furthermore, according to this disclosure, an information processing method is provided in which the information processing of the information processing system is performed by a computer, and a program is provided in which the computer implements the information processing of the information processing system.

[0008] This is an explanatory diagram of virtual live creation. This is a diagram explaining a method for generating 3D characters using stereo video data. This is a diagram explaining a method for generating 3D characters using stereo video data. This is a diagram showing an example of distortion occurring in character images. This is a diagram showing an example of the configuration of an information processing system. This is a diagram explaining an example of billboard display control. This is a diagram explaining an example of demonstration video generation. This is a diagram explaining an example of demonstration video generation. This is a diagram explaining live creator co-creation. This is a diagram explaining an example of the appropriate display range for character images. This is a diagram explaining the field of view conflict between character images and 3DCG objects. This is a diagram explaining the field of view conflict between character images and 3DCG objects. This is a diagram explaining the drawing order of character images and 3DCG objects. This is a diagram showing an example of the hardware configuration of an information processing system.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] The explanation will proceed in the following order: [1. Virtual Live Creation] [2. 3D Character Generation Method Using Stereo Video Data] [3. Example of Information Processing System Configuration] [4. Video Production Method] [4-1. Demonstration Video] [4-2. Live Creator Co-creation] [4-3. Appropriate Display Range of Character Video] [4-4. Field of View Conflict Between Character Video and 3DCG Objects] [4-5. Drawing Order of Character Video and 3DCG Objects] [5. Others] [6. Example Hardware Configuration] [7. Effects]

[0011] [1. Virtual Live Creation] Figure 1 is an explanatory diagram of virtual live creation.

[0012] Virtual live creation is a service that provides live video (LV) content using 3DCG. In virtual live creation, the mainstream approach is to use 2.5D characters as intellectual property, and this has already become commoditized. Therefore, this disclosure proposes a new method of live creation: incorporating 3D models of real people into live video (LV) content.

[0013] Virtual live creation is based on HMD viewing, which allows for a wide field of view and life-scale display, and also enables MR (Mixed Reality) and VR (Virtual Reality) representation. Furthermore, the 3DoF# method is adopted as the viewing method. This reflects the viewer's AU's motion parallax onto the background, making VR sickness less likely to occur.

[0014] The 3DoF# method refers to a 3DoF (Degree of Freedom) method that allows for natural head movement, similar to seated viewing. The 3DoF# method is similar to the 6DoF method, but unlike the 6DoF method, the allowed forward / backward, up / down, and left / right movements are smaller. Therefore, when viewing characters stereoscopically using binocular parallax, the image distortion is smaller.

[0015] In the example in Figure 1, a live video (LV) of a music event is produced using computer graphics (CG). The virtual space (VS) is composed of various CG-generated objects (3DCG objects OB). The live video (LV) is displayed from various viewpoints (VP). The creator of the live video (LIVE creator) provides rendered video corresponding to the viewer's (AU) viewpoint (VP) and line of sight. In Figure 1, lines LN1 and LN2 indicate differences in the height of the viewpoint (VP). The height of the viewpoint (VP) corresponds to the height of the viewing seats and viewing posture (e.g., sitting and standing).

[0016] [2. Method for Generating 3D Characters Using Stereo Video Data] Figures 2 and 3 illustrate a method for generating 3D characters SC using stereo video data. In Figure 3, the Z direction represents the depth direction (the direction directly facing the viewer AU's viewpoint VP). The Y direction represents the vertical direction. The X direction represents the direction perpendicular to the Z and Y directions (horizontal direction).

[0017] The 3D character SC is displayed as a billboard BB. The billboard BB is a thin, textured board oriented to face the viewer AU's viewpoint VP. By projecting the target person's stereo video data (right-eye image, left-eye image) onto a stereo screen PL, which is set as a virtual plane, a stereoscopic billboard BB is generated. The billboard BB is perceived as a three-dimensional person due to binocular parallax.

[0018] Stereo video data is acquired from various angles. Viewers (AU) can watch live video (LV) while moving their heads or standing up. In this case, if appropriate stereo video data is not selected according to the change in viewpoint (VP), distortion will occur in the image of the 3D character (SC) (character image).

[0019] Figure 4 shows examples of distortion that occur in character images. The left side of Figure 4 shows an example of distortion when the viewpoint VP is shifted upwards. The right side of Figure 4 shows an example of distortion when the viewpoint VP is shifted to the left. In the example on the left side of Figure 4, the viewer AU will be viewing Billboard BB from diagonally above. As a result, the character image will be distorted as if the 3D character SC has been vertically compressed. In the example on the right side of Figure 4, the viewer AU will be viewing Billboard BB from diagonally left. As a result, the character image will be distorted as if the 3D character SC has been horizontally compressed.

[0020] In this disclosure, appropriate stereo video data is selected according to changes in the viewer's (AU) viewing posture in order to suppress such distortion. For example, in this disclosure, among multiple stereo video data with different camera angles, the stereo video data corresponding to the camera angle closest to the viewer's (AU) viewing posture is selected as the appropriate data. The billboard BB is generated using the appropriate data. Viewing posture is represented by the position of the viewer's (AU) viewpoint and the direction of their line of sight LS. By generating the billboard using the appropriate data, character images with less distortion are provided.

[0021] For example, low-angle VP Ａ For AU viewers viewing from a low angle, stereo video data shot from a low position is selected as appropriate data. Ｂ For viewers (AU) watching from a high position, stereo video data shot from a high position is selected as appropriate data. The same applies to differences in viewing position in the left-right direction. For example, for viewers (AU) watching from the right, stereo video data shot from the right is selected as appropriate data. For viewers (AU) watching from the left, stereo video data shot from the left is selected as appropriate data.

[0022] The viewing posture of the viewer (AU) and the camera angle of the stereo video data do not necessarily match. To accurately reproduce the posture of the target person, it is preferable to position Billboard BB in an orientation corresponding to the camera angle at the time of filming. However, with this method, if the viewing posture differs from the camera angle, the viewer (AU) will view Billboard BB from an oblique angle. Although the difference between the camera angle at the time of filming and the viewing posture is kept to a minimum, this difference may cause slight distortion in the character image.

[0023] Therefore, in this disclosure, the orientation of the billboard BB is adjusted to match the orientation of the line of sight LS. For example, the billboard BB rotates in the yaw direction to match the orientation of the line of sight LS so that it faces the front of the viewer AU. The rotation of the billboard BB is achieved by the rotation of the stereo screen PL. For example, the orientation of the stereo screen PL is set so that the vertical plane VI containing the line of sight LS is orthogonal to the stereo screen PL. The rotation axis RA is set on the billboard BB so that the position of the billboard BB in the horizontal plane does not change significantly due to the rotation.

[0024] Billboard BB is perceived stereoscopically when right-eye and left-eye images projected onto a stereo screen PL are viewed by the right and left eyes, respectively. Based on binocular parallax, the distance D from the viewpoint VP where Billboard BB is viewed stereoscopically is limited to a certain range. Therefore, Billboard BB is displayed within a range that satisfies the distance conditions for stereoscopic viewing (the appropriate range). If the viewing position changes, the display position of Billboard BB is adjusted to satisfy the aforementioned distance conditions.

[0025] [3. Example of Information Processing System Configuration] The above processing is realized by an information processing system 1 as shown in Figure 5. Figure 5 is a diagram showing an example of the configuration of the information processing system 1. The information processing system 1 includes a content playback / display device 100, a content distribution server 200, and a content recording medium 300. The content playback / display device 100 plays and displays content.

[0026] For example, the content playback and display device 100 includes an HMD input / output unit 101, an HMD camera image recognition unit 102, a motion recognition unit 103, a biometric information recognition unit 104, an HMD viewpoint movement detection unit 105, an HMD output unit 106, a user action analysis unit 111, a content playback control unit 112, a stereo screen control unit 113, a content output unit 114, a communication unit 121, a storage unit 122, and a media interface (IF).

[0027] The HMD input / output unit 101 acquires user input (hand gestures, voice, etc.) and outputs HMD information other than content (haptic signals, etc.). The HMD camera image recognition unit 102 uses the HMD's built-in camera to detect the viewer AU's gaze (LS) and facial expressions. The motion recognition unit 103 uses motion sensors to detect the viewer AU's whole body movements. The biometric information recognition unit 104 detects biometric information such as the viewer AU's brain waves, heart rate, sweating, and body temperature. The HMD viewpoint movement detection unit 105 detects the viewer AU's viewpoint (VP). The HMD output unit 106 controls the output of video and audio related to the content.

[0028] The user action analysis unit 111 acquires information detected by the HMD input / output unit 101, the HMD camera image recognition unit 102, the motion recognition unit 103, the biometric information recognition unit 104, and the HMD viewpoint movement detection unit 105 as user actions. Based on the user actions, the user action analysis unit 111 generates control information for playing the content.

[0029] The content playback control unit 112 controls content playback based on control information. The stereo screen control unit 113 acquires the viewer AU's viewpoint VP position and line of sight LS direction as the viewer AU's viewing posture. The stereo screen control unit 113 controls the position and orientation of the stereo screen PL based on the viewer AU's viewing posture. The content output unit 114 outputs content data such as video and audio to the HMD output unit 106. The HMD output unit 106 controls the display unit and speakers, etc., to play the data.

[0030] The communication unit 121 communicates with the content distribution server 200. The content distribution server 200 stores the content data. The media interface 123 acquires the content data from the content recording medium 300. The storage unit 122 acquires the content data via the communication unit 121 and the media interface. The storage unit 122 stores the content data and outputs it to the content playback control unit 112.

[0031] Figure 6 illustrates an example of display control for Billboard BB.

[0032] The stereo screen control unit 113 acquires the position of the viewer AU's viewpoint VP and the direction of their line of sight LS as the viewer AU's viewing posture. The stereo screen control unit 113 controls the position and posture of the billboard BB based on the viewer AU's viewing posture. For example, the stereo screen control unit 113 includes a billboard rotation control unit 131, a depth direction delay movement control unit 132, and a viewpoint-specific clip switching control unit 133.

[0033] The memory unit 122 stores multiple stereo video data with different camera angles. Each stereo video data is stored as data (viewpoint-specific clip) linked to the camera angle at the time of shooting. The viewpoint-specific clip switching control unit 133 selects the stereo video data from among the multiple stereo video data that corresponds to the camera angle closest to the viewer AU's viewing posture as the appropriate data. The viewpoint-specific clip switching control unit 133 generates a stereoscopic billboard BB by projecting the appropriate data onto the stereo screen PL, which is set as a virtual plane.

[0034] For example, multiple stereo video data sets include seated stereo video data and standing stereo video data. Seated stereo video data is stereo video data captured at a camera angle corresponding to a seated position. Standing stereo video data is stereo video data captured at a camera angle corresponding to a standing position. In the example in Figure 3, the viewpoint VP Ａ This corresponds to a seated position, and the viewpoint VP Ｂ This corresponds to standing.

[0035] The viewpoint-based clip switching control unit 133 switches the stereo video data used for the billboard BB according to the viewer AU's viewing posture. For example, if the viewpoint-based clip switching control unit 133 detects a viewing posture at a height corresponding to sitting, it generates the billboard BB using the sitting stereo video data as the appropriate data. If the viewpoint-based clip switching control unit 133 detects a viewing posture at a height corresponding to standing, it generates the billboard BB using the standing stereo video data as the appropriate data.

[0036] The billboard rotation control unit 131 adjusts the orientation of the billboard BB to match the direction of the viewer AU's line of sight LS, so that the billboard BB always faces the viewer AU. For example, the billboard rotation control unit 131 monitors the viewer AU's viewing posture and rotates the billboard BB to face the viewer AU in accordance with changes in the yaw direction of the viewing posture. To prevent the position of the billboard BB in the horizontal plane from changing significantly due to the rotation, the billboard rotation control unit 131 rotates the billboard BB using a vertical axis passing through the center of the billboard BB as the rotation axis RA.

[0037] The depth direction delay movement control unit 132 controls the depth direction position of the billboard BB. The depth direction delay movement control unit 132 adjusts the display position of the billboard BB so that the distance conditions for stereoscopic viewing are met. For example, the depth direction delay movement control unit 132 obtains the distance conditions for stereoscopic viewing of the billboard BB based on binocular parallax. The depth direction delay movement control unit 132 sets the display position of the billboard BB to a position within an appropriate range where the distance D from the viewer AU satisfies the distance conditions.

[0038] When the viewer AU moves in the depth direction, the distance D between the viewer AU and the billboard BB may no longer satisfy the distance condition described above. When the distance condition is no longer satisfied due to a change in viewing position, the depth direction delay movement control unit 132 moves the display position of the billboard BB to a position within the newly acquired appropriate range in accordance with the change in viewing position.

[0039] When the movement of the display position is instantaneously performed, the viewer AU may feel discomfort. Therefore, the depth-direction delay movement control unit 132 moves the 3D character SC (billboard BB) while causing the 3D character SC to perform natural movements. For example, when the 3D character SC is a guitar player, it is conceivable to generate an image in which the 3D character slowly retreats while playing the guitar. The time (delay time) until the movement of the billboard BB is completed can be arbitrarily set by the system developer.

[0040] In the above example, the movement of the billboard BB has been described. However, situations such as there being no place to move the 3D character SC or being unable to move the 3D character SC with natural movements within the preset delay time are also conceivable. In this case, instead of moving the display position of the billboard BB, the depth-direction delay movement control unit 132 can also stop the display of the billboard BB.

[0041] [4. Video production method] [4-1. Demonstration video] FIGS. 7 and 8 are diagrams for explaining an example of generating a demonstration video DV.

[0042] The live creator can create not only the live video LV viewed from the viewpoint VP of the viewer AU but also the demonstration video DV viewed from an arbitrary viewpoint. In the examples of FIGS. 7 and 8, a video approaching the stage from a distance is provided. The stereoscopic screen control unit 113 can switch the display mode of the billboard BB between 2D display and 3D display based on the distance from the 3D character SC on the stage.

[0043] For example, in a video showing the entire stage from a distance (see FIG. 7), the 3D character SC on the stage is displayed small. Since it is difficult to feel a sense of three-dimensionality for an object with a small size, there is no need to perform 3D display. Therefore, the stereoscopic screen control unit 113 can project only one of the right-eye video and the left-eye video onto the stereoscopic screen PL and display the 3D character SC in 2D.

[0044] In the video showing the area near the stage (see Fig. 8), the 3D character SC is displayed large. For large-sized objects, it is necessary to perform 3D display because it is easier to feel a sense of three-dimensionality. For example, the stereoscopic screen control unit 113 acquires the position of the viewpoint VP and the direction of the line of sight LS specified by the live creator as user viewpoint information. The stereoscopic screen control unit 113 selects stereoscopic video data corresponding to the camera angle closest to the user viewpoint information as appropriate data. The stereoscopic screen control unit 113 generates the billboard BB using the stereoscopic video data selected as the appropriate data.

[0045] The stereoscopic video data acquired in advance is a video close to the viewpoint VP of the viewer AU. There is little stereoscopic video data viewed from the viewpoint VP used in the demonstration video DV. Therefore, there is a possibility of using, in the demonstration video DV, a video of a viewpoint VP that has not been photographed at all (such as a viewpoint that wraps around behind the 3D character SC).

[0046] In this case, it is conceivable to generate a video of an arbitrary viewpoint using, for example, a generative AI (Artificial Intelligence). For example, the stereoscopic screen control unit 113 applies the generative AI to the stereoscopic video data to generate a video of the object indicated by the billboard BB as seen from the position of the viewpoint VP and the direction of the line of sight LS specified by the live creator. Thereby, a demonstration video DV with less sense of incongruity can be provided.

[0047] In the above description, an example in which the live creator creates the demonstration video DV has been described. However, the demonstration video DV may be created not only by the user (live creator) who produces the live video LV but also by a user who is not involved in the production of the live video LV.

[0048] [4-2. Live Creator Co-creation] Fig. 9 is a diagram for explaining live creator co-creation.

[0049] Live Creator Co-creation refers to a process where multiple Live Creators (CRs) collaborate, each taking on a specific role, to produce a single live video (LV) for live streaming or real-time content creation. In the example shown in Figure 9, five CRs are displayed as avatars on the UI screen for generating the live video. Each Live Creator can communicate with other Live Creators while independently performing their respective roles within the UI. Possible roles include the following:

[0050] (A) Design and drawing of the shape and color of the stage (B) Design and drawing of the color and shape of the lights and beams for lighting (C) Design and drawing of the shape, color, texture and sound of the speakers (D) Design and drawing of the shape and color of the structures that will serve as the performers' pathways (E) Design and drawing of the camera work pathways

[0051] [4-3. Appropriate Display Range for Character Images] Figure 10 is a diagram illustrating an example of the appropriate display range for character images.

[0052] Figure 10 shows a view of a 3D character SC moving on a stage, seen from behind. This image can be used, for example, in a demonstration video DV showing a live venue from behind the 3D character SC. Alternatively, this image can be used in game footage where the 3D character SC can be controlled with a controller. The rendering viewpoint VP is set at a certain distance D away from the 3D character SC.

[0053] The 3D character SC is viewed stereoscopically through binocular parallax. The 3D character SC moves around the stage. If the viewpoint VP is fixed, the distance between the 3D character SC and the viewpoint VP increases, making it impossible to view the 3D character SC stereoscopically. Therefore, the distance D between the 3D character SC and the viewpoint VP is set to an appropriate range based on the distance conditions at which the 3D character SC can be viewed stereoscopically. When the 3D character SC moves around the stage, the viewpoint VP also moves while maintaining a constant distance D from the 3D character SC.

[0054] [4-4. Visual Field Conflict Between Character Images and 3DCG Objects] Figures 11 and 12 illustrate the visual field conflict between character images and 3DCG objects OB.

[0055] Visual rivalry refers to the phenomenon where, when a 3D CG object OB and a billboard BB are placed close together, the billboard BB becomes difficult to perceive in 3D. In the example in Figure 11, a 3D character SC is peeking out from behind a door. The face of the 3D character SC is displayed close to the door. The door is a 3D CG object OB created with computer graphics. Therefore, the face area enclosed by the dotted line becomes difficult to perceive in 3D. For this reason, it is preferable to restrict the positions of the billboard BB and the 3D object OB to a range where visual rivalry does not occur.

[0056] For example, the stereo screen control unit 113 acquires the conditions for the occurrence of visual rivalry due to the distance between the billboard BB and the 3DCG object OB. The stereo screen control unit 113 restricts the display position of the 3DCG object OB to a range where no visual rivalry occurs with the billboard BB. In the example in Figure 11, control can be performed such as widening the gap between the 3D character SC and the door depending on how open the door is.

[0057] In the example in Figure 12, a 3D character SC appears among a crowd holding penlights. The penlights are 3D CG objects OB created using computer graphics. The stereoscopic effect of the 3D character SC may be impaired in areas close to the penlights. Therefore, it is possible to pre-calculate that penlights should not be placed close to the 3D character SC. Alternatively, penlights placed close to the 3D character SC (indicated by dotted lines) can be hidden.

[0058] [4-5. Drawing Order of Character Images and 3DCG Objects] Figure 13 is a diagram illustrating the drawing order of character images and 3DCG object OB.

[0059] Unlike CG-created 3D character SCs, which are viewed stereoscopically using binocular parallax, 3D character SCs have specific distance conditions for stereoscopic viewing. If a 3D character SC is displayed outside the appropriate range that satisfies these distance conditions, it will be difficult to perceive it as a three-dimensional object. Therefore, in the production of live video (LV) and similar content, the positions of other 3D CG objects (OB) are determined based on the position of the 3D character SC.

[0060] For example, when a 3D character SC moves, the distance between the viewpoint VP and the 3D character SC remains fixed, while the surrounding scenery (other 3D CG objects OB) moves. The 3D character SC is displayed by incorporating a billboard BB into the CG-created scenery. Because the 3D character SC and the 3D CG objects OB are not rendered together, the rendering order between them becomes an issue.

[0061] If the drawing order of the 3D character SC and other 3DCG objects OB is fixed, the image will look unnatural when the 3D character SC moves. Therefore, the stereo screen control unit 113 determines the drawing order of the 3D character SC (billboard BB) and 3DCG objects OB so that the object closer to the viewing position is drawn first.

[0062] In the example in Figure 13, a 3D character SC is depicted opening a door floating in mid-air and entering another dimension. Before opening the door, the 3D character SC is displayed in front of the door. Therefore, the rendering order is the 3D character SC in the foreground and the door in the background. When the 3D character SC enters the other dimension, the 3D character SC is hidden behind the door. Therefore, the rendering order is the door in the foreground and the 3D character SC in the background.

[0063] [5. Other] The above describes the display control of 3D character SC in virtual live creation. The above description mainly focused on the selection of appropriate data based on viewing posture and the posture control of Billboard BB. In the above embodiment, the production of live video LV was considered, but the method disclosed can be applied to other purposes as well. For example, this could apply to scenario-based game videos in which a 3D character SC acts as the host.

[0064] In video production, various controls are required in addition to the display controls described above. For example, in a scenario-based game experience, it is necessary to switch stereo video data according to the scenario. Optimization of sound and other elements is also necessary. In the above embodiment, the live venue is set as a virtual space VS, but adjustments to volume and sound are necessary depending on the size of the live venue.

[0065] For example, Figure 7 shows an example of a large-scale concert venue. In this example, it is desirable to apply a large amount of reverb to the voices of the MC and the artist (vocalist) or to the live music to achieve a live sound with the reverberation characteristic of a large hall. In addition, the sounds of instruments such as drums, guitars, and pianos may be made into three-dimensional sound to give the effect of distance attenuation and sound image localization.

[0066] On the other hand, there are also scenarios where communication, live performances, and acoustic performances take place in small spaces such as conference rooms or living rooms, like fan meetings or home concerts. In such small spaces, it is natural to reduce the amount of reverb and volume of speech and instrumental sounds. Furthermore, localizing the sound image to the speaker or performer can enhance realism.

[0067] In cases where a gamer plays a mini-game like roulette within the same virtual space (VS) as a 3D character SC, it is natural to switch the 3D character SC's reaction depending on the roulette result (type of prize, win, lose, etc.). It is also possible to change the 3D character SC's movements based on the gamer's actions (such as handing over a gift) or biometric information (such as brainwaves, heart rate, sweating, and body temperature). Alternatively, effects such as particles can be generated as needed.

[0068] [6. Hardware Configuration Example] Figure 14 shows an example of the hardware configuration of the information processing system 1.

[0069] Information devices such as the information processing system 1 and the content playback / display device 100 are implemented by a computer 1000 having a configuration such as that shown in Figure 14. The computer 1000 includes a processing circuit 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of the computer 1000 are connected by a bus 1050.

[0070] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0071] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0072] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing system 1 according to the embodiment of this disclosure, which is an example of program data 1450.

[0073] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the processing circuit 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0074] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, or semiconductor memory.

[0075] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (OLED display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device. The display unit 1700 corresponds to the display unit of the HMD provided in the information processing system 1.

[0076] The camera unit 1800 is an interface for the computer 1000 to capture images. The camera unit 1800 corresponds to the built-in camera of the HMD or the HMD camera image recognition unit 102 provided by the information processing system 1. The microphone 1900 is an interface for the computer 1000 to capture sound. The microphone 1900 corresponds to the HMD input / output unit 101 provided by the information processing system 1. The speaker 2000 is an interface for outputting the sound processed by the computer 1000. The speaker 2000 corresponds to the speaker or the HMD output unit 106 provided by the information processing system 1.

[0077] The various components of the computer 1000 are connected by a bus 1050. Each interface does not necessarily have to be located inside the computer 1000; it may be located outside the computer 1000 via a network or the like. Furthermore, each component of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing that is provided within the display unit 1700.

[0078] For example, when the computer 1000 functions as an information processing system 1 and content playback / display device 100 according to the embodiment of this disclosure, the processing circuit 1100 of the computer 1000 functions as an HMD input / output unit 101, an HMD camera image recognition unit 102, a motion recognition unit 103, a biometric information recognition unit 104, an HMD viewpoint movement detection unit 105, an HMD output unit 106, a user action analysis unit 111, a content playback control unit 112, a stereo screen control unit 113, a content output unit 114, a communication unit 121, a storage unit 122, and a media interface by executing a program loaded on the RAM 1200.

[0079] Furthermore, the secondary storage device 1400 stores the information processing program and various data related to this disclosure. The processing circuit 1100 reads the program data 1450 from the secondary storage device 1400 and executes it, but as an alternative, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside the computer 1000, but may be located outside the computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0080] [7. Effects] The information processing system 1 has a stereo screen control unit 113. The stereo screen control unit 113 selects the stereo video data corresponding to the camera angle closest to the viewing posture of the viewer AU from among multiple stereo video data with different camera angles as appropriate data. The stereo screen control unit 113 generates a billboard BB by projecting the appropriate data onto the stereo screen PL. In the information processing method of this disclosure, the processing of the information processing system 1 is executed by the computer 1000. The program of this disclosure causes the computer 1000 to implement the processing of the information processing system 1.

[0081] With this configuration, stereo video data with an appropriate camera angle is selected from multiple pre-acquired stereo video data sets to match changes in the viewer's posture, and then displayed in 3D. As a result, distortion is less likely to occur in the stereoscopic 3D character SC image.

[0082] Multiple stereo video data sets include seated stereo video data and standing stereo video data. The seated stereo video data is stereo video data captured at a camera angle corresponding to the seated position. The standing stereo video data is stereo video data captured at the same camera angle corresponding to the standing position. When the stereo screen control unit 113 detects a viewing posture corresponding to the seated position, it uses the seated stereo video data as appropriate data to generate the billboard BB. When the stereo screen control unit 113 detects a viewing posture corresponding to the standing position, it uses the standing stereo video data as appropriate data to generate the billboard BB.

[0083] This configuration provides stereoscopic vision with minimal distortion in both seated and standing positions.

[0084] The stereo screen control unit 113 rotates the billboard BB so that it faces the viewer AU directly, following the change in the yaw direction of the viewing posture.

[0085] With this configuration, Billboard BB always faces the viewer's direction, regardless of their viewing posture. Therefore, image distortion caused by changes in the yaw direction of the viewing posture is less likely to occur.

[0086] The stereo screen control unit 113 rotates the billboard BB using a vertical axis passing through the center of the billboard BB as the rotation axis RA.

[0087] This configuration suppresses significant shifts in the display position of the Billboard BB along the circumference centered on the axis of rotation.

[0088] The stereo screen control unit 113 acquires the distance conditions for Billboard BB to be viewable in 3D based on binocular parallax. The stereo screen control unit 113 sets the display position of Billboard BB to a position within an appropriate range where the distance from the viewer AU satisfies the distance conditions.

[0089] This configuration provides good stereoscopic vision with minimal distortion.

[0090] When the stereo screen control unit 113 detects that the distance condition is no longer met due to a change in the viewer's AU position, it moves the display position of Billboard BB to a newly acquired appropriate range in accordance with the change in viewing position, or stops the display of Billboard BB.

[0091] This configuration makes it less likely for distorted images to be provided due to changes in viewing posture.

[0092] The stereo screen control unit 113 acquires the conditions for the occurrence of visual rivalry due to the distance between the billboard BB and the 3DCG object. The stereo screen control unit 113 restricts the display position of the 3DCG object to a range where no visual rivalry occurs with the billboard BB.

[0093] This configuration makes it less likely for the three-dimensionality of the 3D character SC to be lost.

[0094] The stereo screen control unit 113 determines the drawing order of the billboard BB and 3DCG objects so that objects closer to the viewer AU position are drawn first.

[0095] With this configuration, even if the relative positions of Billboard BB and the 3DCG objects change as the video progresses, inconsistencies in the display of Billboard BB and the 3DCG objects are less likely to occur.

[0096] The stereo screen control unit 113 acquires the position of the viewpoint VP and the direction of the line of sight LS specified by the live creator CR as user viewpoint information. The stereo screen control unit 113 selects the stereo video data corresponding to the camera angle closest to the user viewpoint information as appropriate data.

[0097] With this configuration, stereo video data with an appropriate camera angle corresponding to the user's viewpoint information is displayed in 3D. Therefore, distortion is less likely to occur in the stereoscopic 3D character SC image.

[0098] The stereo screen control unit 113 applies generation AI to the stereo video data to generate an image of the object indicated by the billboard BB as seen from the viewpoint VP and line of sight LS specified by the live creator CR.

[0099] This configuration provides more natural-looking 3D character SC (Scroll Character) images.

[0100] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0101] [Note] The technology can also be configured as follows: (1) An information processing system having a stereo screen control unit that selects stereo video data from among a plurality of stereo video data with different camera angles that corresponds to the camera angle closest to the viewer's viewing posture as appropriate data, and generates a billboard by projecting the appropriate data onto a stereo screen. (2) The plurality of stereo video data include seated stereo video data taken at the camera angle corresponding to a seated position and standing stereo video data taken at the camera angle corresponding to a standing position, wherein the stereo screen control unit generates the billboard using the seated stereo video data as appropriate data when the viewing posture corresponding to a seated position is detected, and generates the billboard using the standing stereo video data as appropriate data when the viewing posture corresponding to a standing position is detected. (3) The information processing system according to (1) or (2) above, wherein the stereo screen control unit rotates the billboard to face the viewer in accordance with the change in the yaw direction of the viewing posture. (4) The information processing system according to (3) above, wherein the stereo screen control unit rotates the billboard using a vertical axis passing through the center of the billboard as the axis of rotation. (5) The information processing system according to any one of (1) to (4) above, wherein the stereo screen control unit acquires distance conditions for stereoscopic viewing of the billboard based on binocular parallax and sets the display position of the billboard to a position within an appropriate range where the distance from the viewer satisfies the distance conditions. (6) The information processing system according to (5) above, wherein when the distance conditions are no longer met due to a change in viewing position, the stereo screen control unit moves the display position of the billboard to a newly acquired position within the appropriate range in accordance with the change in viewing position, or stops the display of the billboard.(7) The information processing system according to any one of (1) to (6) above, wherein the stereo screen control unit acquires the conditions for the occurrence of visual rivalry due to the distance between the billboard and the 3DCG object, and restricts the display position of the 3DCG object to a range in which no visual rivalry occurs with the billboard. (8) The information processing system according to (7) above, wherein the stereo screen control unit determines the drawing order of the billboard and the 3DCG object such that the object closer to the viewing position is drawn first. (9) The information processing system according to any one of (1) to (8) above, wherein the stereo screen control unit acquires the position of the viewpoint and the direction of the line of sight specified by the user as user viewpoint information, and selects the stereo video data corresponding to the camera angle closest to the user viewpoint information as the appropriate data. (10) The information processing system according to (9) above, wherein the stereo screen control unit applies generation AI to the stereo video data to generate an image of the object indicated by the billboard as seen from the viewpoint position and the direction of the line of sight specified by the user. (11) A computer-based information processing method comprising: selecting stereo video data from among a plurality of stereo video data with different camera angles that corresponds to the camera angle closest to the viewer's viewing posture as appropriate data; and generating a billboard by projecting the appropriate data onto a stereo screen. (12) The plurality of stereo video data include seated stereo video data captured at a camera angle at a height corresponding to a seated position, and standing stereo video data captured at a camera angle at a height corresponding to a standing position, wherein the billboard generation process generates the billboard using the seated stereo video data as appropriate data when the viewing posture at a height corresponding to a seated position is detected; and generates the billboard using the standing stereo video data as appropriate data when the viewing posture at a height corresponding to a standing position is detected.(13) The information processing method according to (11) or (12) above, comprising rotating the billboard so that it faces the viewer in accordance with the change in the yaw direction of the viewing posture. (14) The information processing method according to (13) above, wherein the rotation of the billboard comprises rotating the billboard with a vertical axis passing through the center of the billboard as the axis of rotation. (15) The information processing method according to any one of (11) to (14) above, comprising obtaining a distance condition for stereoscopic viewing of the billboard based on binocular parallax, and setting the display position of the billboard to a position within an appropriate range where the distance from the viewer satisfies the distance condition. (16) The information processing method according to (15) above, wherein, when the distance condition is no longer met due to a change in viewing position, the display position of the billboard is moved to a newly acquired position within the appropriate range in accordance with the change in viewing position, or the display of the billboard is stopped. (17) An information processing method according to any one of (11) to (16) above, comprising: obtaining the conditions for the occurrence of visual rivalry due to the distance between the billboard and the 3DCG object, and restricting the display position of the 3DCG object to a range in which no visual rivalry occurs with the billboard. (18) An information processing method according to (17) above, comprising: determining the drawing order of the billboard and the 3DCG object such that the object closer to the viewing position is drawn first. (19) An information processing method according to any one of (11) to (18) above, comprising: obtaining the position of the viewpoint and the direction of the line of sight specified by the user as user viewpoint information, and selecting the stereo video data corresponding to the camera angle closest to the user viewpoint information as the appropriate data. (20) A program that causes a computer to generate a billboard by selecting the stereo video data corresponding to the camera angle closest to the viewer's viewing posture from among a plurality of stereo video data with different camera angles as the appropriate data, and projecting the appropriate data onto a stereo screen.

[0102] 1. Information Processing System 113. Stereo Screen Control Unit AU: Viewer BB: Billboard LS: Line of Sight OB: 3DCG Object PL: Stereo Screen RA: Rotation Axis VP: Viewpoint

Claims

1. An information processing system comprising: a stereo screen control unit that selects stereo video data corresponding to the camera angle closest to the viewer's viewing posture from among multiple stereo video data with different camera angles as appropriate data, and generates a billboard by projecting the appropriate data onto a stereo screen.

2. The information processing system according to claim 1, wherein the plurality of stereo video data includes seated stereo video data captured at a camera angle corresponding to a seated position and standing stereo video data captured at a camera angle corresponding to a standing position, and the stereo screen control unit generates the billboard using the seated stereo video data as the appropriate data when the viewing posture corresponding to a seated position is detected, and generates the billboard using the standing stereo video data as the appropriate data when the viewing posture corresponding to a standing position is detected.

3. The information processing system according to claim 1, wherein the stereo screen control unit rotates the billboard to face the viewer in accordance with changes in the yaw direction of the viewing posture.

4. The information processing system according to claim 3, wherein the stereo screen control unit rotates the billboard with a vertical axis passing through the center of the billboard as the axis of rotation.

5. The information processing system according to claim 1, wherein the stereo screen control unit obtains distance conditions for the billboard to be viewable in stereo based on binocular parallax, and sets the display position of the billboard at a position within an appropriate range where the distance from the viewer satisfies the distance conditions.

6. The information processing system according to claim 5, wherein the stereo screen control unit, when the distance condition is no longer met due to a change in the viewing position, moves the display position of the billboard to a newly acquired position within the appropriate range in accordance with the change in the viewing position, or stops the display of the billboard.

7. The information processing system according to claim 1, wherein the stereo screen control unit acquires the conditions for the occurrence of visual rivalry due to the distance between the billboard and the 3DCG object, and restricts the display position of the 3DCG object to a range in which no visual rivalry occurs with the billboard.

8. The information processing system according to claim 7, wherein the stereo screen control unit determines the drawing order of the billboard and the 3DCG object so that the object closer to the viewing position is drawn first.

9. The information processing system according to claim 1, wherein the stereo screen control unit acquires the position and direction of the viewpoint specified by the user as user viewpoint information, and selects the stereo video data corresponding to the camera angle closest to the user viewpoint information as the appropriate data.

10. The information processing system according to claim 9, wherein the stereo screen control unit applies generation AI to the stereo video data to generate an image of the object indicated by the billboard as seen from the viewpoint position and direction of line of sight specified by the user.

11. A computer-based information processing method comprising: selecting stereo video data from among multiple stereo video data with different camera angles that corresponds to the camera angle closest to the viewer's viewing posture as appropriate data; and generating a billboard by projecting the appropriate data onto a stereo screen.

12. The information processing method according to claim 11, wherein the plurality of stereo video data includes seated stereo video data captured at a camera angle corresponding to a seated position and standing stereo video data captured at a camera angle corresponding to a standing position, and the billboard generation process generates the billboard using the seated stereo video data as the appropriate data when the viewing posture corresponding to a seated position is detected, and generates the billboard using the standing stereo video data as the appropriate data when the viewing posture corresponding to a standing position is detected.

13. The information processing method according to claim 11, comprising rotating the billboard to face the viewer in accordance with a change in the yaw direction of the viewing posture.

14. The information processing method according to claim 13, wherein the billboard rotation process comprises rotating the billboard with a vertical axis passing through the center of the billboard as the axis of rotation.

15. The information processing method according to claim 11, comprising obtaining distance conditions for stereoscopic viewing of the billboard based on binocular parallax, and setting the display position of the billboard at a position within an appropriate range where the distance from the viewer satisfies the distance conditions.

16. The information processing method according to claim 15, wherein, when the distance condition is no longer met due to a change in the viewing position, the display position of the billboard is moved to a position within the appropriate range newly acquired in accordance with the change in the viewing position, or the display of the billboard is stopped.

17. The information processing method according to claim 11, comprising obtaining the conditions for the occurrence of visual competition due to the distance between the billboard and the 3DCG object, and restricting the display position of the 3DCG object to a range in which no visual competition occurs with respect to the billboard.

18. The information processing method according to claim 17, comprising determining the drawing order of the billboard and the 3DCG object such that the object closer to the viewing position is drawn first.

19. The information processing method according to claim 11, comprising acquiring the position of the viewpoint and the direction of the line of sight specified by the user as user viewpoint information, and selecting the stereo video data corresponding to the camera angle closest to the user viewpoint information as the appropriate data.

20. A program that enables a computer to generate a billboard by selecting, from among multiple stereo video data with different camera angles, the stereo video data corresponding to the camera angle closest to the viewer's viewing position as the appropriate data, and projecting the appropriate data onto a stereo screen.

Images