Information processing system, information processing method, and information processing program

Abstract

An information processing system according to the present disclosure comprises: an acquisition unit that acquires motion data of a first subject in a first period, information relating to an imaging device that images the first subject, and motion data of a second subject in a second period; and an output control unit that outputs third time-series data, on the basis of first time-series data in which the motion data of the first subject and the information relating to the imaging device are arranged in a virtual space and second time-series data in which the motion data of the second subject is arranged in the virtual space.

Description

Information Processing System, Information Processing Method, and Information Processing Program

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program.

[0002] Techniques for generating a three-dimensional virtual space using motion capture or an imaging device are known. For example, in the prior art, there is a known technique for viewing three-dimensional objects displayed based on motion capture data of a human or the like moving in a three-dimensional virtual space from various viewpoints (for example, Patent Document 1).

[0003] By the way, the technique for generating a three-dimensional virtual space using motion capture or an imaging device is also used for the production of a preview visualization. For example, it is known that a preview visualization for producing a video work is generated using a large studio and a number of equipment for handling large-scale video works.

[0004] Japanese Unexamined Patent Application Publication No. 2022-93079

[0005] However, it is difficult for the prior art to flexibly generate a three-dimensional virtual space. For example, when creating a preview visualization of a three-dimensional virtual space using motion capture data of a plurality of performers, the plurality of performers perform in the same studio. However, it is difficult for the prior art to flexibly handle a situation where one performer performs the acts of a plurality of performers or performs at another location. Therefore, it is difficult for the prior art to flexibly generate a three-dimensional virtual space without being limited by the number or location of performers.

[0006] Therefore, the present disclosure proposes an information processing system, an information processing method, and an information processing program that can flexibly generate a three-dimensional virtual space.

[0007] To solve the above problems, an information processing system according to this disclosure comprises: an acquisition unit that acquires motion data of a first subject during a first period, information about an imaging device that images the first subject, and motion data of a second subject during a second period; and an output control unit that outputs a third time-series data based on a first time-series data in which the motion data of the first subject and information about the imaging device are arranged in a virtual space, and a second time-series data in which the motion data of the second subject is arranged in the virtual space.

[0008] This figure shows an example configuration of an information processing system according to an embodiment. This figure shows an overview of an information processing system according to an embodiment. This figure shows an example configuration of an information processing device according to an embodiment. This figure shows an example configuration of an output control unit according to an embodiment. This figure shows an example of a GUI (Graphical User Interface) according to an embodiment. This figure shows an example of a GUI according to an embodiment. This figure shows an example of a multi-view image of a subject. This figure shows an example of a 3D spatial image output to a spatial reproduction display. This figure shows an example of synchronizing time-series data. This figure shows an example of placing an LED wall in a virtual space. This figure shows an overview of an information processing procedure according to an embodiment. This figure shows an example of a setting processing procedure according to an embodiment. This figure shows an example of a shooting processing procedure according to an embodiment. This figure shows an example of an integrated processing procedure and an output processing procedure according to an embodiment. This is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device.

[0009] Embodiments of this disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.

[0010] This disclosure will be described in the following order of items: 1. Prior Art 2. Embodiments 2-1. Overview of the Information Processing System According to the Embodiment 2-2. Configuration of the Information Processing Device According to the Embodiment 2-3. Other Configurations According to the Embodiment 2-4. Information Processing Procedure According to the Embodiment 3. Other Embodiments 4. Effects of the Information Processing Device According to the Disclosure 5. Hardware Configuration

[0011] (1. Conventional Technology) In recent years, video productions are sometimes made using highly complex technologies such as VFX (Visual Effects) and VP (Virtual Production). These complex technologies increase the workload on the production workflow.

[0012] Therefore, pre-visualization is sometimes used in the production of video works to streamline the workflow and improve production efficiency. Pre-visualization, also known as pre-vis, refers to simplified CG (Computer Graphics) or other video materials created to consider production methods, necessary budgets, and production images.

[0013] The applications of pre-visualization are diversifying beyond video production to include games, animation, and spatial time-series data. However, conventional technologies require large studios for pre-visualization production, posing challenges to their applicability to small-scale productions, individual projects, and multi-purpose applications such as games and animation.

[0014] (2. Embodiments) (2-1. Overview of the Information Processing System According to the Embodiment) First, an overview of the information processing system according to the embodiment will be described using Figures 1 and 2. Figure 1 is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment.

[0015] In Figure 1, the information processing system 1 consists of a motion sensor 10a, a display device 10b, an imaging device 10c, and an information processing device 100. The motion sensor 10a, the display device 10b, the imaging device 10c, and the information processing device 100 are each connected to each other via a network, whether wired or wirelessly, so as to be able to communicate with each other.

[0016] The motion sensor 10a is an information processing device used by the subject. For example, the motion sensor 10a is a motion capture device. The motion sensor 10a includes an acceleration sensor built into the device itself, or a controller operated by the user or various sensors worn by the user. Alternatively, the motion sensor 10a may be a head-mounted display (HMD) used as a display device 10b. The head-mounted display may be an optical see-through type, a video see-through type, or any other type.

[0017] The display device 10b may be, for example, a terminal device, a liquid crystal display, an organic EL (Electro-Luminescence) display, a head-mounted display (HMD), or a spatial reproduction display (SRD). The spatial reproduction display is a glasses-free stereoscopic image display device that can display time-series data in three dimensions without the use of special glasses. The head-mounted display may also be used as a motion sensor 10a.

[0018] The imaging device 10c is a terminal device or camera that captures images of the subject. Examples of terminal devices include mobile phones, smart devices (smartphones or tablets), PDAs (Personal Digital Assistants), and personal computers.

[0019] The information processing device 100 is a computer that synchronizes multiple time-series data. The information processing device 100 also includes a cloud server that provides users with a three-dimensional virtual space (hereinafter also referred to as the virtual space) built within a computer or computer network.

[0020] By including these components, information processing system 1 can provide users with a portable pre-visualization creation system that can be created by a small number of people.

[0021] Next, the processing flow of the information processing system 1 according to the embodiment will be briefly explained. For example, the information processing system 1 according to the embodiment acquires time-series data obtained from actor A's performance and time-series data obtained from actor B's performance, and synchronizes them by so-called overdubbing. In this embodiment, time-series data is information that shows the positional changes of a subject (it may also be in a stationary state), such as the movement of a subject, over a certain period of time. The time-series data includes motion capture data obtained from the actor's performance, imaging data of the actor, information related to the imaging viewpoint of the imaging device 10c that imaged the actor, and an overhead view of the entire virtual space shown on the spatial reproduction display. For example, the time-series data is shown as moving images such as CG.

[0022] The outline of the information processing system 1 according to the embodiment will be explained using Figure 2. Figure 2 is a diagram showing an outline of the information processing system 1 according to the embodiment. First, the method for aligning the motion sensor 10a and the imaging device 10c will be explained. For example, the information processing device 100 receives from the user the position of a base point in the virtual space displayed on the spatial reproduction display.

[0023] Next, the motion sensor 10a and the imaging device 10c determine their positions in real space. For example, the motion sensor 10a and the imaging device 10c photograph a QR code (registered trademark) that is placed in real space to serve as a base point and set it as the base point.

[0024] The information processing device 100 then reflects the base point in real space onto the base point in virtual space to determine the positions of the motion sensor 10a and the imaging device 10c in virtual space.

[0025] In Figure 2, the information processing device 100 acquires data 51 obtained from actor A71's performance (step S1). The data 51 obtained from actor A71's performance includes motion capture data 511, imaging data, and information 512 related to the imaging device 30.

[0026] The information processing device 100 acquires motion capture data 511 from the motion sensor 10a. For example, the information processing device 100 acquires motion capture data of actor A71 playing the role of bear 61. Specifically, the information processing device 100 acquires motion capture data of actor A71 as motion capture data 511. The information processing device 100 also acquires the position information of the motion capture data 511 in real space.

[0027] The information processing device 100 acquires imaging data and information 512 related to the imaging device 30 from the imaging device 10c. For example, the imaging data refers to video footage (hereinafter, "video" is also referred to as "moving image") of actor A71 during his performance. For example, the information related to the imaging device 30 refers to the position information of the imaging device 10c during actor A71's performance, and information related to the camera and lens, such as the current angle of view and focal length of the lens.

[0028] The information processing device 100 may, for example, integrate motion capture data 511 of actor A71 playing the role of bear 61, imaging data, and information 512 regarding the imaging device 30, and output it to the spatial reproduction display 20. Specifically, the information processing device 100 places actor A71 and the imaging device 10c that images actor A71 on the virtual space output on the spatial reproduction display 20. The information processing device 100 also outputs the field of view of the imaging device 10c on the virtual space.

[0029] Next, the information processing device 100 acquires data 52 obtained from actor B72's performance (step S2). The data 52 obtained from actor B72's performance includes motion capture data 521 of actor B72.

[0030] The information processing device 100 acquires motion capture data 521 from the motion sensor 10a. For example, the information processing device 100 acquires motion capture data of actor B72 playing the role of a hunter 62 hunting a bear 61. In addition to motion capture data, the information processing device 100 may also acquire imaging data of actor B72 and information about the imaging device as data 52 obtained from actor B72's performance.

[0031] Note that actor A71 and actor B72 may be the same person. Also, the actors are not limited to humans; they may be robots or other entities. Actor A71 and actor B72 may perform in different locations. When actor A71 and actor B72 perform in different locations, the motion sensor 10a and imaging device 10c set a base point in real space for each location.

[0032] Then, the information processing device 100 synchronizes the data 51 obtained from actor A71's performance with the data 52 obtained from actor B72's performance (step S3). For example, the information processing device 100 places and synchronizes the performances of the bear 61, the imaging device 10c that images the bear 61 (actor A71), and the hunter 62 on the virtual space output by the spatial reproduction display 20. In other words, the information processing device 100 performs what is known as overdubbing.

[0033] The information processing device 100 outputs synchronized data 53 of the performances of the bear 61 and the hunter 62 to the user (step S4). For example, the information processing device 100 outputs the synchronized data 53 to the display device 10b. The terminal device outputs synchronized data 531 from the viewpoint of the virtual camera. The terminal device can also be operated by the user to select synchronized data from a virtual camera placed in any virtual space.

[0034] Furthermore, the spatial reproduction display 20 outputs synchronized data 532 in a 3D spatial image that provides an overview of the entire virtual space. For example, the spatial reproduction display 20 displays the entire virtual space in which the subject is performing as a 3D spatial image 81, and outputs the position of the performing subject in space, as well as the position and field of view of the camera capturing the image.

[0035] Therefore, the information processing device 100 enables one person to perform multiple roles by synchronizing multiple time-series data. Furthermore, since the information processing device 100 reflects a base point in real space onto the virtual space, even if the location (real space) where the role is performed changes, it can reflect the changes back into the virtual space before the change by setting a base point again in real space. Thus, the information processing device 100 can flexibly generate a three-dimensional virtual space.

[0036] (2-2. Configuration of the Information Processing Device According to the Embodiment) Next, the configuration of the information processing system 1 according to the embodiment will be described using Figure 3. Figure 3 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment.

[0037] As shown in Figure 3, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0038] The communication unit 110 is implemented, for example, by a NIC (Network Interface Card) or a Network Interface Controller. The communication unit 110 is connected to the network N by wire or wireless connection and transmits and receives information with the motion sensor 10a, the display device 10b, and the imaging device 10c via the network N. The network N is implemented by wireless communication standards or methods such as Bluetooth®, the Internet, Wi-Fi®, UWB (Ultra Wide Band), or LPWA (Low Power Wide Area).

[0039] The memory unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks, SSDs (Solid State Drives), and optical discs.

[0040] The control unit 130 is implemented, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), etc., which executes a program stored inside the information processing device 100 (for example, the information processing program according to this disclosure) using RAM or the like as a working area. The control unit 130 is also a controller and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or MCU (Micro Controller Unit).

[0041] Next, a specific configuration example of the control unit 130 is shown. As shown in Figure 3, the control unit 130 comprises an acquisition unit 131, an output control unit 132, an estimation unit 133, and a calibration unit 134. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 3, and other configurations are also acceptable as long as they perform the information processing described later.

[0042] The acquisition unit 131 acquires motion data of the first subject during the first period, information about the imaging device that images the first subject, and motion data of the second subject during the second period. The acquisition unit 131 acquires the movement of the first subject and the movement of the second subject as motion data. The motion data is, for example, motion capture data that records the movement of real people or objects. In this disclosure, motion data is also referred to as motion capture data.

[0043] For example, the acquisition unit 131 acquires motion capture data as inertial skeleton data or optical skeleton data using an inertial motion capture system, an optical motion capture system, or the like. Note that the acquisition unit 131 may acquire motion capture data using an existing inertial motion capture system or an existing optical motion capture system of the related art.

[0044] For example, as information regarding the imaging device, the acquisition unit 131 acquires an imaging viewpoint for imaging a first subject. The acquisition unit 131 acquires, as the imaging viewpoint, an angle of view for imaging the first subject. The acquisition unit 131 acquires the position of the imaging device 10c in the virtual space as information regarding the imaging device (also referred to as information regarding the camera and the lens).

[0045] In addition, the acquisition unit 131 acquires the following information as an example of information regarding the imaging device. - Position and orientation of the camera / lens - Current focal length (angle of view) of the lens - Distance (focus) from the subject - Iris (aperture) - FPS (Frames Per Second) - Shutter speed - ND (Neutral Density) filter - EI (Exposure Index) (ISO) - White Balance - LUT (Look Up Table) - Monitor LUT

[0046] For example, the acquisition unit 131 acquires imaging data of the first subject. Specifically, the acquisition unit 131 acquires, as the imaging data, data obtained by imaging the first subject in the real space. The acquisition unit 131 may also acquire, as the imaging data, a moving image obtained by converting the first subject into an avatar. Note that the acquisition unit 131 may also acquire imaging data obtained by imaging a second subject and information regarding the imaging device.

[0047] Note that the first subject and the second subject are, for example, actors (also referred to as performers). For example, the first period refers to the time when the subject is performing. Also, the second period refers to the time when the subject is performing. For the sake of distinction, in the present disclosure, the subject performing during the first period is referred to as the first subject. Also, in the present disclosure, the subject performing during the second period is referred to as the second subject. To explain the outline of the information processing system 1 according to the above-described embodiment, the first subject is actor A. Also, the second subject is actor B. Note that the actor is not limited to a human and may be a robot or the like.

[0048] The acquisition unit 131 acquires object data. For example, the acquisition unit 131 acquires environment data defining a virtual space, an avatar, a prop, and the like. Specifically, the acquisition unit 131 acquires information on a light source used for creating CG as environment data. Also, a prop is an object used for creating CG used in the virtual space or the background of a subject in imaging data. Note that the environment data may be data obtained by sensing the real space with a terminal device such as a smartphone or a head-mounted display.

[0049] FIG. 4 is a diagram showing a configuration example of the output control unit 132 according to the embodiment. As shown in FIG. 4, the output control unit 132 includes an object data storage unit 321, a moving image data storage unit 322, a motion capture data storage unit 323, an additional data storage unit 324, a position and direction data storage unit 325, an object operation reception unit 326, a playback operation reception unit 327, a viewpoint operation reception unit 328, and a recording operation reception unit 329. Note that the internal configuration of the output control unit 132 is not limited to the configuration shown in FIG. 3 and may be any other configuration as long as it performs the information processing described later.

[0050] For example, the object data storage unit 321, the moving image data storage unit 322, and the motion capture data storage unit 323 store the respective data and information acquired by the acquisition unit 131.

[0051] The object data storage unit 321 stores acquired object data such as environment data, avatars, and props in the storage unit 120. For example, the object data storage unit 321 stores motion capture data that includes polygon meshes and skins that define the outline, and information on the arrangement of joints of the skeleton to be interpolated into the avatar (skeleton data), when converting a first subject or a second subject into an avatar.

[0052] The video data storage unit 322 stores image data of the acquired first subject, second subject, etc. For example, the video data storage unit 322 stores video captured by an imaging device 10c such as a camera, head-mounted display, or tablet. Note that the video may include not only video but also still images.

[0053] The motion capture data storage unit 323 stores motion capture data. For example, the motion capture data storage unit 323 stores motion capture data acquired using a motion sensor 10a such as a head-mounted display.

[0054] The ancillary data storage unit 324 stores information related to the imaging device and equipment information. For example, the ancillary data storage unit 324 stores the settings of the imaging device 10c, such as the camera and lens, when the subject acquired by the acquisition unit 131 is imaged. The ancillary data storage unit 324 also stores equipment information such as the area of ​​the space where the first subject, the second subject, etc., perform.

[0055] Figure 5 shows an example of a GUI (Graphical User Interface) according to the embodiment. As shown in Figure 5, the ancillary data storage unit 324 stores the settings 501 of the imaging device 10c when the image is captured. The settings 501 of the imaging device 10c include, for example, Focal length, FPS, EL, Iris, ND, Monitor LUT, and White Balance, which the ancillary data storage unit 324 stores as settings 501 of the imaging device 10c.

[0056] The position and direction data storage unit 325 stores motion sensors 10a such as a head-mounted display, gyroscopes such as an imaging device 10c, and beacons and UWBs installed in the surrounding environment.

[0057] The object operation reception unit 326 accepts user operations. For example, the object operation reception unit 326 accepts operations from the user that correspond to the creation of CG. As an operation corresponding to the creation of CG, the object operation reception unit 326 accepts operations from the user that correspond to the creation of CG, such as the creation of CG for time-series data using light source information and props. Note that existing technologies may be used for the creation of CG.

[0058] The object manipulation reception unit 326 receives a base point on the virtual interval from the user. For example, the object manipulation reception unit 326 receives an arbitrary position on the virtual space output to the spatial reproduction display from the user as the base point.

[0059] Furthermore, the object operation reception unit 326 receives a reference point in real space from the user. For example, the object operation reception unit 326 receives a device operation from the user at a location designated as a reference point in real space. The object operation reception unit 326 accepts the location as a reference point in real space, triggered by the user's button operation on the motion sensor 10a or imaging device 10c, a specific gesture action, or the reading of a QR code placed at the reference point. The device refers to, for example, the motion sensor 10a or the imaging device 10c.

[0060] The playback operation reception unit 327 accepts operations corresponding to the selection of time series data to be synchronized from multiple time series data. The playback operation reception unit 327 also accepts adjustments to the timelines of the time series data to be synchronized. For example, the playback operation reception unit 327 accepts adjustments to the timing of synchronizing each time series data among multiple time series data.

[0061] The playback operation reception unit 327 may accept the selection of time-series data to be output. For example, the playback operation reception unit 327 may accept the selection of time-series data to be output from the user before outputting the time-series data.

[0062] The playback operation reception unit 327 may accept requests for adjustments to the timeline of the time-series data to be output. For example, before outputting the time-series data, the playback operation reception unit 327 may accept requests from the user to adjust the timing of synchronization for each of the multiple time-series data to be output.

[0063] Figure 6 shows an example of a GUI according to the embodiment. As shown in Figure 6, the playback operation reception unit 327 displays a plurality of time series data 602 and accepts the user's selection 604 of the time series data to be synchronized or output. The playback operation reception unit 327 also accepts adjustments to the timing of synchronizing each time series data among the plurality of time series data 602. For example, the playback operation reception unit 327 accepts the user's selection 604 of three different time series data (AC01, AC02, AC04 in the figure) from the plurality of time series data 602 using checkboxes.

[0064] Furthermore, the playback operation reception unit 327 accepts the user's arbitrary setting to exclude specific time-series data (AC03 in the figure). The playback operation reception unit 327 also accepts the user's arbitrary setting to set the range 603 of the period to be synchronized or output among the synchronized time-series data. The playback operation reception unit 327 may also accept the selection of time-series data to be synchronized or output, and the adjustment of the timeline, while displaying the time-series data on the data display screen 601.

[0065] The viewpoint operation reception unit 328 receives input from the user to switch cameras. For example, the viewpoint operation reception unit 328 receives input from the user regarding the position and angle of a virtual camera placed in the virtual space.

[0066] The recording operation reception unit 329 accepts image quality settings from the user. For example, if the user wants to output simple time-series data, the recording operation reception unit 329 accepts a setting that lowers the image quality. On the other hand, if the user wants to output high-quality time-series data, the recording operation reception unit 329 accepts a setting that increases the image quality.

[0067] The output control unit 132 synchronizes (or integrates) the data stored in the object data storage unit 321, the motion image data storage unit 322, the motion capture data storage unit 323, the ancillary data storage unit 324, and the position and direction data storage unit 325, as well as the data received by the object operation reception unit 326, the playback operation reception unit 327, the viewpoint operation reception unit 328, and the recording operation reception unit 329, and outputs them.

[0068] The output control unit 132 outputs a third time-series data based on a first time-series data set in a virtual space containing motion data of a first subject and information about the imaging device, and a second time-series data set in a virtual space containing motion data of a second subject. For example, the output control unit 132 synchronizes the first time-series data and the second time-series data and outputs them as a third time-series data. In other words, the output control unit 132 outputs the first time-series data and the second time-series data by so-called overdubbing. For example, the output control unit 132 overdubs the first time-series data and the second time-series data onto the entire CG in the virtual space generated from an operation corresponding to the creation of CG and outputs it.

[0069] For the purpose of distinction, in this disclosure, time-series data including video and motion capture data of a first subject placed in a virtual space performing is referred to as the first time-series data. Also for the purpose of distinction, in this disclosure, time-series data including motion capture data of a second subject placed in a virtual space performing is referred to as the second time-series data. Furthermore, in this disclosure, data obtained by synchronizing the first time-series data and the second time-series data is referred to as the third time-series data.

[0070] For example, the output control unit 132 outputs multi-view images of the subject and images showing part or all of the virtual space as third time-series data. For example, the output control unit 132 outputs third time-series data as multi-view images from the viewpoints of multiple virtual cameras provided in the virtual space. Specifically, the output control unit 132 outputs images of the third time-series data from the viewpoint where the first subject was captured in real space as multi-view images of the subject. The output control unit 132 also outputs images of the third time-series data from the viewpoint where the subject is captured from the viewpoint of a virtual camera installed in the virtual space by the user. The output control unit 132 outputs information about the imaging device, such as the distance to the subject, acquired by the acquisition unit 131 while the subject is being captured, as multi-view images. The images showing part or all of the virtual space are, for example, images showing part or all of a 3D spatial image, and are also simply called 3D spatial images.

[0071] Figure 7 shows an example of a multi-view image of a subject. As shown in Figure 7, the output control unit 132 outputs a video 711 to the user that synchronizes a first time-series data including a first subject (a bear played by actor A) 61 and a second time-series data including a second subject (a hunter played by actor B) 62. The output control unit 132 switches the viewpoint using the viewpoint switching switch 701 and outputs the video. The output control unit 132 also outputs information about the camera and lens as information about the imaging device at the time of imaging 702.

[0072] For example, the output control unit 132 may synchronize past time-series data with time-series data currently being captured by the imaging device through real-time processing, and output a video of a third time-series data as seen from the imaging viewpoint of the imaging device 10c while it is capturing the subject.

[0073] The output control unit 132 outputs an image representing part or all of the virtual space, in which the first subject, the second subject, and information about the imaging device included in the third time-series data are arranged in the virtual space. For example, the output control unit 132 outputs an image to the spatial reproduction display that provides an overview of the virtual space, including the first subject, the second subject, and information about the imaging device included in the third time-series data, as an image representing part or all of the virtual space.

[0074] Figure 8 shows an example of a 3D spatial image 81 output to the spatial reproduction display 20. As shown in Figure 8, the output control unit 132 outputs the first subject (a bear played by actor A) 61, the second subject (a hunter played by actor B) 62, the imaging device 1001, and the field of view 1006 of the imaging device 10c onto the virtual space. The output control unit 132 switches the viewpoint using the viewpoint switching switch 801 and outputs the image.

[0075] Figure 9 shows an example of synchronizing time-series data. In Figure 9, the output control unit 132 synchronizes time-series data 901 with other time-series data and outputs the resulting time-series data 902 as a 3D spatial image. Time-series data 901 is data that includes time-series data 901, one bear 61, and one hunter 62. The other time-series data is data that includes three people: hunters 62a, hunter 62b, and hunter 62c. The output control unit 132 synchronizes these time-series data and outputs time-series data 902 that includes the bear 61 and four people: hunter 62, hunter 62a, hunter 62b, and hunter 62c. As a result, the output control unit 132 can output multiple data of different performers filmed at different times as if they were filmed simultaneously. Furthermore, when outputting, the output control unit 132 uses the imaging device 1001 as a virtual camera and outputs time-series data 901 and time-series data 902 with a field of view 1006 that shows the bear 61 and hunter 62 from the position where the imaging device 1001 is located.

[0076] Furthermore, the output control unit 132 may synchronize multiple time series data, not limited to the first and second time series data. Also, after outputting the third time series data, the output control unit 132 may output yet another time series data synchronized with the third time series data. In other words, the output control unit 132 repeatedly synchronizes and outputs time series data.

[0077] For example, the output control unit 132 outputs a third time-series data with an image generated by user operation as the background. Specifically, the output control unit 132 outputs a third time-series data with an image generated by an operation corresponding to the creation of CG as the background. The output control unit 132 outputs the background of the subject based on the generated CG as a multi-view image of the subject. The output control unit 132 also outputs a CG of a 3D spatial image showing the entire virtual space based on the generated CG.

[0078] For example, the output control unit 132 further places the background of the subject in the virtual space and outputs the area of ​​the background captured from the imaging viewpoint. For example, the background is a screen placed in the virtual space. The output control unit 132 virtually places an LED wall as the screen that will be the background of the subject in the virtual space and outputs the area of ​​the background captured from the imaging viewpoint. The area of ​​the background captured from the imaging viewpoint is the shooting range captured by the imaging device 10c.

[0079] Figure 10 shows an example of an LED wall placed in a virtual space. As shown in Figure 10, the output control unit 132 outputs the background 1002 that is captured when the imaging device 1001 captures an image. The output control unit 132 outputs the portion 1004 that does not fit within the LED wall placed in the virtual space, from the area 1003 that forms the background of the subjects (bear 61 played by actor A and hunter 62 played by actor B) when the imaging device 1001 captures an image. It also outputs the field of view 1006 from the imaging device 1001. The output control unit 132 switches the viewpoint using the viewpoint switching switch 1005 and outputs the result. Figure 10 is a 3D spatial image.

[0080] The output control unit 132 outputs the first time-series data to the second subject. For example, the output control unit 132 outputs the first time-series data to the second subject from the second subject's viewpoint. Specifically, when the second subject acts in response to the first subject's acting, the output control unit 132 outputs a video of the first subject acting to the second subject.

[0081] The output control unit 132 outputs the estimated cost. For example, the output control unit 132 outputs the cost related to the actual shooting estimated by the estimation unit 133, which will be described later.

[0082] The estimation unit 133 estimates the costs related to actual filming from the third time-series data. For example, the estimation unit 133 estimates at least one of the following costs: the cost related to the filming time of the actual filming, the cost related to the equipment required for the actual filming, and the cost related to the human resources required for the actual filming. Actual filming refers to the video work created in accordance with the third time-series data, assuming that the third time-series data is the time-series data related to pre-visualization. Specifically, the estimation unit 133 estimates the costs related to actual filming by adding up the filming time, the cost of general equipment, personnel costs, and other costs described below.

[0083] For example, the estimation unit 133 estimates the cost related to the actual shooting time as a cost. The estimation unit 133 estimates the shooting time from third time-series data, according to the number of scenes, the length of each scene, the number of angles in each scene, the total length of the scenes, and the number of angles, as the cost related to the actual shooting time. Then, the estimation unit 133 estimates the cost from the shooting time.

[0084] Specifically, the estimation unit 133 may estimate the cost from the shooting time by adding the total scene length × cost per second and the number of angles × cost per angle of the third time-series video data captured in the virtual space. The total scene length and the number of angles may be set arbitrarily by the user. Note that the calculation method is not limited to this.

[0085] For example, the estimation unit 133 estimates the costs related to the equipment necessary for actual filming. The estimation unit 133 estimates the costs related to the equipment necessary for actual filming from the third time-series data, including studio rental fees based on the size of the studio, the cameras used, lenses used, lighting, drones, cranes, rails and other special equipment, and general equipment such as power supply and air conditioning.

[0086] Specifically, the estimation unit 133 may estimate the cost of general equipment, etc., by adding up the studio rental fee, the cost of special equipment such as the number of cameras, and the cost of power and air conditioning, from the number of angles mentioned above and the 3D spatial image of a third time-series data representing the entire virtual space. For example, the estimation unit 133 estimates the size of the studio by considering the area required for acting and the area required for setting up equipment.

[0087] For example, the estimation unit 133 estimates the costs related to the human resources required for actual filming. The estimation unit 133 estimates the personnel costs of actors, directors, cameramen, lighting technicians, and art staff, etc., from the third time-series data as costs related to the human resources required for actual filming.

[0088] Specifically, the estimation unit 133 estimates the number of actors, directors, cameramen, lighting technicians, and art staff from a 3D spatial image of a third time-series data representing the entire virtual space. For example, the estimation unit 133 estimates the number of actors from the number of subjects. The estimation unit 133 also estimates the number of cameramen from the number of angles. The number of directors and lighting technicians may be estimated according to the number of cameramen and subjects. On the other hand, the number of actors and cameramen may be arbitrarily set by the user. The estimation unit 133 estimates personnel costs by multiplying the total number of actors, directors, etc. by the average personnel cost. However, the calculation method is not limited to this.

[0089] For example, the estimation unit 133 estimates other costs necessary for actual filming as costs related to actual filming. The estimation unit 133 estimates costs such as props, location costs, permit application fees, transportation costs, and shipping costs from the third time-series data as other costs necessary for actual filming.

[0090] Specifically, the estimation unit 133 may estimate other costs from a 3D spatial image of a third time-series data representing the entire virtual space, as well as the number of actors and the number of special equipment mentioned above.

[0091] The calibration unit 134 determines the placement of the subject, indicated by the motion data, in the virtual space from a base point position set in the real space. For example, the calibration unit 134 determines the placement of the motion capture data in the virtual space from the base point in the virtual space received by the object operation reception unit 326 and a base point set in the real space. The calibration unit 134 reflects the base point in the real space into the virtual space and determines the placement of the motion capture data in the virtual space according to the movement of the motion sensor 10a and the imaging device 10c. In other words, the calibration unit 134 determines the placement of the motion capture data in the virtual space by linking the movement of the motion sensor 10a and the imaging device 10c in the real space with the movement in the virtual space. The movement of the motion sensor 10a and the imaging device 10c is detected by communicating with the gyroscopes of the motion sensor 10a and the imaging device 10c, as well as beacons and UWB systems installed in the surrounding environment.

[0092] The calibration unit 134 determines the virtual space placement of the imaging device 10c from a base point position installed in real space. The calibration unit 134 also determines the virtual space placement of the imaging device 10c from a base point position installed in real space using the same process as described above for the motion sensor 10a.

[0093] The above describes an example configuration of the information processing device 100. However, the information processing system in this disclosure consists of a motion sensor 10a, a display device 10b, an imaging device 10c, and the like, in addition to the information processing device 100. The acquisition unit 131, the output control unit 132, the components constituting the output control unit 132, the estimation unit 133, and the calibration unit 134 described above may be configured in other devices.

[0094] For example, the information processing system 1 includes an imaging device 10c, a display device 10b, and an information processing device 100, and at least one of the imaging device 10c and the display device 10b may be configured to include an output control unit 132. That is, the information processing system 1 may be configured such that the imaging device 10c or the display device 10b includes an output control unit 132, and the other components are provided by the information processing device 100. The information processing system 1 displays time-series data on the display device 10b and the imaging device 10c. Therefore, the information processing system 1 can provide a system that generates a three-dimensional virtual space that does not require a large studio and is excellent in portability and ease of setup.

[0095] (2-3. Other Configurations According to the Embodiment) The motion sensor 10a is an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. The motion sensor 10a is configured to communicate with other devices (such as the information processing device 100). For example, the motion sensor 10a is used by being attached to a first subject or a second subject. For example, the motion sensor 10a transmits motion capture data and real-space position information of the motion sensor 10a to the information processing device 100. The HMD used as the motion sensor 10a also receives data from the information processing device 100 regarding the environment in which the performance is being carried out, such as CG, and video of other subjects performing.

[0096] The display device 10b is a spatial reproduction display, a terminal device, or a display. The display device 10b is configured to communicate with other devices (information processing device 100). For example, the display device 10b receives time-series data output from the information processing device 100. The display device 10b outputs time-series data output from the information processing device 100. For example, the display device 10b transmits to the information processing device 100 a base point in the virtual space received from the user.

[0097] The imaging device 10c is a terminal device or a camera. The imaging device 10c is configured to communicate with other devices (such as an information processing device). The imaging device 10c may be used to acquire motion capture data of a first subject or a second subject. For example, the imaging device 10c transmits imaging data to the information processing device. The imaging device 10c also receives time-series data from the information processing device 100.

[0098] (2-4. Information Processing Procedure According to the Embodiment) Next, the processing of each part constituting the information processing device 100 described above will be described in detail in order to flow using Figures 11 to 14. Figure 11 is a diagram showing an overview of the information processing procedure according to the embodiment.

[0099] As shown in Figure 11, the information processing device 100 sets the base point of the motion sensor 10a and the imaging device 10c (step S101). Next, the information processing device 100 acquires motion capture data of the subject, imaging data, and information about the imaging device (step S102). Subsequently, the information processing device 100 integrates (synchronizes) the acquired motion capture data of the subject, imaging data, and information about the imaging device (step S103). Then, the information processing device 100 outputs the integrated time-series data (step S104).

[0100] The internal processing of step S101 performed by the information processing device 100 will be explained in detail using Figure 12. Figure 12 is a diagram showing an example of a setting processing procedure according to the embodiment. First, the object operation reception unit 326 of the information processing device 100 determines whether it has received a request from the user to determine a base point in the virtual space (step S11). The object operation reception unit 326 repeats the process until it receives a base point in the virtual space from the user (step S11: No).

[0101] Next, if the object operation reception unit 326 receives a determination of a base point in the virtual space from the user (step S11: Yes), it determines whether it has received an operation to set the base point in the real space set by the user for the device (step S12). The object operation reception unit 326 repeats the process until it receives an operation for the device (step S12: No). Note that the device refers to the imaging device 10c and the motion sensor 10a.

[0102] Next, when the calibration unit 134 receives a device operation (step S12: Yes), it determines the position in the virtual space in conjunction with the position of the device in the real space. The output control unit 132 reflects and displays the movement of the device in the real space in the virtual space (step S13). For example, the output control unit 132 outputs the movement of the device in the virtual space, which is linked to the movement of the device in the real space.

[0103] The object operation reception unit 326 then determines whether it has accepted the addition of a device (step S14). If the object operation reception unit 326 has accepted the addition of a device (step S14: Yes), it repeats the process from step S11. If it does not accept the addition of a device (step S14: No), it starts the process in step S21.

[0104] The internal processing of step S102 performed by the information processing device 100 will be explained in detail using Figure 13. Figure 13 is a diagram showing an example of the imaging processing procedure according to the embodiment. First, the output control unit 132 generates a virtual space (step S21).

[0105] Next, the acquisition unit 131 determines whether it has acquired information about the imaging device, motion capture data, and imaging data (step S22). If the acquisition unit 131 does not acquire information about the imaging device, motion capture data, and imaging data, it repeats the process in step S22 (step S22: No).

[0106] If the object operation reception unit 326 acquires information regarding the imaging device, motion capture data, and imaging data (step S22: Yes), it determines whether it has received an operation request from the user for the production of CG (step S23).

[0107] When the output control unit 132 receives an operation from the user regarding the creation of CG (step S23: Yes), it determines whether to synchronize with other time-series data (step S24). The output control unit 132 repeats the process in step S23 until it receives an operation from the user regarding the creation of CG (step S23: No).

[0108] The playback operation reception unit 327 determines whether it has received the selection of time series data to be synchronized if it is to synchronize with other time series data (step S24: Yes) (step S25). If the playback operation reception unit 327 does not synchronize with other time series data (step S24: No), it starts the process in step S31. The playback operation reception unit 327 repeats the process in step S25 until it receives the selection of time series data (step S25: No).

[0109] If the playback operation reception unit 327 receives a selection of time-series data (step S25: Yes), it determines whether it has received a request to adjust the timeline of the synchronized time-series data (step S26). The playback operation reception unit 327 repeats the process in step S26 until it receives a request to adjust the timeline (step S26: No).

[0110] If the output control unit 132 accepts the timeline adjustment (step S26: Yes), it determines whether to continue synchronization (step S27). If the output control unit 132 decides to continue synchronization (step S27: Yes), it starts processing from step S21. On the other hand, if it decides not to continue synchronization (step S27: No), it starts step S31.

[0111] The internal processing of steps S103 and S104 performed by the information processing device 100 will be explained in detail using Figure 14. Figure 14 is a diagram showing an example of an integrated processing procedure and an output processing procedure according to the embodiment.

[0112] The playback operation reception unit 327 determines whether it has received a selection of time-series data to be output (step S31). The playback operation reception unit 327 repeats the process in step S31 until it receives a selection of time-series data to be played back (step S31: No). If the playback operation reception unit 327 has received a selection of time-series data to be output (step S31: Yes), it determines whether it has received a request to adjust the timeline of the time-series data to be output (step S32).

[0113] If the playback operation reception unit 327 receives a request to adjust the timeline of the time-series data to be output (step S32: Yes), it determines whether it has received a request to output content or a 3D storyboard (step S33). Here, "content" refers to time-series data including multi-view video. A 3D storyboard refers to time-series data including 3D spatial video output to the spatial reproduction display and matters related to actual filming. The playback operation reception unit 327 repeats the process in step S32 until it receives a request to adjust the timeline of the time-series data to be output (step S32: No).

[0114] When the viewpoint operation reception unit 328 receives a content selection (step S33: content), it determines whether it has received the setting of the virtual camera's position and angle (step S34). The viewpoint operation reception unit 328 repeats the process in step S34 until it receives the setting of the angle (step S34: No).

[0115] When the recording operation reception unit 329 receives the virtual camera's position and angle settings (step S34: Yes), it determines whether scaling has been accepted (step S35). The recording operation reception unit 329 repeats the process in step S35 until scaling is accepted (step S35: No). Scaling refers to the setting of the image quality of the output video.

[0116] If the output control unit 132 accepts scaling (step S35: Yes), it outputs time-series data including multi-view images (step S36).

[0117] On the other hand, when the playback operation reception unit 327 receives a selection for outputting a 3D storyboard (step S33: 3D storyboard), it determines whether it has received a selection for the data type (step S37). For example, as a selection for the data type, the playback operation reception unit 327 accepts a selection for the type of shooting estimate to be output (also called the cost related to actual shooting). Specifically, the playback operation reception unit 327 accepts a selection for labor costs from the user as the type of shooting estimate. The playback operation reception unit 327 repeats the process in step S37 until it receives a selection for the data type (step S37: No).

[0118] If the estimation unit 133 receives a selection of data type (step S37: Yes), it outputs an estimate of the shooting location (step S38). Subsequently, the output control unit 132 outputs a 3D storyboard (step S39).

[0119] (3. Other Embodiments) Of the processes described in the embodiments of this disclosure described above, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0120] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0121] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas that do not contradict the processing content. Also, the steps shown in the sequence diagram or flowchart of this embodiment can be changed in order as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel.

[0122] (4. Effects of the Information Processing Device According to the Disclosure) As described above, the information processing device according to the Disclosure (in the embodiment, the information processing device 100) comprises an acquisition unit (in the embodiment, the acquisition unit 131), an output control unit (in the embodiment, the output control unit 132), an estimation unit (in the embodiment, the estimation unit 133), a calibration unit (in the embodiment, the calibration unit 134), and an object operation receiving unit (in the embodiment, the object operation receiving unit 326). The acquisition unit acquires motion data of a first subject during a first period, information about an imaging device that images the first subject, and motion data of a second subject during a second period. The output control unit outputs a third time-series data based on a first time-series data in which the motion data of the first subject and information about the imaging device are placed in a virtual space, and a second time-series data in which the motion data of the second subject is placed in a virtual space.

[0123] Thus, the information processing device can output 3D CG that reflects the movement of a subject from motion capture data. Furthermore, by superimposing the performances of multiple subjects captured separately, the information processing device can generate a 3D virtual space without being constrained by time, place, or the number of personnel involved (from individuals to small groups). Therefore, the information processing device can flexibly generate a 3D virtual space.

[0124] Furthermore, for example, the information processing device can output information about the imaging device, such as the subject's position, the subject's movement, the imaging device's position, the imaging device's angle, the field of view during imaging, focus, and aperture, to a virtual space in a time-series manner.

[0125] The output control unit synchronizes the first time-series data and the second time-series data and outputs them as a third time-series data. In this way, the information processing device can generate a three-dimensional virtual space by synchronously superimposing the performances of multiple subjects that were captured separately.

[0126] The acquisition unit further acquires imaging data of the first subject.

[0127] In this way, the information processing device can verify the image from the image data obtained from the first subject. For example, the information processing device can verify the image from the image data obtained from the image data captured by the user from an arbitrary position.

[0128] The output control unit outputs, as a third time-series data set, multi-view images of the subject and images showing part or all of the virtual space.

[0129] In this way, the information processing device can output multi-viewpoint video so that users can ideaize through trial and error. For example, the information processing device can output video showing part or all of the virtual space related to pre-visualization so that actual filming can be carried out efficiently.

[0130] The output control unit outputs a third time-series data as multi-viewpoint video from the viewpoints of multiple virtual cameras placed in the virtual space.

[0131] Thus, the information processing device can output 3D CG that can be used not only for pre-visualization but also for various other purposes, such as actual production. For example, by accepting settings such as image quality, the information processing device can output higher quality 3D CG.

[0132] The output control unit outputs an image in which the first subject, the second subject, and information about the imaging device included in the third time-series data are placed in the virtual space, representing part or all of the virtual space.

[0133] In this way, the information processing device allows the user to view the position and angle of the imaging device from a virtual space, thereby enabling the user to verify the position and angle of the imaging device relative to the virtual space.

[0134] The acquisition unit acquires the imaging viewpoint for imaging the first subject as information about the imaging device.

[0135] In this way, the information processing device can allow the user to verify the background area from the imaging viewpoint.

[0136] Furthermore, the output control unit places the subject's background in the virtual space and outputs the background area captured from the imaging viewpoint. The background is a screen placed in the virtual space.

[0137] In this way, the information processing device can output to the user in real time the relationship between the angle of the imaging device and the area of ​​the background. Specifically, by using an LED wall as a simulated screen to verify the angle of the imaging device, the information processing device can output multi-view images and 3D spatial images that reflect the physical constraints of actual shooting.

[0138] The output control unit outputs the first time-series data to the second subject.

[0139] In this way, the information processing device can output first time-series data to the second subject so that the second subject can perform actions or movements in response to video footage of the first subject's performance or other actions.

[0140] The calibration unit determines the virtual placement of the subject, indicated by motion data, based on a base point located in real space.

[0141] In this way, the information processing device can reflect the movement and position of an object in real space onto the movement and position of an object in virtual space. Furthermore, for example, the information processing device can provide the user with a simple calibration method using a motion sensor (in this embodiment, motion sensor 10a).

[0142] The calibration unit determines the virtual placement of the imaging device based on a reference point located in real space.

[0143] In this way, the information processing device can reflect the position of the imaging device in real space to the position of the imaging device in sparsely populated space. Furthermore, for example, the information processing device can provide the user with a simple calibration method using the imaging device (in this embodiment, imaging device 10c).

[0144] The estimation unit estimates the cost of actual filming from the third time-series data.

[0145] In this way, when the information processing device outputs time-series data related to pre-visualization as a third time-series data, it can estimate the cost so that actual shooting can be carried out efficiently.

[0146] The estimation unit estimates at least one of the following costs: the cost related to the actual shooting time, the cost related to the equipment required for the actual shooting, and the cost related to the human resources required for the actual shooting.

[0147] In this way, when the information processing device outputs time-series data related to pre-visualization as a third time-series data, it can estimate the cost related to the shooting time so that actual shooting can be carried out efficiently.

[0148] Furthermore, when the information processing device outputs time-series data related to pre-visualization as a third type of time-series data, it can estimate the costs of the necessary equipment to enable efficient actual filming.

[0149] Furthermore, when the information processing device outputs time-series data related to pre-visualization as a third type of time-series data, it can estimate the cost of the necessary human resources so that actual filming can be carried out efficiently.

[0150] Furthermore, the output control unit outputs the estimated cost. In this way, the information processing device can efficiently carry out actual shooting by presenting the cost to the user.

[0151] The object manipulation reception unit receives user input. The output control unit outputs a third time-series data with the image generated by the user input as the background.

[0152] In this way, the information processing device can receive the generation of arbitrary computer graphics (CG) from a user and output time-series data that reflects those CGs.

[0153] The information processing system includes an imaging device (in this embodiment, an imaging device 10c), a display device (in this embodiment, a display device 10b), and an information processing device, wherein at least one of the imaging device and the display device is equipped with an output control unit.

[0154] In this way, the information processing system can output a highly portable and easy-to-set-up 3D virtual space by displaying time-series data on display devices such as spatial reproduction displays and terminal devices, as well as imaging devices such as cameras and terminal devices. Furthermore, because the information processing system is highly portable and easy to set up, it can output a 3D virtual space regardless of location.

[0155] (5. Hardware Configuration) The information processing device 100, display device 10b, and imaging device 10c according to the embodiments of this disclosure described above are realized by a computer 1000 having a configuration such as that shown in Figure 15. The information processing device 100 will be explained as an example. Figure 15 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a processing circuitry 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The parts of the computer 1000 are connected by a bus 1050.

[0156] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0157] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0158] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing device 100 according to the embodiment of this disclosure, which is an example of program data 1450.

[0159] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550. The communication interface 1500 corresponds to the communication unit 110 of the information processing device 100. For example, the processing circuit 1100 receives data from other devices or transmits data generated by the processing circuit 1100 to other devices via the communication interface 1500.

[0160] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0161] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (Organic Electro Luminescence Display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device.

[0162] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.

[0163] For example, when computer 1000 functions as an information processing device 100 according to an embodiment of this disclosure, the processing circuit 1100 of computer 1000 functions as a control unit 130 by executing a program loaded onto RAM 1200. The secondary storage device 1400 stores the information processing program according to this disclosure and various data stored by the storage unit 120. The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside computer 1000, but may be located outside computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0164] Furthermore, this technology can also be configured as follows: (1) An information processing system comprising: an acquisition unit that acquires motion data of a first subject during a first period, information about an imaging device that images the first subject, and motion data of a second subject during a second period; and an output control unit that outputs a third time-series data based on a first time-series data in which the motion data of the first subject and information about the imaging device are arranged in a virtual space, and a second time-series data in which the motion data of the second subject is arranged in the virtual space. (2) The information processing system according to (1), wherein the output control unit synchronizes the first time-series data and the second time-series data and outputs them as the third time-series data. (3) The information processing system according to (1) or (2), wherein the acquisition unit further acquires imaging data of the first subject. (4) The information processing system according to any one of (1) to (3), wherein the output control unit outputs multi-view video of the subject and video showing part or all of the virtual space as the third time-series data. (5) The information processing system according to (4), wherein the output control unit outputs the third time-series data from the viewpoints of a plurality of virtual cameras provided in the virtual space as the multi-view video. (6) The information processing system according to (4), wherein the output control unit outputs video showing part or all of the virtual space, in which the first subject, the second subject, and information about the imaging device included in the third time-series data are arranged in the virtual space. (7) The information processing system according to any one of (1) to (6), wherein the acquisition unit acquires an imaging viewpoint for imaging the first subject as information about the imaging device. (8) The information processing system according to (7), wherein the output control unit further arranges the background of the subject in the virtual space and outputs the region of the background imaged from the imaging viewpoint. (9) The information processing system described in (8), wherein the background is a screen placed in the virtual space.(10) The information processing system according to any one of (1) to (9), wherein the output control unit outputs the first time-series data to the second subject. (11) The information processing system according to any one of (1) to (10), further comprising a calibration unit that determines the placement of the subject indicated by the motion data in the virtual space from a base point position installed in real space. (12) The information processing system according to any one of (1) to (10), further comprising a calibration unit that determines the placement of the imaging device in the virtual space from a base point position installed in real space. (13) The information processing system according to any one of (1) to (12), further comprising an estimation unit that estimates the cost related to actual shooting from the third time-series data. (14) The information processing system according to (13), wherein the estimation unit estimates at least one of the following costs: the cost related to the shooting time of actual shooting, the cost related to the equipment required for actual shooting, and the cost related to the human resources required for actual shooting. (15) The information processing system according to (13), wherein the output control unit outputs the estimated cost. (16) The information processing system according to any one of (1) to (15), further comprising an object operation receiving unit that accepts user operations, wherein the output control unit outputs the third time-series data with the image generated by the user operation as the background. (17) The information processing system according to any one of (1) to (16), wherein the information processing system includes the imaging device, the display device, and the information processing device, wherein at least one of the imaging device and the display device comprises the output control unit. (18) An information processing method comprising: a computer acquiring motion data of a first subject during a first period, information about an imaging device for imaging the first subject, and motion data of a second subject during a second period; and outputting a third time series data based on a first time series data in which the motion data of the first subject and the information about the imaging device are placed in a virtual space, and a second time series data in which the motion data of the second subject is placed in the virtual space.(19) An information processing program for causing a computer to function as an information processing device comprising: an acquisition unit that acquires motion data of a first subject in a first period, information about an imaging device that images the first subject, and motion data of a second subject in a second period; and an output control unit that outputs a third time-series data based on a first time-series data in which the motion data of the first subject and information about the imaging device are arranged in a virtual space, and a second time-series data in which the motion data of the second subject is arranged in the virtual space.

[0165] 1 Information Processing System 100 Information Processing Device 10a Motion Sensor 10b Display Device 10c Imaging Device 110 Communication Unit 120 Storage Unit 130 Control Unit 131 Acquisition Unit 132 Output Control Unit 133 Estimation Unit 134 Calibration Unit 321 Object Data Storage Unit 322 Moving Image Data Storage Unit 323 Motion Capture Data Storage Unit 324 Ancillary Data Storage Unit 325 Position and Direction Data Storage Unit 326 Object Operation Reception Unit 327 Playback Operation Reception Unit 328 Viewpoint Operation Reception Unit 329 Recording Operation Reception Unit

Claims

1. An information processing system comprising: an acquisition unit that acquires motion data of a first subject during a first period, information about an imaging device that images the first subject, and motion data of a second subject during a second period; and an output control unit that outputs a third time-series data based on a first time-series data in which the motion data of the first subject and information about the imaging device are arranged in a virtual space, and a second time-series data in which the motion data of the second subject is arranged in the virtual space.

2. The information processing system according to claim 1, wherein the output control unit synchronizes the first time series data and the second time series data and outputs them as the third time series data.

3. The information processing system according to claim 1, wherein the acquisition unit further acquires imaging data of the first subject.

4. The information processing system according to claim 1, wherein the output control unit outputs, as the third time-series data, a multi-view image of a subject and an image showing a part or the whole of the virtual space.

5. The information processing system according to claim 4, wherein the output control unit outputs the third time-series data as the multi-view image from the viewpoints of a plurality of virtual cameras provided in the virtual space.

6. The information processing system according to claim 4, wherein the output control unit outputs an image in which the first subject, the second subject, and information regarding the imaging device included in the third time-series data are arranged in the virtual space as an image showing part or all of the virtual space.

7. The information processing system according to claim 1, wherein the acquisition unit acquires an imaging viewpoint for imaging the first subject as information relating to the imaging device.

8. The information processing system according to claim 7, wherein the output control unit further places the background of the subject in the virtual space and outputs the region of the background captured from the imaging viewpoint.

9. The information processing system according to claim 8, wherein the background is a screen placed in the virtual space.

10. The information processing system according to claim 1, wherein the output control unit outputs the first time-series data to the second subject.

11. The information processing system according to claim 1, further comprising a calibration unit that determines the placement of an object indicated by motion data in the virtual space from a base point position installed in real space.

12. The information processing system according to claim 1, further comprising a calibration unit that determines the placement of the imaging device in the virtual space from a base point position installed in real space.

13. The information processing system according to claim 1, further comprising an estimation unit for estimating the cost related to actual shooting from the third time-series data.

14. The information processing system according to claim 13, wherein the estimation unit estimates at least one of the following costs: the cost related to the shooting time of the actual shooting, the cost related to the equipment required for the actual shooting, and the cost related to the human resources required for the actual shooting.

15. The information processing system according to claim 13, wherein the output control unit outputs the estimated cost.

16. The information processing system according to claim 1, further comprising an object operation receiving unit that accepts user operations, wherein the output control unit outputs the third time-series data with the image generated by the user operation as the background.

17. The information processing system according to claim 4, wherein the information processing system includes the imaging device, the display device, and the information processing device, and at least one of the imaging device and the display device is equipped with the output control unit.

18. An information processing method comprising: a computer acquiring motion data of a first subject during a first period, information about an imaging device for imaging the first subject, and motion data of a second subject during a second period; and outputting a third time-series data based on a first time-series data in which the motion data of the first subject and the information about the imaging device are placed in a virtual space, and a second time-series data in which the motion data of the second subject is placed in the virtual space.

19. An information processing program for causing a computer to function as an information processing device comprising: an acquisition unit that acquires motion data of a first subject during a first period, information about an imaging device that images the first subject, and motion data of a second subject during a second period; and an output control unit that outputs a third time-series data based on a first time-series data in which the motion data of the first subject and information about the imaging device are arranged in a virtual space, and a second time-series data in which the motion data of the second subject is arranged in the virtual space.

Images