Information processing device, information processing method, and program

Abstract

A system for three-dimensional reconfiguration of an object comprising: circuitry configured to: in response to a first input, provide on a display a three-dimensional construction of the object and a plurality of visual indicators positioned relative to the three-dimensional construction of the object, and in response to a selection to select one of the plurality of visual indicators that visually identifies the requirement for correction, transition from providing on the display the three-dimensional construction of the object to providing the selected corresponding two-dimensional view of the object. Each of the visual indicators can be associated with previously captured image view data corresponding to different two-dimensional views of the object and can visually identify whether or not the corresponding two-dimensional view of the object requires correction. The selected corresponding two-dimensional view of the object can be provided on the display with an identification corresponding to the requirement for correction.

Description

INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, AND PROGRAMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2025-007508 filed on January 20, 2025, the entire contents of which are incorporated herein by reference.

[0002] The present technology relates to an information processing device, an information processing method, and a program.

[0003] As a technique in the related art, a technique of generating a three-dimensional model by performing a three-dimensional reconfiguration using a plurality of videos captured by a plurality of cameras has been proposed. In addition, a technique for presenting a UI that displays a missing part in such a three-dimensional model has also been proposed (PTL 1).

[0004] WO 2019-244944 ASummary

[0005] PTL 1 discloses displaying a caption indicating the presence of a missing part, and expressing the missing part with a specific color or pattern. However, in generating a reconfiguration result of 3D data, 4D data, or the like by a reconfiguration process, a more detailed information presentation method is desired.

[0006] The present technology has been made in view of such a problem, and it is desired to provide an information processing device, an information processing method, and a program capable of displaying information regarding a reconfiguration result.

[0007] According to an embodiment of the present disclosure / technology, a first technology is an information processing device including a display processing unit that performs a process of displaying a reconfiguration result generated by a reconfiguration process on the basis of a plurality of pieces of imaging data and information regarding the reconfiguration result on a display unit.

[0008] Furthermore, a second technology is an information processing method of performing a process of displaying a reconfiguration result generated by a reconfiguration process on the basis of a plurality of pieces of imaging data and information regarding the reconfiguration result on a display unit.

[0009] Furthermore, a third technology is a program for causing a computer to execute an information processing method of performing a process of displaying a reconfiguration result generated by a reconfiguration process on the basis of a plurality of pieces of imaging data and information regarding the reconfiguration result on a display unit.

[0010] Fig. 1 is an explanatory diagram of 4D data according to the present technology.Fig. 2 is a block diagram illustrating a configuration of an information processing system 10.Fig. 3 is a diagram illustrating an example of a structure of image data.Fig. 4 is a diagram illustrating a configuration of a processing block of an information processing device 100 according to the first embodiment.Fig. 5 is a diagram illustrating a hardware configuration of the information processing device 100.Fig. 6 is a flowchart illustrating processing of the information processing device 100 according to the first embodiment.Fig. 7 is a diagram illustrating a GUI according to the first embodiment.Fig. 8 is a diagram illustrating a GUI according to the first embodiment.Fig. 9 is a diagram illustrating a GUI according to the first embodiment.Fig. 10 is a diagram illustrating a GUI according to the first embodiment.Fig. 11 is a diagram illustrating a GUI according to the first embodiment.Fig. 12 is a diagram illustrating a GUI according to the second embodiment.Fig. 13 is a diagram illustrating a GUI according to the second embodiment.Fig. 14 is a diagram illustrating a GUI according to the second embodiment.Fig. 15 is a diagram illustrating a GUI according to the second embodiment.Fig. 16 is a diagram illustrating a GUI according to the second embodiment.Fig. 17 is a diagram illustrating a configuration of a processing block of the information processing device 100 according to the third embodiment.Fig. 18 is a flowchart illustrating processing of an information processing device 100 according to the third embodiment.Fig. 19 is a diagram illustrating a GUI according to the third embodiment.Fig. 20 is a diagram illustrating a GUI according to the third embodiment.Fig. 21 is a diagram illustrating a GUI according to the third embodiment.Fig. 22 is a diagram illustrating a GUI according to the third embodiment.

[0011] Hereinafter, an embodiment of the present technology will be described with reference to the drawings. Note that the description will be given in the following order. <First embodiment> (4D Data) (Configuration of information processing system 10) (Configuration of information processing device 100) (Processing in information processing device 100) <Second embodiment> (Processing in information processing device 100) <Third embodiment> (Configuration of information processing device 100) (Processing in information processing device 100) <Utilization of present technology> <Modification>

[0012] <First embodiment> (4D Data) 4D data in the present technology will be described with reference to Fig. 1. An information processing device 100 included in an information processing system 10 according to the present technology generates 3D data or 4D data as a reconfiguration result by a reconfiguration process from a plurality of pieces of imaging data captured by a plurality of cameras 200 at different positions and postures. Generating 3D data from two-dimensional image data or moving image data as imaging data may be referred to as 3D reconfiguration, and further generating 4D data obtained by adding a time axis to the 3D data may be referred to as 4D reconfiguration. In the first to third embodiments, the imaging data is image data captured by the camera 200, but the imaging data may be moving image data (a plurality of frame images constituting moving image data).

[0013] In the present technology, 4D data is data including information (including color, material, and the like) on a 3D spatial axis moving along a time axis. Specifically, the data is animated 3D mesh data, 3D point cloud data, polygon data, Neural Radiance Fields (NeRF) corresponding to a dynamic scene, 3D Gaussian Splatting (3DGS), or the like.

[0014] The generated 4D data can be distributed to a large number of devices, and the distributed 4D data is rendered on each device, and finally is a stereo moving image of a time axis and a 2D axis.

[0015] It is possible to pre-render the 4D data to generate a stereo moving image and deliver the stereo moving image to the device of the viewer / listener. In this case, viewing from a fixed viewpoint or viewing in a degree of freedom (3DoF) in a case of a format such as the VR180 is possible.

[0016] On the other hand, by distributing in the state of 4D data instead of a stereo moving image, real-time rendering can be performed in the device of the viewer / listener. In this case, it is possible to view in 6DoF in which “shake one's head left and right, back and forth, turn one's head to look around” is added to “move back and forth, up and down, left and right”. As a result, the viewer / listener can view the 4D data from any viewpoint. At this time, the device estimates the posture of the device itself by simultaneous localization and mapping (SLAM) or the like.

[0017] Examples of the viewing device include a head mounted display (HMD), an augmented reality (AR) device such as a glasses-type wearable device or a smartphone, a spatial reproduction display, and the like.

[0018] Note that, in the above description, the method of providing the 4D data and the stereo moving image is referred to as “distribution”, but the providing method may be, for example, storing the 4D data and the stereo moving image in a storage medium and distributing the storage medium to the viewer / listener, or displaying and showing the 4D data and the stereo moving image.

[0019] As a standard of 4D data distribution, there is video-based dynamic mesh coding (V-DMC) which is an international standard related to compression and transmission of a 3D mesh changing in a time direction. There is a video-based point cloud compression (V-PCC), which is an international standard related to compression and transmission of a 3D point cloud that changes in a time direction.

[0020] The present technology can also be applied to volumetric capture. The volumetric capture is a technique in which an entire space is captured by 100 or more cameras 200 surrounding the space in 360 degrees, and a real space is captured as three-dimensional digital data and reproduced in high quality. The generated data can be converted into a 2D moving image viewed from any direction as free viewpoint representation, or can be converted into a 3D moving image viewed by an AR, a stereoscopic monitor, an HMD, or the like.

[0021] In addition, for providing 4D data, it is possible to display the 4D data on a browser basis using a technology such as graphics library (WebGL) or WebXR capable of realizing 3D expression or an extended reality (XR) application on a web browser.

[0022] (Configuration of information processing system 10) A configuration of the information processing system 10 will be described with reference to Fig. 2. The information processing system 10 includes the information processing device 100, the plurality of cameras 200, a terminal device 300, a first database 400, a second database 500, and a third database 600.

[0023] The information processing device 100 generates 3D data or 4D data as a reconfiguration result by performing a reconfiguration process with image data captured by the plurality of cameras 200 as an input. The information processing device 100 is used by a person who intends to generate a reconfiguration result. For example, in a case where a target of imaging by the camera 200, that is, a target of the reconfiguration process is an event, an operator or the like who operates the event uses the information processing device 100 to generate a reconfiguration result. The camera 200 may capture an image in a state where the position and the posture are fixed, or may capture an image while moving, that is, while changing the position and the posture.

[0024] The event may be, for example, any event such as a large scale event in which the operator gathers many guests, such as life's major ceremonial occasions such as a wedding or the like, an event, a stage, a recital, a live show, a festival, or a sport event, or a small scale event such as a family or friend gathering. Note that the target of the reconfiguration process is not limited to the event, and may be any target as long as the target can be imaged by the camera 200, such as animals, nature, scenery, and buildings.

[0025] In the present specification, a person, such as an operator of the event described above, who generates a reconfiguration result using the information processing device 100 is referred to as a user. The user can provide the reconfiguration result, and a person who receives the reconfiguration result from the user and views the reconfiguration result is referred to as a viewer / listener. In addition, a person who images an event with the camera 200 is referred to as a camera person.

[0026] It is assumed that the camera 200 includes a camera that has been calibrated and whose position and posture and camera parameters are known, and a camera that has not been calibrated and whose position and posture and camera parameters are unknown. In a case where it is necessary to distinguish the plurality of cameras 200, they are referred to as a camera 200A, a camera 200B, a camera 200C, a camera 200D,...

[0027] For example, the calibrated camera 200 is prepared by an event operator (user) who uses the information processing system 10, and is used by an event managing staff or the like. Furthermore, the camera 200 that has not been calibrated is used by a participant of an event or the like. The plurality of cameras 200 may have different sensor characteristics, F values, white balance, exposure, camera parameters, and the like due to different manufacturers, types, models, and the like, or different ways of use by the camera person.

[0028] The information processing device 100, the plurality of cameras 200, the terminal device 300, the first database 400, the second database 500, and the third database 600 have communication functions and are connected via a network. The network connection method may be wired connection or wireless connection. Examples of the wired connection method include a high-definition multimedia interface (HDMI) (registered trademark) and a Universal Serial Bus (USB). Examples of the wireless connection method include Wi-Fi, Bluetooth (registered trademark), a wireless local area network (LAN), near field communication (NFC), a fourth generation mobile communication system (4G), a fifth generation mobile communication system (5G), and Ethernet (registered trademark).

[0029] The camera 200 includes an imaging element, a signal processing circuit, and the like, and can obtain moving image data and image data of color or single color by imaging. As the imaging element, a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like is used. Note that the camera 200 may be any device having a camera function, such as a smartphone, a tablet terminal, or a wearable device. The plurality of cameras 200 may have a difference between the plurality of pieces of image data due to a difference in camera parameters, imaging settings, imaging positions, imaging postures, and the like.

[0030] At the time of imaging, the camera 200 acquires and generates various types of information regarding imaging, and adds the various types of information to the image data as metadata corresponding to the image data. The camera 200 may acquire or generate metadata after imaging and add the metadata to the image data. Then, the camera 200 transmits the image data and the metadata to the information processing device 100. The transmission of the image data and the metadata may be automatically performed at a timing when the imaging mode of the camera 200 ends, or may be performed in response to an input of a transmission instruction by the camera person. Note that the image data and the metadata may be supplied from the camera 200 to the information processing device 100 via a recording medium such as a USB flash memory or an SD memory card.

[0031] The position and posture information of the camera 200 is necessary in the generation of the reconfiguration result by the reconfiguration process. Therefore, it is desirable that the camera 200 include a global positioning system (GPS) sensor that detects position information, an inertial measurement unit (IMU) for detecting posture information, an inertial sensor (acceleration sensor, angular speed sensor, gyro sensor for two-axis or three-axis direction), and the like. In a case where the camera 200 includes a GPS sensor and an IMU, the camera 200 can add position information detected by the GPS sensor, posture information detected by the IMU, or the like to image data as metadata. The camera 200 may include an acceleration sensor, an angular velocity sensor, a gyro sensor, and the like as the motion sensor, and in this case, the camera 200 may add metadata and motion detection information to the image data. Furthermore, the camera 200 may include, as a distance sensor, light detection and ranging or laser imaging detection and ranging (LiDAR), a time of flight (ToF) sensor, a stereo camera, a structured light camera, or the like.

[0032] The camera 200 may have a known function (subject detection function, face detection function, scene detection function, and the like) capable of detecting various types of information from image data obtained by imaging. In a case where the camera 200 has such an information detection function, the camera 200 can add the detected information to the image data as metadata. These information detection functions can be realized by a method based on machine learning or deep learning, a method based on template matching, a matching method based on luminance distribution information of a subject, a method using artificial intelligence, instance segmentation, and the like.

[0033] Fig. 3 is a diagram illustrating an example of a structure of a frame image constituting image data or moving image data as imaging data. The image data includes FS, Embedded Data Lines, Image Pixels, and MetaData, and general metadata such as an imaging time, an exposure time, and a white balance is stored in Embedded Data Lines. Further, metadata obtained by image analysis such as scene information is stored in Meta Data.

[0034] Note that, in addition to the configuration illustrated in Fig. 3, it is possible to add metadata by a method of embedding at predetermined frame intervals (for example, embedding is performed for each fixed frame in units of N frames) or embedding only in a frame in which a subject or an operation instructed by the user has occurred / ended.

[0035] Examples of the metadata include a camera parameter, an exposure time, white balance, an imaging time, an imaging date and time (season), an imaging position (for example, a detection result by a GPS sensor included in the camera 200), a posture (for example, a detection result by an IMU included in the camera 200), motion information of the camera 200 (detection result by a motion sensor such as an acceleration sensor, an angular velocity sensor, or a gyro sensor), a manufacturer of the camera 200, camera identification information (for example, the International Mobile Equipment Identifier (IMEI) number of the camera 200), camera model information, a camera person ID (for example, a login ID at the time of upload), authenticity information (for example, signature information), depth map information (for example, PhaseDetection Auto Focus (PDAF) data), polarization information, subject detection information, a name of a subject, a type of a subject, face detection information, detection information of part of a subject, scene information, distance information to a subject, a focal length, illuminance at the time of imaging, brightness of image data, a distortion parameter, and camera person information. The metadata may be any information as long as the information can be acquired by the camera 200, the information can be generated by the camera 200, and the information held by the camera 200 in advance.

[0036] Note that the plurality of cameras 200 may have different characteristics, camera parameters, signal processes, settings at the time of imaging, geometric conditions, and the like.

[0037] Note that one camera 200 may be used instead of a plurality of cameras. However, since a plurality of pieces of image data to be an input of the reconfiguration process is necessary, in a case where there is one camera 200, it is necessary for the camera 200 to perform imaging with a plurality of different positions and postures to generate a plurality of pieces of image data.

[0038] The terminal device 300 is used by a user to input an instruction related to a reconfiguration process to the information processing device 100, check and manage a reconfiguration result, and the like. The terminal device 300 transmits the instruction input from the user to the information processing device 100 as instruction information. As the terminal device 300, an electronic device such as a personal computer, a smartphone, or a tablet terminal can be used. Note that the terminal device 300 and the information processing device 100 may be configured by the same electronic device. In addition, the user may input an instruction related to generation of the reconfiguration result, and check and manage the reconfiguration result by the information processing device 100.

[0039] The first database 400 is a storage device that stores various pieces of data used by the information processing device 100 to generate a reconfiguration result. The various pieces of data are characteristics of the camera 200, a camera model used for 4D data generation, a reconfiguration processing model, and the like.

[0040] In addition, the first database 400 can hold image data, 2D data, 3D data, and the like that can be used for correction of a reconfiguration result by the information processing device 100, and can provide the image data, the 2D data, the 3D data, and the like in response to a request from the information processing device 100.

[0041] The second database 500 is a storage device that stores imaging data generated by imaging by the camera 200.

[0042] The third database 600 is a storage device that stores the reconfiguration result generated by the information processing device 100.

[0043] The first database 400, the second database 500, and the third database 600 are each configured by a server, a cloud, a personal computer, or the like. Note that the first database 400, the second database 500, and the third database 600 may be configured by the same device, server, cloud, or the like. Any or all of the first database 400, the second database 500, and the third database 600 may be configured by the same electronic device as the information processing device 100 and the terminal device 300.

[0044] Only the first database 400 may be configured by a cloud, and the information processing device 100, the camera 200, the terminal device 300, the second database 500, and the third database 600 may be locally configured. Furthermore, the processing block of the information processing device 100, the first database 400, the second database 500, and the third database 600 may be configured by a cloud, and the terminal device 300, the hardware configuration of the information processing device 100, and the camera 200 may be configured locally.

[0045] (Configuration of information processing device 100) Next, a configuration of a processing block of the information processing device 100 will be described with reference to Fig. 4.

[0046] An acquisition unit 101 acquires image data and metadata transmitted from the plurality of cameras 200 to output the acquired image data and metadata to a reconfiguration processing unit 102.

[0047] The reconfiguration processing unit 102 receives a plurality of pieces of image data as an input and generates 3D data or 4D data as a reconfiguration result by a reconfiguration process.

[0048] A correction unit 103 performs the correction process on the reconfiguration result. Examples of the correction process include a complement process of compensating for an insufficient portion of the reconfiguration result, a replacement process of replacing a configuration element of the reconfiguration result with another element, a filling process of filling part of the reconfiguration result, and an erasing process of erasing part of the reconfiguration result.

[0049] The correction unit 103 outputs the reconfiguration result subjected to the correction process, the image data used in the reconfiguration process, the metadata, and the correction-related information to a display processing unit 104. The correction-related information includes a correction portion in a reconfiguration result R, identification information of image data constituting the correction portion, position information and the posture information of a camera that captures the image data constituting the correction portion, a type of the correction process, information indicating content of the correction process, image data used for correction, a reconfiguration result, and the like.

[0050] The display processing unit 104 performs display processing for displaying the reconfiguration result corrected by the correction unit 103 and information regarding the reconfiguration result on a display unit 159 as a GUI.

[0051] An output unit 105 outputs the reconfiguration result subjected to the correction process by the correction unit 103 to the camera 200, the terminal device 300, the third database 600, other external devices, and the like via the network. In addition, the output unit 105 may output the reconfiguration result via a recording medium such as a USB flash memory or an SD memory card.

[0052] Note that the information processing device 100 may be configured to be able to output the reconfiguration result not subjected to the correction process to the outside by the output unit 105. Furthermore, the information processing device 100 may not include the output unit 105.

[0053] Next, a hardware configuration of the information processing device 100 will be described with reference to Fig. 5.

[0054] A central processing unit (CPU) 151 functions as an arithmetic processing unit that performs various processes, and controls the entire information processing device 100 and each unit. The CPU 151 executes various processes according to a program stored in a read only memory (ROM) 152 or a program loaded from a storage unit 160 to a random access memory (RAM) 153. The RAM 153 appropriately stores data and the like necessary for the CPU 151 to execute various processes. Each processing block constituting the information processing device 100 can be implemented by a processor including the CPU 151, the ROM 152, and the RAM 153 executing a program.

[0055] The CPU 151, the ROM 152, and the RAM 153 are connected to one another via a bus 154, and the bus 154 is connected to a bridge 155.

[0056] An interface 157 is connected to the bridge 155 via a bus 156.

[0057] An input unit 158, a display unit 159, a storage unit 160, a drive 161, a connection port 162, and a communication unit 163 are connected to the interface 157.

[0058] The input unit 158 is, for example, various operators and operation devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, and a remote controller. An operation by a user is detected by the input unit 158, and a signal corresponding to the input operation is interpreted by the CPU 151.

[0059] The display unit 159 is a liquid crystal display, an organic EL display, or the like that displays a video, an image, a graphical user interface (GUI) in the present technology, a message, or the like.

[0060] The storage unit 160 is, for example, a mass storage medium such as a hard disk or a flash memory. The storage unit 160 stores various applications, data, information, and the like.

[0061] A removable storage medium 164 can be connected to the information processing device 100 via the drive 161. The removable storage medium 164 includes a USB flash memory, an SD memory card, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like. The image data and the metadata can be supplied from the camera 200 to the information processing device 100 using the removable storage medium 164.

[0062] A data file such as a program used for each process can be read from the removable storage medium 164 by the drive 161. The read data file is stored in the storage unit 160. Furthermore, the program and the like read from the removable storage medium 164 are installed in the storage unit 160 as necessary. Furthermore, the information processing device 100 may transfer information and data to an external device via the removable storage medium 164.

[0063] An external connection device 165 can be connected to the information processing device 100 via a connection port 162.

[0064] The communication unit 163 includes various communication terminals and communication modules that perform communication processing via a network NW such as the Internet, wired / wireless communication with various devices, bus communication, and the like. An external device can be connected to the information processing device 100 via the communication unit 163. The communication method may be either wired or wireless. Examples of the communication method include cellular communication, 4G, 5G, Wi-Fi, Bluetooth (registered trademark), NFC, Ethernet (registered trademark), HDMI (registered trademark), and USB. The information processing device 100 can receive image data and metadata transmitted from the camera 200 through communication by the communication unit 163. Furthermore, the information processing device 100 can communicate with the terminal device 300 and the database 400 through communication by the communication unit 163.

[0065] Other external devices can be connected via the connection port 162 and the communication unit 163.

[0066] Note that it is not necessary for the information processing device 100 to include all the configurations illustrated in Fig. 5. For example, in a case where the information processing device 100 only performs a process to output 4D data to the outside, the display unit 159 is not necessary.

[0067] Functions implemented by configuration elements described herein may be implemented in a circuitry or processing circuitry, including a general purpose processor, an application specific processor, an integrated circuit, an application specific integrated circuit (ASIC), a CPU, a circuit in the related art, and / or combinations thereof programmed to implement the described functions. The processor includes a transistor and other circuits, and is regarded as circuitry or processing circuitry. The processor may be a programmed processor that executes a program stored in a memory. In the present specification, the circuitry, the unit, and the means are hardware programmed to realize the described functions or hardware executing the functions. The hardware may be any hardware disclosed herein or any hardware programmed or known to perform the described functions. In a case where the hardware is a processor regarded as a type of circuitry, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and / or the processor.

[0068] In the information processing device 100, for example, a program or an application for processing of the present technology can be installed via network communication by the communication unit 163 or the removable storage medium 164. In addition, the program and the application may be stored in the ROM 152, the storage unit 160, or the like in advance.

[0069] The information processing device 100 is configured as described above. The information processing device 100 may be configured as a single device, or may be configured by an electronic device having an information processing function and a communication function, such as a personal computer, a smartphone, or a tablet terminal. Furthermore, the information processing device 100 and the information processing method may be implemented by an electronic device having a function as a computer executing a program. The program may be installed in the electronic device in advance, or may be distributed by downloading, a storage medium, or the like and installed by a user or the like.

[0070] The information processing device 100 may be configured in a cloud server. Furthermore, the information processing device 100 may transmit the generated reconfiguration result R to an external device, a cloud server, or the like.

[0071] The cloud server is not limited to being configured by a single computer device, and may be configured by a plurality of systemized computer devices. The plurality of computer devices is systematized by, for example, a local area network (LAN) or the like. Furthermore, a plurality of computer devices disposed at remote locations may be systematized by a virtual private network (VPN) or the like using the Internet or the like. The plurality of computer devices may include a computer device as a server group (cloud) that can be used by a cloud computing service.

[0072] (Processing in information processing device 100) Next, processing in the information processing device 100 will be described with reference to Fig. 6.

[0073] In step S11, the acquisition unit 101 acquires image data and metadata transmitted from the plurality of cameras 200. The acquisition unit 101 outputs the image data and the metadata to the reconfiguration processing unit 102.

[0074] Next, in step S12, the reconfiguration processing unit 102 generates, by the reconfiguration process, 3D data or 4D data as a reconfiguration result by using the plurality of pieces of image data as inputs. The reconfiguration process can be performed by, for example, Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS), but may be performed by another already known method or may be performed by a method to be realized in the future. The reconfiguration processing unit 102 outputs the reconfiguration result to the correction unit 103.

[0075] Next, in step S13, the correction unit 103 performs the correction process on the reconfiguration result.

[0076] The correction unit 103 can complement the reconfiguration result by supplementing the whole or part of the image data with image data obtained in the past. Furthermore, the correction unit 103 can replace the configuration elements of the reconfiguration result by replacing the whole or part of the image data with image data obtained in the past. Image data obtained in the past and used for complementation and replacement is determined on the basis of metadata added to image data used for the reconfiguration process. The past image data can be obtained from the second database 500.

[0077] The correction unit 103 can complement the reconfiguration result by supplementing the whole or part of the image data with the virtual image generated by the generative artificial intelligence (AI). Furthermore, the correction unit 103 can replace the configuration elements of the reconfiguration result by replacing the whole or part of the image data with the virtual image generated by the generative AI. The virtual image data used for complementation and replacement is determined on the basis of the metadata added to the image data used for the reconfiguration process.

[0078] The correction unit 103 can supplement part of the reconfiguration result generated by the reconfiguration processing unit 102 with a reconfiguration result generated in the past. In addition, the correction unit 103 can replace the configuration elements of the reconfiguration result generated by the reconfiguration processing unit 102 by replacing them with the reconfiguration result generated in the past. In any case, the reconfiguration result generated in the past and used for complementation and replacement is determined on the basis of the metadata added to the image data used in the reconfiguration process. The past reconfiguration result can be obtained from the third database 600.

[0079] The correction process by the correction unit 103 may be performed by a known method or may be performed by a method to be realized in the future.

[0080] The correction unit 103 outputs the reconfiguration result subjected to the correction process, the image data and the metadata used in the reconfiguration process, and the correction-related information to the display processing unit 104.

[0081] Next, in step S14, the display processing unit 104 performs a process of displaying the GUI on the display unit 159. In the following description, it is assumed that the GUI is viewed by a user who uses the information processing device 100 to generate a reconfiguration result. However, as will be described in detail later, a viewer / listener or a camera person other than the user may be allowed to view the GUI.

[0082] In the GUI according to the first embodiment illustrated in Fig. 7, the reconfiguration result R is displayed, and the imaging viewpoint of the image data identified from the position and the posture of the camera 200 that has captured the image data as the input of the reconfiguration process is displayed as the information regarding the reconfiguration result by an icon IC. In Fig. 7, an automobile and the icon IC1 to the icon IC7 are displayed as the reconfiguration result R.

[0083] The display processing unit 104 identifies the imaging viewpoint on the basis of the position information and the posture information of the camera 200 as the metadata added to the image data. Note that the reconfiguration processing unit 102 may identify the imaging viewpoint as the metadata on the basis of the position information and the posture information of the camera 200 to output the imaging viewpoint information to the display processing unit 104. Furthermore, the information processing device 100 may include a dedicated processing unit that identifies an imaging viewpoint on the basis of the position information, the posture information, and the like of the camera 200 as metadata.

[0084] The icon IC indicating the imaging viewpoint is, for example, a quadrangular pyramid. The imaging position of the camera 200 is indicated by a quadrangular pyramid position. In addition, the imaging posture of the camera 200 is indicated by a quadrangular pyramid posture. A quadrangular face which is a bottom face of the quadrangular pyramid indicates a direction of the lens of the camera 200, that is, an imaging direction. Therefore, it is possible to indicate the imaging viewpoint identified from the position and the posture of the camera 200 that has captured the image data used for the reconfiguration process by the icon IC of the quadrangular pyramid. However, the icon IC is not limited to the quadrangular pyramid, and may have another shape, for example, an arrow.

[0085] The imaging viewpoints indicated by the icon IC include imaging viewpoints of image data whose quality and quantity is sufficient (uncorrected imaging viewpoints), imaging viewpoints of image data whose quality or quantity is insufficient (corrected imaging viewpoints), and imaging viewpoints without image data (corrected imaging viewpoints). Such an uncorrected imaging viewpoint and the corrected imaging viewpoint can be identified on the basis of the correction-related information. When the quality or the quantity is insufficient, it does not necessarily mean that both the quality and the quantity are insufficient, but at least one of the quality or the quantity may be insufficient.

[0086] For example, the generated reconfiguration result R and the 3D data and the 4D data rendered from the reconfiguration result R are evaluated, and it can be said that the quality and the quantity of image data corresponding to the portion of the reconfiguration result R in which the evaluation result does not satisfy the predetermined standard are insufficient. Examples of the evaluation method and the index include a signal to noise ratio (SNR), a peak signal to noise ratio (PSNR), a structural similarity (SSIM), a learned perceptual image patch similarity (LPIPS), and an F score. In addition, a standard of the quantity and the image quality (quality) of image data to be input in the reconfiguration process is set in advance, and image data that does not meet the standard can be made insufficient image data.

[0087] In Fig. 7, the icon IC1, the icon IC2, and the icon IC3 indicated by solid lines represent imaging viewpoints of image data whose quality and quantity are sufficient. The icon IC4, the icon IC5, and the icon IC7 indicated by two-dot chain lines represent imaging viewpoints (corrected imaging viewpoints) of image data whose quality and quantity are insufficient. In addition, the icon IC6 indicated by a broken line represents an imaging viewpoint (corrected imaging viewpoint) at which no image data exists. However, the quantity of icon ICs displayed in the GUI is not limited.

[0088] As a display method of distinguishing the plurality of types of imaging viewpoints, there are methods such as changing the color, changing the size, changing the shape, changing the thickness of the line, performing blinking of, and displaying a character string, of the icon IC. Note that any display method may be used as long as a plurality of types of imaging viewpoints can be distinguished. With such display of the icon IC, the user can easily grasp what kind of imaging viewpoint is displayed.

[0089] Note that all of the imaging viewpoints of the plurality of pieces of image data as inputs of the reconfiguration process may be displayed by the icon IC, or only representative imaging viewpoints may be displayed by the icon IC. The representative imaging viewpoint is, for example, an imaging viewpoint obtained as a result of thinning out a plurality of imaging viewpoints according to a predetermined rule. In a case where the quantity of image data as an input of the reconfiguration process is large, the quantity of imaging viewpoints is large. Therefore, in a case where all the imaging viewpoints are displayed with the icon IC, the visibility of the GUI is lowered, but by displaying only representative imaging viewpoints, it is possible to prevent the visibility from being lowered.

[0090] In the display of the icon IC, whether or not the insufficient imaging 2D image data from the imaging viewpoint is actually generated by the generative AI or the like is irrelevant. This is because there may be a method and an algorithm for complementing the reconfiguration result R by generating insufficient 2D image data of the imaging viewpoint by the generative AI and performing the reconfiguration process using the generated 2D image data, or there may be a method and an algorithm for directly complementing the reconfiguration result R.

[0091] The display mode (color, shape, size, etc.) of the icon IC may be changed according to the insufficient quantity of image data, the degree of quality, and the like. For example, the icon IC indicating the imaging viewpoint in which the image data is insufficient is displayed in a specific color, and the quantity of insufficiency of the image data is represented by the gradation of the color. For example, the icon IC is displayed darker as the quantity of insufficiency of the image data is larger, and the icon IC is displayed lighter as the quantity of insufficiency of the image data is smaller.

[0092] In the GUI according to the first embodiment, as illustrated in Fig. 8A, in a case where the user makes a selection input such as a cursor operation, a click, or a touch on any of the icons IC, the display processing unit 104 displays, on the display unit 159, the reconfiguration result R viewed from the imaging viewpoint indicated by the selected icon IC as illustrated in Fig. 8B. In Fig. 8A, as a result of the selection input by the user to the icon IC5 with the cursor, the reconfiguration result R is displayed from the viewpoint of the icon IC5 as illustrated in Fig. 8B. As a result, the user can confirm the state of the reconfiguration result R viewed from the selected imaging viewpoint.

[0093] In the GUI according to the first embodiment, it is possible to make an input to designate a range in which the user performs the correction process on a portion having low quality in the reconfiguration result R illustrated in Fig. 9A, an erroneous portion in the reconfiguration result R illustrated in Fig. 9B, and the like. The designation of the range includes tracing input on the touch panel, gesture input, and input for adjusting the position and the size of the frame superimposed and displayed on the reconfiguration result R. Any method may be used as long as the range can be designated.

[0094] In a case where the user makes a tracing input, a line indicating a traced trajectory may be displayed as illustrated in Figs. 9A and 9B. In addition, the range designated by the user may be colored or highlighted.

[0095] Furthermore, in the GUI of the first embodiment, a menu indicating the type of the correction process may be displayed as illustrated in Fig. 9. In the example of Fig. 9, types of correction such as complement, replace, fill, and erase are displayed. In this case, the user selects any correction process from the menu after or before the range is designated, so that the correction unit 103 executes the selected correction process. The correction unit 103 may automatically execute the correction process after the user designates the range.

[0096] In order to execute the correction process on the basis of the input of range designation from the user, it is necessary to supply information indicating the input range to the correction unit 103. The user may be allowed to input image data to be used for correction with respect to the designated range to the information processing device 100.

[0097] Furthermore, in the GUI in the first embodiment, as illustrated in Fig. 10, a correction portion in the reconfiguration result R may be displayed by coloring the displayed reconfiguration result R. In addition, the color may be changed according to the insufficient amount or the degree of quality of the image data used to generate the reconfiguration result R.

[0098] In a case where the correction process is performed on the reconfiguration result R, the boundary between the correction portion and the non-correction portion may be clear, but in a case where a technique such as 3DGS or NeRF is used, there may be a region where the correction portion and the non-correction portion are mixed. Therefore, it is possible to display a portion in which the correction portion and the non-correction portion are mixed by color gradation.

[0099] In a case where the reconfiguration result R is 4D data including time information, the display of the imaging viewpoint may be changed according to a change in the display of the 4D data with the lapse of the replay time. In the example of Fig. 11, the replay time elapses in the order of Fig. 11A, Fig. 11B, and Fig. 11C, and the viewpoint at which the reconfiguration result R is displayed changes as the replay time elapses. Therefore, the display of the icon IC indicating the imaging viewpoint also changes according to the change in the viewpoint. In a case where the reconfiguration result R is 4D data including time information, the display processing unit 104 may display a numerical value or a bar indicating the replay time in the GUI as illustrated in Fig. 11. Furthermore, in a case where the user designates a specific replay time, the display processing unit 104 may display the reconfiguration result R and the imaging viewpoint at the replay time. In a case where the user designates a specific replay time, the display processing unit 104 can display a list of imaging viewpoints for the replay time in a list format or the like.

[0100] In a case where the camera person who has viewed the GUI of the first embodiment performs additional imaging on the spot and additional image data is input, the information processing device 100 may perform the reconfiguration process again in real time. Then, the display processing unit 104 displays the new reconfiguration processing result and the new imaging viewpoint on the display unit 159 as a GUI.

[0101] The processing of the first embodiment is performed as described above. According to the first embodiment, it is possible to easily check the imaging viewpoint of the imaging data used to generate the reconfiguration result together with the reconfiguration result. As a result, the user can easily check a portion where the quality is sufficient or a portion where the quality may be deteriorated in the reconfiguration result. In addition, the user can easily check from which imaging viewpoint the image data is sufficient and from which imaging viewpoint the image data is insufficient. As a result, it is possible to reduce the labor of checking the reconfiguration result by the user.

[0102] <Second embodiment> (Processing in information processing device 100) Next, the second embodiment of the present technology will be described. Configurations of the information processing system and the information processing device 100 in the second embodiment are similar to those in the first embodiment. Furthermore, steps S11 to S13 of the processing in the information processing device 100 illustrated in Fig. 6 are similar to those of the first embodiment.

[0103] In the second embodiment, the display processing unit 104 displays the reconfiguration result on the display unit 159 in the display processing of step S14, and further displays the configuration elements of the reconfiguration result in layers as information regarding the reconfiguration result. In the following description, it is assumed that the GUI is viewed by a user who uses the information processing device 100 to generate a reconfiguration result. However, as will be described in detail later, a viewer / listener or a camera person other than the user may view the GUI of the present technology.

[0104] As illustrated in Fig. 12A, the GUI according to the second embodiment includes a reconfiguration result display region DS1, a configuration element display region DS2, and an information display region DS3.

[0105] The reconfiguration result display region DS1 is a region for displaying the reconfiguration result R generated by the reconfiguration processing unit 102.

[0106] The configuration element display region DS2 is a region in which the configuration elements of the reconfiguration result R are displayed in layers. The configuration elements of the reconfiguration result R include an original portion and a correction portion. The original portion is a portion not subjected to the correction process by the correction unit 103 in the reconfiguration result R. The correction portion is a portion subjected to the correction process by the correction unit 103 in the reconfiguration result R. In the present embodiment, it is assumed that the complement process and the replacement process are performed as correction processes on the reconfiguration result R, and the correction portion includes a complement portion and a replacement portion.

[0107] In order to realize the display of the original portion and the correction portion, the correction unit 103 outputs information indicating the original portion and the correction portion in the reconfiguration result R to the display processing unit 104.

[0108] In Fig. 12A, the original portion is displayed as a first layer, and the correction portion is displayed as a second layer.

[0109] As illustrated in Fig. 12B, it is assumed that configuration elements of the reconfiguration result R are an original portion A that is a person, a correction portion A that is a face of a replaced person, an original portion B that is a chair, a correction portion B1 that is a backrest of the complemented chair, and a correction portion B2 that is a left back leg of the complemented chair. In a case where the configuration element of the reconfiguration result R is replaced by the correction process, the original portion is in a state before the replacement. In addition, in a case where the configuration elements of the reconfiguration result R are complemented by the correction process, the original portion is a portion excluding the complement portion.

[0110] In a case where a plurality of objects is included as configuration elements in the reconfiguration result R, each of the plurality of configuration elements is displayed as an individual layer in the configuration element display region DS2. In Fig. 12, since the reconfiguration result R includes two objects of a person and a chair as configuration elements, the original portion A of the person and the original portion B of the chair are displayed as the first layer. In addition, the correction portion A which is the face of the person, the correction portion B1 which is the backrest of the chair, and the correction portion B2 which is the left back leg of the chair are displayed as the second layer.

[0111] In the configuration element display region DS2, specific names of configuration elements such as a person, a chair, a face, a backrest, and a leg may be displayed. The display of the specific name can be realized on the basis of the subject detection result as the metadata added to the image data that is the input of the reconfiguration result R.

[0112] The information display region DS3 is a region for displaying detailed information regarding the configuration element selected by the user in the configuration element display region DS2. Examples of the information displayed in the information display region DS3 include the name of the selected configuration element, metadata regarding the selected configuration element, on / off of a display method such as emphasizing display, and copyright information. However, any information may be displayed as long as the information relates to the selected configuration element.

[0113] In the GUI of the second embodiment, in a case where there is a plurality of layers or a plurality of configuration elements, the display processing unit 104 displays only the layer or the configuration element selected according to an input to select any layer or configuration element by the user in the reconfiguration result display region DS1. The layer and the configuration element can be selected in the configuration element display region DS2. For example, the selection can be performed by checking the names of the layer and the configuration element displayed in the configuration element display region DS2. However, any method may be used as long as the layer and the configuration element can be selected.

[0114] In the case that the first layer and the second layer exist, in a case where the user makes an input to select the first layer as illustrated in Fig. 13, the display processing unit 104 displays only the original portion A and the original portion B, which are the configuration elements included in the first layer, in the reconfiguration result display region DS1.

[0115] As illustrated in Fig. 14, in a case where the user makes an input to select the second layer, the display processing unit 104 displays only the correction portion A1, the correction portion B1, and the correction portion B2, which are the configuration elements included in the second layer, in the reconfiguration result display region DS1.

[0116] As illustrated in Fig. 15, when the user makes an input to select the original portion B included in the first layer and the correction portion B1 and the correction portion B2 included in the second layer, the display processing unit 104 displays only the original portion B, the correction portion B1 and the correction portion B2 in the reconfiguration result display region DS1.

[0117] The display processing unit 104 may switch between display and non-display of the configuration element display region DS2 and the information display region DS3. The display processing unit 104 may execute switching between display and non-display in accordance with the user's input, or may display or non-display the region only in a specific case (in a case where the user pays attention to a specific configuration element or portion).

[0118] In the reconfiguration result display region DS1, a specific configuration element or a specific portion of the reconfiguration result R, for example, a configuration element or a portion based on the image generated by the generative AI may be displayed in an emphasizing manner. The emphasizing method may be any method such as coloring, blinking, emphasizing an outline, or displaying an icon. In addition, a layer or a configuration element in the configuration element display region DS2 corresponding to a configuration element or a portion displayed in an emphasizing manner in the reconfiguration result display region DS1 may be displayed in an emphasizing manner.

[0119] In addition, in the display of the reconfiguration result display region DS1, a configuration element or a portion that the user pays attention to may be displayed in an emphasizing manner. The configuration element or the portion that the user pays attention to is, for example, a configuration element or a portion designated by the user, an enlarged configuration element or portion, or the like.

[0120] As illustrated in Fig. 11 in the first embodiment, 4D data including time information may be displayed in the GUI of the second embodiment, and at the time, a numerical value or a bar indicating the replay time may be displayed.

[0121] As illustrated in Fig. 16, in the configuration element display region DS2, the configuration elements of the reconfiguration result R may be displayed for respective objects (a person, an object, and the like) and configuration elements included in the objects instead of the layers. In the example of Fig. 16, an object A (person) and an object B (chair) are displayed, an original portion A and a correction portion A, which are configuration elements included in the object A (person), are displayed, and an original portion B, a correction portion B1, and a correction portion B2, which are configuration elements included in the object B (chair), are displayed. Whether or not the display in the configuration element display region DS2 is to be displayed for each object or display of the above-described layer may be selectable by the user.

[0122] The processing of the second embodiment is performed as described above. According to the second embodiment, by displaying the reconfiguration result for each layer or each configuration element, it is possible to easily check what configuration element the reconfiguration result includes. In addition, it is possible to improve the visibility and operability of the reconfiguration result and to facilitate checking, management, correction, and the like of the reconfiguration result.

[0123] <Third embodiment> (Configuration of information processing device 100) Next, the third embodiment of the present technology is described. The configuration of the information processing system in the third embodiment is similar to that in the first embodiment.

[0124] As illustrated in Fig. 17, an information processing device 100 according to the third embodiment includes the acquisition unit 101, the reconfiguration processing unit 102, the display processing unit 104, and the output unit 105. These units are similar to those of the first embodiment.

[0125] (Processing in information processing device 100) Processing of the information processing device 100 in the third embodiment will be described with reference to Fig. 18.

[0126] In the third embodiment, the display processing unit 104 displays the reconfiguration result and the information regarding the reconfiguration result on the display unit 159 as a GUI. Here, as an example, it is assumed that the information regarding the reconfiguration result is copyright information about a configuration element of the reconfiguration result.

[0127] The copyright information includes a mark or a character string indicating that a configuration element of a reconfiguration result is a work, an author name, a name of a work, a year of copyright occurrence, a year of copyright termination, and the like, but may be any information as long as the information is information regarding the work.

[0128] First, in step S31, the acquisition unit 101 acquires the image data and the metadata transmitted from the camera. The acquisition unit 101 outputs the image data and the metadata to the reconfiguration processing unit 102.

[0129] In order to realize the processing of the display processing unit 104 in the third embodiment, the camera 200 should perform the subject detection process and acquire the copyright information regarding the detected subject, and add the subject detection information and the copyright information as metadata to the image data in advance.

[0130] However, the camera 200 may perform only the subject detection process, and the information processing device 100 that has acquired the imaging data transmitted from the camera 200 may acquire copyright information regarding the subject and add the copyright information as metadata to the image data.

[0131] Furthermore, the information processing device 100 may be configured to perform the subject detection process and acquire the copyright information regarding the detected subject, and add the subject detection information and the copyright information as metadata to the image data.

[0132] The copyright information can be obtained by referring to the Internet, various copyright databases, or the like on the basis of the subject detection information. Alternatively, the copyright information may be stored in the first database 400, and the camera 200 or the information processing device 100 may acquire the copyright information by referring to the first database 400.

[0133] Next, in step S32, the reconfiguration processing unit 102 generates, by the reconfiguration process, 3D data or 4D data as a reconfiguration result by using the plurality of pieces of image data as inputs. The reconfiguration processing unit 102 adds the subject detection information and the copyright information corresponding thereto to the reconfiguration result, and the display processing unit 104 outputs the information. Note that the subject detection information and the copyright information corresponding thereto may be output from the acquisition unit 101 to the display processing unit 104.

[0134] Next, in step S33, the display processing unit 104 performs a process of displaying the GUI on the display unit 159. In the following description, it is assumed that the GUI is viewed by a user who uses the information processing device 100 to generate a reconfiguration result. However, as will be described in detail later, a viewer / listener or a camera person other than the user may view the GUI of the present technology.

[0135] Since the reconfiguration result is 3D data or 4D data, the display processing unit 104 can change the viewpoint for displaying the reconfiguration result according to the user's input, a predetermined algorithm, a preset viewpoint path, or the like.

[0136] In the present embodiment, as illustrated in Fig. 19, as an example, it is assumed that a work CW in the reconfiguration result R is a chick character. As illustrated in Fig. 19A, the display processing unit 104 does not display the copyright information while the work CW that is a configuration element of the reconfiguration result R is not displayed.

[0137] When part or all of the work CW is displayed as illustrated in Fig. 18B as a result of changing the viewpoint, the display processing unit 104 displays the copyright information on the display unit 159. As a result, the viewer / listener can easily recognize that the displayed configuration element is a copyrighted object.

[0138] In a case where there is a plurality of configuration elements in the reconfiguration result R, copyright information may be displayed in the vicinity of the work CW so that which configuration element is the work CW can be easily understood, or a mark or a sign indicating which configuration element of the reconfiguration result R is the work CW may be displayed.

[0139] As illustrated in Fig. 20, a cursor that can change the position of the cursor may be displayed in response to the user's input while superimposed on the reconfiguration result R, and the copyright information may be displayed in a case where the cursor points to the work CW or is located near the work CW.

[0140] Furthermore, the copyright information may be displayed in a case where the viewpoint of displaying the reconfiguration result R may be changed, and the ratio of the work CW, which is the configuration element of the reconfiguration result R, to the display region of the display unit 159 is equal to or greater than a predetermined amount, for example, in a case where transition is made from the state of Fig. 21A to the state of Fig. 21B. As a result, for example, the copyright information is not displayed in a case where the work CW, which is the configuration element of the reconfiguration result R, is displayed too small to be visible, and the copyright information can be displayed in a case where the work CW is displayed so large as to be visible.

[0141] As illustrated in Fig. 22A, as a result of changing the viewpoint of display of the reconfiguration result R, in a case where the ratio of the work CW, which is the configuration element of the reconfiguration result R, to the display region of the display unit 159 is a predetermined amount or more, part or all of the configuration element, which is the work CW, may be erased and hidden.

[0142] As illustrated in Fig. 22B, as a result of changing the viewpoint of display of the reconfiguration result R, in a case where the ratio of the work CW, which is the configuration element of the reconfiguration result R, to the display region of the display unit 159 is a predetermined amount or more, part or all of the configuration element, which is the work CW, may be replaced with another configuration element RP. Another configuration element RP used for the replacement may be included in the image data captured by the camera or may be generated by the generative AI.

[0143] Note that the information displayed in the third embodiment is not limited to the copyright information, and may be any information as long as the information is information regarding the reconfiguration result, such as the name, dimension, owner, and producer name of the configuration element of the reconfiguration result.

[0144] As illustrated in Fig. 11 in the first embodiment, 4D data including time information may be displayed in the GUI of the third embodiment, and at that time, a numerical value or a bar indicating the replay time may be displayed.

[0145] The processing according to the third embodiment is executed as described above. According to the third embodiment, the viewer / listener can easily grasp the information regarding the configuration elements of the reconfiguration result R. In addition, by using the information regarding the reconfiguration result R as the copyright information, it is possible to present the copyright information to the viewer / listener to execute copyright countermeasures such as prevention of copyright infringement and explicit indication of copyright.

[0146] In the third embodiment, it is not necessary to perform the correction process on the reconfiguration result, but the correction process may be performed on the reconfiguration result as in the first embodiment, and information regarding the reconfiguration result such as copyright information may be displayed.

[0147] <Utilization of present technology> Next, an example of a utilization scene of the present technology will be described. The user uses the information processing device 100 to generate 3D data or 4D data as a reconfiguration result from a plurality of pieces of imaging data generated by capturing an event with a plurality of cameras. The user provides the viewer / listener with the reconfiguration result by distribution or the like via a network. A viewer / listener can view 3D data and 4D data with an HMD, an augmented reality (AR) device such as a glasses-type wearable device or a smartphone, or a viewing device such as a spatial reproduction display.

[0148] By using the GUI of the first embodiment, the user can check the imaging viewpoint and the correction portion of the imaging data used to generate the reconfiguration result together with the reconfiguration result. In addition, the user can check the reconfiguration result and the configuration element of the reconfiguration result by using the GUI of the second embodiment. Further, the user can check the reconfiguration result and the information regarding the reconfiguration result such as the copyright information by using the GUI of the third embodiment. As a result, the user can easily check, manage, and correct the reconfiguration result.

[0149] Furthermore, the viewing device used by the viewer / listener has the function of the information processing device 100, and the GUI of the present technology is displayed on the viewing device, so that the viewer / listener can check the reconfiguration result, the imaging viewpoint of the imaging data used to generate the reconfiguration result, the configuration elements of the reconfiguration result, the copyright information, and the like. As a result, the viewer / listener can provide feedback such as a defect or a correction request in the reconfiguration result to the user. It is possible to increase the degree of satisfaction of the viewer / listener with the reconfiguration result by the user adding the imaging data or correcting the reconfiguration result using the information processing device 100 on the basis of the feedback. Note that the reconfiguration result corrected on the basis of the feedback may be provided to other viewer / listeners in addition to the viewer / listener who has provided the feedback.

[0150] Furthermore, the camera 200 used by the camera person has the function of the information processing device 100, and the GUI of the present technology is displayed on the camera 200, so that the camera person can easily check the reconfiguration result, the imaging viewpoint of the imaging data used to generate the reconfiguration result, the configuration element of the reconfiguration result, the copyright information, and the like. As a result, the camera person can provide feedback such as a defect or a correction request in the reconfiguration result to the user.

[0151] Furthermore, the viewing device may have a function as the correction unit 103 so that the viewer / listener can correct the reconfiguration result with the viewing device. Furthermore, the camera 200 may have a function as the correction unit 103 so that the camera person can correct the reconfiguration result with the camera 200.

[0152] Furthermore, an instruction for correcting the reconfiguration result may be given to the viewer / listener or the camera person. For example, in a case where the viewer / listener is at or near the location indicated by the reconfiguration result in the real world, a message for an instruction is output from the viewing device or the camera 200.

[0153] The message may be displayed on a display of the viewing device or the camera 200, or may be output as a voice from a speaker. The message is, for example, “We need a plurality of images from the west side of the building. Move 40 meters from the current location to the west side (left side). Then look 30 degrees up. Take a picture there.”.

[0154] The user who has checked the reconfiguration result may create an instruction to the viewer / listener or the camera person using an electronic device such as a personal computer or a smartphone to transmit instruction data from the electronic device to the viewing device or the camera 200. Furthermore, a method in which the information processing device 100 or another device generates an instruction by referring to the reconfiguration result and the metadata of the imaging data using AI or the like to transmit the instruction data to the viewing device may be used.

[0155] The information processing device 100 can correct the reconfiguration result by the viewer / listener or the camera person moving according to the instruction to perform imaging and transmitting new imaging data to the information processing device 100 via the network. In this manner, a reward may be given from the user to a person who has provided the imaging data according to the instruction.

[0156] Since the first to third embodiments relate to the GUI, it is possible to easily check and verify the implementation by another person by viewing the display.

[0157] <Modification> Although the embodiment of the present technology has been specifically described above, the present technology is not limited to the above-described embodiment, and various modifications based on the technical idea of the present technology are possible.

[0158] The data to be input in the reconfiguration process is not limited to the imaging data, but may be image data, moving image data (a plurality of frame images constituting the moving image data), or the like, generated by the generative AI or the like.

[0159] The present technology can be implemented by combining any two or three of the first to third embodiments.

[0160] The terminal device 300 may have the function of the display processing unit 104, and the information processing device 100 may perform the reconfiguration process and the correction process to transmit the reconfiguration result and the correction-related information to the terminal device 300, so that the terminal device 300 may display the GUI of the present technology.

[0161] In the embodiment, the description has been given assuming that the user who is the operator of the event generates the reconfiguration result using the information processing device 100, but an individual who is not the operator may generate the reconfiguration result using the information processing device 100. In addition, the user and the camera person may be the same person, the user and the viewer / listener may be the same person, or the user, the camera person, and the viewer / listener may be the same person. Furthermore, the user of the information processing device 100 of the present technology may be any person, and the use application and the purpose of the information processing device 100 are not limited to the events described above, and may be any use application and purpose.

[0162] The present technology can also have the following configurations. 　　　　(1) A system for three-dimensional reconfiguration of an object comprising: circuitry configured to in response to a first input, automatically provide on a display a three-dimensional construction of the object and a plurality of visual indicators positioned relative to the three-dimensional construction of the object, wherein each of the plurality of visual indicators is associated with previously captured image view data corresponding to different two-dimensional views of the object, and each of the plurality of visual indicators visually identifies whether or not the corresponding two-dimensional view of the object requires correction, and in response to a selection to select one of the plurality of visual indicators that visually identifies the requirement for correction, automatically transition from providing on the display the three-dimensional construction of the object to providing the selected corresponding two-dimensional view of the object, wherein the selected corresponding two-dimensional view of the object is provided on the display with an identification corresponding to the requirement for correction. 　　　　(2) The system according to (1), wherein the circuitry is configured to correct some or all error data associated with the selected corresponding two-dimensional view of the object, reconfigure the three-dimensional construction of the object using the corrected error data associated with the selected corresponding two-dimensional view of the object, and provide on the display the reconfigured three-dimensional representation of the object. 　　　　(3) The system according to (1) or (2), wherein the circuitry is configured to provide on the display the plurality of visual indicators positioned relative to the reconfigured three-dimensional representation of the object, and the plurality of visual indicators include said selected one of the plurality of visual indicators having now been updated to indicate the correction of some or all of the error data associated with the selected corresponding two-dimensional view of the object. 　　　　(4) The system according to any one of (1) to (3), wherein the circuitry is configured to acquire from a plurality of cameras the captured image view data corresponding to the different two-dimensional views of the object, and generate the three-dimensional construction of the object using the captured image view data corresponding to the different two-dimensional views of the object. 　　　　(5) The system according to any one of (1) to (4), wherein the selected corresponding two-dimensional representation of the object is provided on the display with a graphical user interface (GUI) such that said identification corresponding to the requirement for correction is an overlain portion that is overlain on the selected corresponding two-dimensional representation and identifies a specific portion of the two-dimensional representation requiring correction. 　　　　(6) The system according to any one of (1) to (5), wherein the requirement for correction of the corresponding two-dimensional representation is based on missing image capture data and / or noisy image capture data associated with the corresponding two-dimensional representation. 　　　　(7) The system according to any one of (1) to (6), wherein each of the plurality of visual indicators positioned relative to the three-dimensional representation of the object on the display visually identifies a viewing angle and direction for the corresponding two-dimensional view of the object. 　　　　(8) The system according to any one of (1) to (7), wherein a total number of the plurality of visual indicators corresponds to a total number of the two-dimensional views of the object used to generate the three-dimensional representation of the object. 　　　　(9) A method implemented using processing circuitry, comprising: in response to a selection to select one of a plurality of visual indicators respectively corresponding to a plurality of different two-dimensional views of a physical object, switching from showing a three-dimensional representation of the physical object on the display to showing a corresponding one of the two-dimensional views of the physical object associated with the selected visual indicator; correcting error data of the corresponding one of two-dimensional views of the physical object associated with the selected visual indicator; and in response to another selection, transitioning from showing the corresponding one of the two-dimensional views of the physical object to showing a reconfigured three-dimensional representation of the physical object on the display reconfigured based on said correcting the error data of the corresponding one of the two-dimensional views of the physical object associated with the selected visual indicator, wherein each of the plurality of visual indicators is positioned relative to the three-dimensional representation of the object on the display, visually identifies a viewing angle and direction for the corresponding two-dimensional view of the physical object, and visually identifies whether or not the corresponding two-dimensional view of the physical object contains missing or erroneous data. 　　　　(10) The method according to (9), further comprising, in response to a selection to a graphical user interface (GUI) to select one of the plurality of visual indicators that identifies the corresponding two-dimensional view contains the error data, automatically transition from providing on the display the three-dimensional representation of the physical object to providing on the display the selected corresponding two-dimensional view of the physical object. 　　　　(11) The method according to (9) or (10), wherein the corresponding one of two-dimensional views of the physical object, prior to showing the reconfigured three-dimensional representation of the physical object, is provided on the display with a graphical user interface (GUI) such that an overlain portion overlain on the corresponding two-dimensional representation identifies a specific portion of the two-dimensional representation having the error data to be corrected. 　　　　(12) The method according to any one of (9) to (11), wherein a total number of the plurality of visual indicators corresponds to a total number of the two-dimensional views of the object used to produce the three-dimensional representation of the object.

Claims

1. A system for three-dimensional reconfiguration of an object comprising: circuitry configured to in response to a first input, automatically provide on a display a three-dimensional construction of the object and a plurality of visual indicators positioned relative to the three-dimensional construction of the object, wherein each of the plurality of visual indicators is associated with previously captured image view data corresponding to different two-dimensional views of the object, and each of the plurality of visual indicators visually identifies whether or not the corresponding two-dimensional view of the object requires correction, and in response to a selection to select one of the plurality of visual indicators that visually identifies the requirement for correction, automatically transition from providing on the display the three-dimensional construction of the object to providing the selected corresponding two-dimensional view of the object, wherein the selected corresponding two-dimensional view of the object is provided on the display with an identification corresponding to the requirement for correction.

2. The system according to Claim 1, wherein the circuitry is configured to correct some or all error data associated with the selected corresponding two-dimensional view of the object, reconfigure the three-dimensional construction of the object using the corrected error data associated with the selected corresponding two-dimensional view of the object, and provide on the display the reconfigured three-dimensional representation of the object3. The system according to Claim 2, wherein the circuitry is configured to provide on the display the plurality of visual indicators positioned relative to the reconfigured three-dimensional representation of the object, and the plurality of visual indicators include said selected one of the plurality of visual indicators having now been updated to indicate the correction of some or all of the error data associated with the selected corresponding two-dimensional view of the object.

4. The system according to Claim 1, wherein the circuitry is configured to acquire from a plurality of cameras the captured image view data corresponding to the different two-dimensional views of the object, and generate the three-dimensional construction of the object using the captured image view data corresponding to the different two-dimensional views of the object.

5. The system according to Claim 1, wherein the selected corresponding two-dimensional representation of the object is provided on the display with a graphical user interface (GUI) such that said identification corresponding to the requirement for correction is an overlain portion that is overlain on the selected corresponding two-dimensional representation and identifies a specific portion of the two-dimensional representation requiring correction.

6. The system according to Claim 1, wherein the requirement for correction of the corresponding two-dimensional representation is based on missing image capture data and / or noisy image capture data associated with the corresponding two-dimensional representation.

7. The system according to Claim 1, wherein each of the plurality of visual indicators positioned relative to the three-dimensional representation of the object on the display visually identifies a viewing angle and direction for the corresponding two-dimensional view of the object.

8. The system according to Claim 1, wherein a total number of the plurality of visual indicators corresponds to a total number of the two-dimensional views of the object used to generate the three-dimensional representation of the object.

9. A method implemented using processing circuitry, comprising: in response to a selection to select one of a plurality of visual indicators respectively corresponding to a plurality of different two-dimensional views of a physical object, switching from showing a three-dimensional representation of the physical object on the display to showing a corresponding one of the two-dimensional views of the physical object associated with the selected visual indicator; correcting error data of the corresponding one of two-dimensional views of the physical object associated with the selected visual indicator; and in response to another selection, transitioning from showing the corresponding one of the two-dimensional views of the physical object to showing a reconfigured three-dimensional representation of the physical object on the display reconfigured based on said correcting the error data of the corresponding one of the two-dimensional views of the physical object associated with the selected visual indicator, wherein each of the plurality of visual indicators is positioned relative to the three-dimensional representation of the object on the display, visually identifies a viewing angle and direction for the corresponding two-dimensional view of the physical object, and visually identifies whether or not the corresponding two-dimensional view of the physical object contains missing or erroneous data.

10. The method according to Claim 9, further comprising, in response to a selection to a graphical user interface (GUI) to select one of the plurality of visual indicators that identifies the corresponding two-dimensional view contains the error data, automatically transition from providing on the display the three-dimensional representation of the physical object to providing on the display the selected corresponding two-dimensional view of the physical object.

11. The method according to Claim 9, wherein the corresponding one of two-dimensional views of the physical object, prior to showing the reconfigured three-dimensional representation of the physical object, is provided on the display with a graphical user interface (GUI) such that an overlain portion overlain on the corresponding two-dimensional representation identifies a specific portion of the two-dimensional representation having the error data to be corrected.

12. The method according to Claim 9, wherein a total number of the plurality of visual indicators corresponds to a total number of the two-dimensional views of the object used to produce the three-dimensional representation of the object.

Images