3D data generation system, 3D data generation method, and 3D data generation program

Abstract

The present invention makes it possible to select an optimal generation method according to the application of generated 3D data. A 3D data generation system according to the present invention comprises: a parameter estimation means for performing image processing on images obtained by image-capturing rays from a subject and estimating camera parameters in each of the plurality of images; a first analysis means for performing three-dimensional space domain analysis for analyzing rays on the basis of a three-dimensional coordinate system in a three-dimensional space in which the subject is present in the acquired images; a second analysis means for performing ray space domain analysis for analyzing rays on the basis of a ray coordinate system in a ray space defined by positions and directions through which the rays pass on the basis of the acquired images; and an output means for outputting 3D data generated for the subject using the analysis results by the first analysis unit or the analysis results by the second analysis unit or both.

Description

3D Data Generation System, 3D Data Generation Method, and 3D Data Generation Program

[0001] [Related Art] This application claims priority from Japanese Patent Application No. 2025-025398, filed on February 19, 2025, entitled "3D Data Generation System, 3D Data Generation Method, and 3D Data Generation Program", the disclosure of which is incorporated herein by reference in its entirety. This disclosure relates to a 3D data generation system, a 3D data generation method, and a 3D data generation program. [[ID="]]

[0002] As background art in this technical field, there is JP-A-2015-212926 (Patent Document 1). This publication discloses an "information processing apparatus capable of rendering a virtual space image based on a two-dimensional image". This information processing apparatus is disclosed to "assign attributes to elements included in two-dimensional image data, reflect the attributes in the data of a three-dimensional virtual space, and the attribute indicates whether to include an element in the data of the three-dimensional virtual space" (see the abstract).

[0003] JP-A-2024-008779

[0004] Patent Document 1 does not disclose means for selecting an optimal generation method according to the use of the generated 3D data. Therefore, this disclosure provides a 3D data generation system capable of selecting an optimal generation method according to the use of the generated 3D data.

[0005] To solve the above problems, for example, the configuration described in the claims is adopted.

[0006] According to this disclosure, an optimal generation method can be selected according to the use of the generated 3D data. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

[0007] Figure 1 shows an example of the overall configuration of the 3D data generation system. Figure 2 shows an example of the hardware configuration of the management server. Figure 3 shows an example of the hardware configuration of the generation server. Figure 4 shows some of the various information stored in the auxiliary storage device of the generation server. Figure 5 shows the remaining various information stored in the auxiliary storage device of the generation server. Figure 6 shows an example of the hardware configuration of the external storage. Figure 7 shows an example of the hardware configuration of the user terminal. Figure 8 shows an example of the flow of the 3D data generation process. Figure 9 shows an example of the flow of the parameter estimation process. Figure 10 shows an example of the flow of the full generation process. Figure 11 shows an example of the flow of the 3D spatial analysis process as instant generation. Figure 12 shows an example of the flow of the ray spatial domain analysis process as instant generation. Figure 13 shows the first example of the ray spatial domain analysis in Figure 11. Figure 14 shows the second example of the ray spatial domain analysis in Figure 11. Figure 15 shows an example of the analysis method selection screen. Figure 16 shows an example of a processing flow in which a 3D spatial domain analysis is performed after a ray spatial domain analysis as instant generation. Figure 17 shows an example of the 3D spatial domain analysis processing flow shown in Figure 16.

[0008] (1) The schematic diagram 1 of the embodiment shows an example of the configuration of the 3D data generation system 1 (hereinafter simply referred to as system 1). As shown in Figure 1, system 1 consists of a management server 200 (hereinafter simply referred to as management server 200), a generation server 300, and external storage 600. The management server 200 and the generation server 300 are connected to the external storage 600 and a plurality of user terminals 700 via a network. The network can be wired or wireless, and each terminal can send and receive information via the network.

[0009] Each server and terminal in System 1 may be a mobile device such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or a wearable device such as glasses, a wristwatch, or clothing. It may also be a stationary or portable computer, or a server located in the cloud or on a network. Functionally, it may be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, it may be a combination of multiple such terminals. For example, a combination of one smartphone and one wearable device can logically function as a single terminal. Other types of information processing terminals may also be used.

[0010] Each server and terminal in System 1 comprises a processor that runs an operating system, applications, and programs; main memory such as RAM (Random Access Memory); auxiliary storage such as IC cards, hard disk drives, SSDs (SolID State Drives), and flash memory; a communication control unit such as a network card, wireless communication module, or mobile communication module; input devices such as a touch panel, keyboard, mouse, voice input, and camera; and output devices such as monitors and displays. The output devices may also be devices or terminals that transmit information for output to external monitors, displays, printers, or other equipment.

[0011] The main memory stores various programs and applications (also referred to as modules or processing units), and the processor executes these programs and applications to realize each functional element of the entire system 1. These modules (processing units) may be implemented in hardware, such as through integration. Furthermore, each module may be an independent program or application, or it may be implemented as a subprogram or function within a single integrated program or application.

[0012] In this specification, each module is described as the entity (subject) that performs the processing, but in reality, the processor that processes various programs and applications (modules) executes the processing. Various databases (DBs) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a data set so that it can handle any data manipulation (e.g., extraction, addition, deletion, overwriting, etc.) from the processor or an external computer. The implementation method of the database is not limited; for example, it may be a database management system, spreadsheet software, or text files such as XML or JSON.

[0013] System 1 is a system that generates 3D data of a scene including a subject from multiple images (source images) which are a group of multi-view images capturing light rays from the subject. System 1 mainly generates 3D data using one of the following generation methods: Method 1) Three-dimensional spatial domain analysis method Method 2) Light ray spatial domain analysis method Method 3) Three-dimensional spatial domain analysis method and light ray spatial domain analysis method An overview of each of these analysis methods is described below.

[0014] <Overview of 3D Spatial Domain Analysis> 3D spatial domain analysis is an analytical method that uses multiple images of a subject as input data and estimates the 3D shape of the subject based on a 3D coordinate system in the 3D space where the subject exists. Specifically, 3D spatial domain analysis extracts feature points from multiple images, performs matching, estimates camera parameters for each image, and then generates a 3D point cloud or mesh representing the subject's shape. The generated subject shape is then rendered to continuously represent the subject's surface. The difference between the rendered image and the original image (image error, depth error, etc.) is measured, and the camera parameters and point cloud are adjusted again as needed to update the 3D data. This generates 3D data representing the subject. The 3D data generated by this method is represented in the form of a 3D model as a collection of point clouds.

[0015] In this embodiment, 3D Gaussian splatting (hereinafter referred to as the 3DGS method) is primarily used as the method for describing the shape of the subject. The 3DGS method is a technique that describes the shape of a subject by applying a Gaussian distribution to each point on the surface of the object when representing a three-dimensional space based on point cloud data. In the following description, the point cloud data updated by this method may be referred to as the 3DGS point cloud. As a method for describing the shape of the subject, for example, a method of creating a polygon mesh or surface model representing the subject from the point cloud data may also be used.

[0016] On the other hand, adopting the 3DGS method to represent the shape of the subject offers the following advantages: - Instead of meshing the 3D point cloud, each point is represented as a Gaussian distribution with parameters such as "center coordinates," "scale (spread)," and "rotation," allowing for smooth surface representation. - During rendering, each Gaussian distribution is projected into screen space, and the image is generated by superimposing semi-transparent blobs. This ensures that the Gaussian distributions overlap appropriately, making it easier to reproduce natural boundaries. - By adding color information, normal vectors, and spherical harmonic coefficients to the Gaussian distribution, light reflection characteristics and material textures can be expressed. Thus, the 3DGS method is considered more suitable for reproducing visual appearances, i.e., generating photorealistic 3D data, compared to conventional mesh rendering methods.

[0017] <Overview of Ray Space Domain Analysis> Ray space domain analysis is an analytical method that analyzes light rays based on an acquired image, using a ray coordinate system in ray space defined by the position and direction of the light rays' passage as a reference. In other words, ray space domain analysis treats light rays propagating within a scene, including the subject captured in the image, in a multidimensional manner, and represents the entire scene as a collection of light rays by parameterizing the camera position and the direction of the light rays. The 3D data generated by this method is expressed in a data format as a collection of light rays.

[0018] Specifically, in ray spatial domain analysis, multiple images of a subject are acquired from various positions and angles. Camera parameters and correspondences between images are obtained from these images and mapped to a ray coordinate system defined by the light rays. Then, the spatial path and direction of the light rays passing through each pixel are correlated, and the color and brightness of those light rays are estimated.

[0019] Ray space is defined by, for example, the following coordinate systems: • Orthogonal ray space: A ray space defined by the position and direction through which a ray passes in a plane defined in a given space. • Polar ray space: A ray space defined by where a ray traveling in any direction passes in a plane orthogonal to the ray. Note that ray space may be defined by other methods.

[0020] By employing ray space domain analysis in 3D data generation, the following advantages can be obtained: • Because the ray space is understood, high-quality image synthesis is possible even when the viewpoint on the 3D data is changed, making it suitable for synthesizing new viewpoints. • Because it contains multi-directional ray output information, output to glasses-free stereoscopic displays such as light field displays becomes possible.

[0021] Furthermore, by employing ray space domain analysis, advanced analysis utilizing ray information from multiple directions becomes possible, allowing for the reproduction of the following factors that influence how a subject and the scene containing the subject actually appear: • Material factors (matte finish / metallic finish / wood texture, etc.) • Optical factors (reflection / gloss / refraction, etc.) • Environmental factors (lighting / humidity, etc.) In other words, by using ray space domain analysis, the characteristics of the ray space, which possess a wealth of "visual appearance" information, can generate photorealistic 3D data.

[0022] The generation server 300 of System 1 can generate 3D data about the subject using either 3D spatial domain analysis or ray spatial domain analysis, or both. The specific procedure for generating 3D data will be described later.

[0023] (2) Diagram 2 of the Management Server 200 shows an example of the hardware configuration of the management server 200. The management server 200 is composed of one or more server devices located on the cloud, for example. In this embodiment, the management server 200 is represented as a single server device, but multiple server devices may cooperate to realize the functions of the management server 200.

[0024] The management server 200 manages the captured images and, in response to user instructions, causes the generation server 300 to generate 3D data. At this time, the management server 200 causes the generation server 300 to select a method for generating the 3D data according to its intended use. Details of this function will be described later.

[0025] As shown in Figure 2, the main memory 201 stores programs and applications such as the image acquisition module 211, the selection module 212, the generation instruction module 213, and the output module 214. The processor 203 executes these programs and applications to realize each functional element of the management server 200.

[0026] (Image acquisition module 211) The image acquisition module 211 acquires the captured image (capture frame) taken by the user terminal 700 and records it in the external storage 600. The image acquisition module 211 reads the attributes of the image information 611 transmitted from the user terminal 700. Details of the image attributes will be described later.

[0027] (Selection Module 212) The selection module 212 selects to perform an analysis process using either three-dimensional spatial domain analysis or ray spatial domain analysis, or both, according to predetermined selection conditions. Details of the analysis method selection process will be described later.

[0028] (Generation instruction module 213) The generation instruction module 213 responds to a generation request from the user terminal 700 and inputs a 3D data generation instruction to the generation server 300. The user operates the user terminal 700 to input the 3D data generation instruction. As a result, the generation request is input from the user terminal 700 to the management server 200.

[0029] (Output Module 214) The output module 214 outputs the execution results of each of the aforementioned modules to the user terminal 700 and the generation server 300. Details of the information output by the output module 214 will be described later.

[0030] As shown in Figure 2, the auxiliary storage device 202 stores information such as user information 221, device information 222, and shooting log information 223. Other information may also be stored in the auxiliary storage device 202.

[0031] (3) Various information stored in the management server The contents of user information 221, device information 222, and shooting log information 223 shown in Figure 2 will be explained in order.

[0032] <User Information 221> First, let's explain User Information 221. User Information 221 stores account information about users who use System 1. In this embodiment, User Information 221 is composed of a user table. New records are recorded in the user table when a user registers to use System 1 (account registration).

[0033] The user table associates a unique identifier, the user ID, with information such as the password, username, contact details, and registration date and time. The password is the one required to log in to the user account associated with the user ID. The contact details are the user's contact information (email address, social media address, etc.) corresponding to the user ID. The registration date and time stores the registration date and time of the record corresponding to the user ID.

[0034] <Device Information 222> Next, we will explain device information 222. Device information 222 stores information about the user terminal 700 (device) of a user using system 1. In this embodiment, device information 222 is composed of a device table. A new record is recorded in the device table when a user registers to start using system 1.

[0035] The device table associates a unique identifier, the device ID, with the user ID, device type, registration date and time, and other information. The user ID is the identifier of the user using the user terminal 700, which corresponds to the device ID.

[0036] The device type field stores the type of user terminal 700 (such as a smartphone or tablet) corresponding to the device ID. The registration date and time field stores the registration date and time of the record corresponding to the device ID.

[0037] <Shooting Log Information 223> Next, the shooting log information 223 will be described. The shooting log information 223 stores information about the shooting log generated in system 1. In this embodiment, the shooting log information 223 is composed of a shooting log table.

[0038] The shooting log table associates a unique identifier, the shooting log ID, with the device ID, shooting time, shooting status, registration date and time, and other information.

[0039] The device ID stores the identifier of the user terminal 700 (device) that performed the shooting, which corresponds to the shooting log ID. The user terminal 700 includes both a master device and a slave terminal device.

[0040] The "Shooting Time" field stores the time the shooting was performed, corresponding to the shooting log ID. The "Shooting Status" field stores the status of the shooting, corresponding to the shooting log ID. The shooting status can be either "Shooting Successful" or "Shooting Failed." The "Registration Date and Time" field stores the registration date and time of the record, corresponding to the shooting log ID.

[0041] (4) Configuration of the generation server 300 Next, the configuration of the generation server 300 will be described. Figure 3 shows an example of the hardware configuration of the generation server 300. The generation server 300 is composed of servers located on the cloud, for example. In this embodiment, the generation server 300 is represented as a single server device, but multiple server devices may cooperate to realize the functions of the generation server 300.

[0042] The generation server 300 mainly realizes the following functions: - A function of selecting material images from the photographed from photographed images. - A function of estimating their camera parameters from the material images. - A function of generating 3D data from the image data of the material images and the estimated camera parameters.

[0043] The camera parameters of an image include the following information: - Intrinsic parameters: Information defined for a single image, such as the focal length of the lens and distortion information when taking the image. - Extrinsic parameters: Information estimated from multiple images, such as the position of the camera and the shooting direction when taking the image.

[0044] As shown in FIG. 3, programs and applications such as an acquisition module 311, a preprocessing module 312, a parameter estimation module 313, a point cloud estimation module 314, a first analysis module 315, a second analysis module 316, and an output module 317 are stored in the main storage device 401. By the processor 403 executing these programs and applications, each functional element of the generation server 300 is realized.

[0045] (Acquisition module 311) The acquisition module 311 acquires the photographed images stored in the external storage 600. The acquisition module 311 also acquires image attributes from the images. Details of the image attributes will be described later. The acquisition module 311 is an example of the acquisition means of the present disclosure.

[0046] (Preprocessing module 312) The preprocessing module 312 performs preprocessing on the photographed images stored in the external storage 600. Details of the preprocessing on the images will be described later.

[0047] (Parameter estimation module 313) The parameter estimation module 313 performs image processing on the acquired multiple images and estimates the camera parameters for each of the multiple images. Details of the estimation procedure for the extrinsic parameters will be described later. The parameter estimation module 313 is an example of the parameter estimation means of the present disclosure.

[0048] (Point cloud estimation module 314) The point cloud estimation module 314 estimates a point cloud that constitutes a subject based on a plurality of material images and the estimated camera parameters. Details of the point cloud estimation process will be described later. The point cloud estimation module 314 is an example of the point cloud estimation means of the present disclosure.

[0049] (First analysis module 315) The first analysis module 315 performs three-dimensional space domain analysis for analyzing a light ray captured in an image based on a three-dimensional coordinate system in a three-dimensional space where a subject exists with respect to the acquired image. Details of the three-dimensional space domain analysis will be described later. The first analysis module 315 is an example of the first analysis means of the present disclosure.

[0050] (Second analysis module 316) The second analysis module 316 performs ray space domain analysis for analyzing a ray based on a ray coordinate system in a ray space defined by the position and direction through which the ray captured in the image passes with respect to the acquired image. Details of the ray space domain analysis will be described later. The second analysis module 316 is an example of the second analysis means of the present disclosure.

[0051] (Output module 317) The output module 317 outputs various types of information obtained by the processing of other functional modules to the user terminal 700 or the management server 200. The output module 317 outputs 3D data generated for the subject using either one or both of the analysis results by the first analysis module 315 and the analysis results by the second analysis module 316. Details of the 3D data output by the output module 317 will be described later. The output module 317 is an example of the output means of the present disclosure.

[0052] As shown in FIG. 3, the auxiliary storage device 302 stores information such as image attribute information 321, feature point information 322, feature point matching information 323, external parameter information 324, point cloud information 325, model attribute information 326, 3DGS information 327, and ray space information 328. Note that other information may be stored in the auxiliary storage device 302.

[0053] (5) Various Information Stored in the Generation Server 300 Figure 5 shows an example of various information stored in the auxiliary storage device 302 of the generation server 300. Note that this figure is merely an example of the data structure of various information, and equivalent information may be represented by other data structures. The contents of each of these pieces of information will be explained in order below.

[0054] <Image Attribute Information 321> First, let's explain the image attribute information 321. The image attribute information 321 stores attribute information of images taken by the user terminal 700 in system 1. In this embodiment, the image attribute information 321 is composed of an image attribute table.

[0055] The image attribute table associates the image ID, file path, trigger ID, internal parameters, terminal location, image features, shooting date and time, and registration date and time with a unique identifier, the image attribute ID. The image ID stores the identifier of the image that has the image attribute corresponding to the image attribute ID.

[0056] The file path stores the path to the storage area in the external storage 600 where the image with the image attribute corresponding to the image attribute ID is stored. The trigger ID stores the identifier of the trigger used to capture the image with the image attribute corresponding to the image attribute ID. The internal parameters store the internal parameters as image attributes corresponding to the image attribute ID. The internal parameters include the focal length of the lens and distortion information at the time the image was captured.

[0057] The terminal location field stores the location information of the user terminal 700 at the time the image with the image attribute corresponding to the image attribute ID was taken. The location information of the user terminal 700 is obtained, for example, by a GPS antenna installed on the user terminal 700. In other words, the terminal location is the shooting location of the image frame in question, and is information mainly defined by absolute coordinates in real space. The image feature field stores the image feature quantities that have the image attribute corresponding to the image attribute ID. The image feature extraction process will be described later.

[0058] The "Shooting Date and Time" field stores the date and time when the image with the corresponding image attribute ID was taken. The "Registration Date and Time" field stores the registration date and time of the record corresponding to the image attribute ID.

[0059] <Feature Point Information 322> Next, the feature point information 322 will be described. The feature point information 322 stores information about feature points extracted from the source image. In this embodiment, the feature point information 322 is composed of a feature point table.

[0060] The feature point table associates a unique identifier, the feature point ID, with the image ID, in-image coordinates, descriptor, registration date and time, and other information. The image ID stores the identifier of the image containing the feature point corresponding to the feature point ID.

[0061] The "Image Coordinates" field stores the planar position coordinates of the feature point corresponding to the feature point ID within the image. The "Descriptor" field stores information describing the feature point corresponding to the feature point ID. The "Registration Date and Time" field stores the registration date and time of the record corresponding to the feature point ID.

[0062] <Feature Point Matching Information 323> Next, the feature point matching information 323 will be described. The feature point matching information 323 stores information regarding the matching of feature points extracted from each of the multiple source images. In this embodiment, the feature point matching information 323 is composed of a matching table.

[0063] The matching table associates a unique identifier, the feature point matching ID, with the matching feature point, the matching feature point, the similarity score, the registration date and time, and other relevant information.

[0064] The matching feature point stores the identifier of the matching feature point in the feature point matching process, which corresponds to the feature point matching ID. The matched feature point stores the identifier of the matching feature point in the feature point matching process, which corresponds to the feature point matching ID.

[0065] The similarity field stores information about the similarity between the matched feature point and the matched feature point in the feature point matching process corresponding to the feature point matching ID. The registration date and time field stores the registration date and time of the record corresponding to the feature point matching ID.

[0066] <External Parameter Information 324> Next, the external parameter information 324 will be described. The external parameter information 324 stores information about the external parameters of the image handled by system 1. In this embodiment, the external parameter information 324 is composed of an external parameter table.

[0067] The external parameter table associates a unique identifier, the external parameter ID, with the image ID, estimated camera position, estimated shooting direction, registration date and time, and other information. The image ID stores the identifier of the image that has the external parameter corresponding to the external parameter ID.

[0068] The estimated camera position stores the estimated shooting position of the user terminal 700 that captured the image, which is one of the external parameters corresponding to the external parameter ID. In other words, the estimated camera position is the estimated shooting position for the image frame in question, and is information mainly defined by relative coordinates based on the positional relationship of multiple images of a specific subject. The estimated camera position is expressed in three-dimensional coordinates. The shooting position estimation process will be described later.

[0069] The estimated shooting direction stores the estimated shooting direction for the user terminal 700 that captured the image, which is one of the external parameters corresponding to the external parameter ID. The estimated shooting direction is represented by a direction expressed in a three-dimensional coordinate system. The process for estimating the shooting direction will be described later.

[0070] The registration date and time field stores the registration date and time of the record corresponding to the external parameter ID.

[0071] <Point Cloud Information 325> Next, the point cloud information 325 will be described. The point cloud information 325 stores information about the points that make up the point cloud generated by the generation server 300. In this embodiment, the point cloud information 325 is composed of a point table.

[0072] The point table associates a unique identifier, the point ID, with its 3D location, feature point matching ID, color information, session ID, registration date and time, and other information. The 3D location stores the 3D coordinates in virtual space of the point corresponding to the point ID.

[0073] The feature point matching ID stores the identifier of the feature point matching from which the point corresponding to the point ID was extracted. The color information stores the color information of the point corresponding to the point ID. The color information is represented, for example, by RGB values.

[0074] The 3D Data ID stores the identifier of the 3D data used when the point corresponding to the Point ID is used to generate the 3D data. The Registration Date and Time stores the registration date and time of the record corresponding to the Point ID.

[0075] <Model Attribute Information 326> Next, the model attribute information 326 will be described. The model attribute information 326 stores information about the attributes of the 3D data created in the generation server 300. In this embodiment, the model attribute information 326 is composed of a model attribute table.

[0076] The model attribute table associates a unique identifier, the model attribute ID, with various data such as the 3D data ID, file path, session ID, album ID, event ID, creation date and time, and registration date and time. The 3D data ID stores the identifier of the 3D data that has the model attribute corresponding to the model attribute ID.

[0077] The file path stores the path to the storage area in the external storage 600 where the 3D data with model attributes corresponding to the model attribute ID is stored. The session ID stores the identifier of the session in which the 3D data with model attributes corresponding to the model attribute ID was generated.

[0078] The Album ID contains the identifier of the album that stores the source images of the 3D data, which have model attributes corresponding to the Model Attribute ID. Note that if the 3D data was generated without using the album creation function, the Album ID will be blank.

[0079] The Event ID stores the identifier of the event in which 3D data with model attributes corresponding to the Model Attribute ID was generated. Note that if the 3D data was generated without using the event hosting function, the Event ID will be blank.

[0080] The generation date and time field stores the date and time when the 3D data with the model attribute corresponding to the model attribute ID was created. The registration date and time field stores the registration date and time of the record corresponding to the model attribute ID.

[0081] <3DGS Information 327> Next, the 3DGS information 327 will be described. The 3DGS information 327 stores information used in the 3DGS method performed in the 3D spatial domain analysis. In this embodiment, the 3DGS information 327 consists of a Gaussian distribution parameter table, a rendering information table, and a scene information table.

[0082] These three tables are mutually referenced during the 3DGS process. The Gaussian distribution parameter table holds information about the shape and position of the distribution used in splatting. The rendering information table holds information about the splatting viewpoint and rendering settings. The scene information table holds metadata and management information for the entire scene.

[0083] (Gaussian Distribution Parameter Table) The Gaussian distribution parameter table associates a unique identifier, the Gaussian distribution ID, with the scene ID, the center position of x, y, and z, the scale values ​​of x, y, and z, the rotation angle (quaternion), transparency, and parameters of spherical harmonics.

[0084] The Scene ID stores the ID of the scene associated with the Gaussian distribution, which corresponds to the Gaussian distribution ID. A scene refers to the entire area or space represented by 3D data including the subject.

[0085] The x, y, and z center positions indicate the central coordinates (mean vector) of the Gaussian distribution, corresponding to the Gaussian distribution ID. This is used as positional information in 3D space and becomes the center point of the Gaussian distribution during splatting drawing.

[0086] The x, y, and z scale values ​​store parameters that indicate how much the Gaussian distribution, corresponding to the Gaussian distribution ID, spreads in each axis direction.

[0087] The rotation angle (quaternion) stores parameters for rotating the Gaussian distribution, corresponding to the Gaussian distribution ID, in any direction. Storing it in quaternion format makes rotation calculations in three-dimensional space easier to handle.

[0088] The transparency value stores a setting that determines the opacity or transparency of the Gaussian distribution corresponding to the Gaussian distribution ID. Transparency is used in splatter display to adjust the blending of overlapping distributions.

[0089] The parameters of the spherical harmonics store coefficients used to represent not only the color of the object's surface, but also its radiative properties and illumination reflection. These spherical harmonics parameters are used in 3DGS (3D Graph Spectroscopy) to reproduce lighting and material properties.

[0090] The Gaussian distribution parameter table may also store other items. For example, the Gaussian distribution parameter table may store information about the covariance matrix.

[0091] (Rendering Information Table) The rendering information table associates the scene ID, viewpoint x coordinate, viewpoint y coordinate, viewpoint z coordinate, view matrix, etc., with the model attribute ID, which is a unique identifier.

[0092] The Scene ID stores the ID of the scene to which the rendering settings corresponding to the Rendering ID are applied.

[0093] The viewpoint x-coordinate, viewpoint y-coordinate, and viewpoint z-coordinate store the 3D coordinates of the viewpoint used for rendering, corresponding to the rendering settings in the rendering ID.

[0094] The view matrix stores matrices that include the rendering camera's orientation (rotation and translation) and projection matrix in the rendering settings, which correspond to the rendering ID.

[0095] The rendering information table may also store other items. For example, the rendering information table may store information about the rendering resolution.

[0096] (Scene Information Table) The scene information table associates the scene ID, number of points, creation date and time, metadata, etc., with the model attribute ID, which is a unique identifier.

[0097] The point count stores the total number of points in the scene corresponding to the scene ID. This is used to understand the scale when performing 3DGS and as a guideline for memory management.

[0098] The creation date and time record stores the time when the scene corresponding to the scene ID was created or updated.

[0099] Metadata stores additional information associated with a scene, corresponding to the scene ID. This may include shooting conditions, light source settings, and camera parameters. Metadata may also include annotation data. Annotation data is information that describes a subject and is displayed near its location. For example, if the subject is a historical building, information about its name and description may be displayed near the subject in the 3D data.

[0100] The scene information table may also store other items. For example, the scene information table may store information about the coordinate system and bounding box.

[0101] <Ray Spatial Information 328> Next, the ray spatial information 328 will be described. The ray spatial information 328 stores information used in the ray spatial domain analysis performed by the second analysis module 316. In this embodiment, the ray spatial information 42 is composed of basic spatial information, depth information, and error evaluation information.

[0102] These tables are referenced from each other during the ray spatial domain analysis, ensuring consistent management of results such as analysis, rendering, and error evaluation. Spatial basic information stores information about multi-viewpoint data and virtual camera settings. Depth information stores depth data obtained from the ray spatial domain analysis. Error evaluation information stores the error evaluation results obtained from the ray spatial domain analysis.

[0103] (Basic Spatial Information) Basic spatial information includes a unique identifier, the Ray Space ID, and is associated with the Ray Space ID, spatial type, spatial parameters, coordinates on the reference plane, virtual camera information, virtual camera external parameters, spatial correction parameters, reference 3D data ID, and the position of the virtual camera (direction of the rays).

[0104] The spatial type stores the type of ray space corresponding to the ray space ID. For example, it stores information about the coordinate system that represents the group of rays.

[0105] Spatial parameters store a set of parameters that define the structure and range of the ray space, corresponding to the ray space ID. These parameters include settings such as ray resolution.

[0106] The coordinates on the reference plane store positional information on the plane (or coordinate system) used as the reference when defining the ray space corresponding to the ray space ID. These serve as reference points for geometrically arranging the ray space.

[0107] The virtual camera information stores basic information about the virtual camera used when visualizing and analyzing the ray space, which corresponds to the ray space ID. For example, it includes settings such as field of view and resolution, or identifiers.

[0108] The virtual camera external parameters store external parameters that indicate the position and orientation (rotation and translation) of the virtual camera in ray space, corresponding to the ray space ID. These are used to determine the direction and manner in which the light rays are viewed.

[0109] The spatial correction parameters store a set of parameters used for distortion correction within the ray space, corresponding to the ray space ID, as well as lighting correction. These parameters are referenced when optimizing the mapping between the scene and the ray space.

[0110] The reference 3D data ID stores the ID of the 3D data generated by the 3D spatial domain analysis associated with the ray space, which corresponds to the ray space ID.

[0111] The virtual camera's position (ray direction) stores coordinates or a direction vector that indicate the virtual camera's location from which it is observing the ray (in which direction the ray is traveling) when constructing the ray space corresponding to the ray space ID.

[0112] Note that the spatial basic information may also store other items. For example, the spatial basic information may store four-dimensional parameters such as the U-V plane and S-T plane, or it may store the sampling resolution.

[0113] (Depth Information) Depth information includes a unique identifier called a Depth ID, which is associated with the image ID, depth image file path, creation date and time, etc.

[0114] The image ID stores the ID of the image from which the corresponding depth image was generated. This is used to link depth information with image information.

[0115] The depth image file path stores the path to the depth image (depth map) file corresponding to the depth ID. This path is used when the system references and reads depth images.

[0116] The generation date and time field stores the date and time when the depth image corresponding to the depth ID was generated and recorded.

[0117] Furthermore, depth information may also store other items. For example, depth information may store resolution, depth generation method, etc. Also, depth information may be temporarily stored as intermediate generated data in a series of analysis processes.

[0118] (Error Evaluation Information) Error evaluation information includes an error ID, which is a unique identifier, associated with the evaluation data ID, comparison data ID, error value, evaluation date and time, etc.

[0119] The valuation data ID stores the ID of the image or 3D data that was evaluated in the calculation of error information corresponding to the error ID. Images include source images and depth images.

[0120] The comparison data ID stores the ID of the image or 3D data used for comparison in calculating the error information corresponding to the error ID. Images include source images and depth images.

[0121] The error value stores the error value in the error information corresponding to the error ID.

[0122] The evaluation date and time record the date and time when the error information corresponding to the error ID was calculated and recorded. This is referenced when tracking the fluctuations of the error over time.

[0123] Furthermore, the error evaluation information may also store other items. For example, the error evaluation information may store information about the evaluation method. Also, the error evaluation information may be information that is temporarily stored as intermediate generated data in a series of analysis processes.

[0124] (6) Configuration of External Storage 600 Next, the configuration of the external storage 600 will be described. Figure 6 is a diagram showing an example of the hardware configuration of the external storage 600. The external storage 600 is composed of, for example, one or more storage servers located on the cloud. In this embodiment, the external storage 600 is represented as a single storage server, but multiple storage servers may cooperate to realize the functions of the external storage 600.

[0125] As shown in Figure 6, the external storage 600 comprises a storage device 601, a storage processor 602, and an input / output interface (I / O) 603. The storage device 601 mainly contains image information 611 and 3D data 612.

[0126] <Image Information 611> Image information 611 is information about images taken by the user terminal 700. Image information 611 stores all images taken by the user terminal 700, each linked to an image ID that uniquely identifies the image. The session ID is linked to the image ID.

[0127] <3D Data 612> 3D data 612 is information about the 3D data generated by the generation server 300. 3D data 612 stores all the 3D data generated by the generation server 300, each linked to a 3D data ID that uniquely identifies the 3D data.

[0128] (7) Configuration of User Terminal 700 Next, the configuration of the user terminal 700 will be described. Figure 7 is a diagram showing an example of the hardware configuration of the user terminal 700. The user terminal 700 is a terminal device having a camera 706, such as a smartphone, tablet, notebook PC, or desktop PC.

[0129] As shown in Figure 7, the main memory 701 of the user terminal 700 stores programs and applications such as the shooting module 711, the cooperation module 712, and the display module 713. The processor 703 executes these programs and applications 721 to realize each functional element of the user terminal 700.

[0130] The shooting module 711 captures images using the camera 706 in response to user input. The shooting module 711 also automatically captures images according to input triggers. The coordination module 712 works in conjunction with the management server 200 to execute the 3D data generation process.

[0131] The display module 713 displays a specified image or 3D data on the display, which is the output device 705, in response to user input. The specific display modes of the display module 713 will be described later.

[0132] (8) 3D Data Generation Process Next, the 3D data generation process of System 1 will be described. Figure 8 is a diagram showing an example of the 3D data generation process flow 800. The 3D data generation process in System 1 is the process of generating a 3D image of a subject from an image of light rays captured from that subject.

[0133] The flow shown in Figure 8 mainly illustrates the steps from preprocessing of the source image to selecting the analysis process to be executed. In this embodiment, it is possible to execute one of three different generation flows (A, B, C) depending on the user's selection or externally set conditions.

[0134] - Connector A: Proceed to the flow for full generation. - Connector B: Proceed to the flow for instant generation (3D spatial domain analysis). - Connector C: Proceed to the flow for instant generation (ray spatial domain analysis).

[0135] In connector A, "full generation" refers to a generation flow that generates 3D data using both 3D spatial domain analysis and ray spatial domain analysis. This generation flow is applicable in the following situations: • When the number of images of the subject exceeds a certain amount. • When the user requires high-precision 3D data.

[0136] Instant generation (3D spatial domain analysis) in connector B refers to a generation flow that generates 3D data using only 3D spatial domain analysis. This generation flow is applicable in the following situations: • When the number of images of the subject does not reach a certain number. • When extremely high-precision 3D data is not required, and the 3D data is intended to be used as model data.

[0137] Instant generation (ray space domain analysis) in connector C refers to a generation flow that generates 3D data using only ray space domain analysis. This generation flow is applicable in the following situations: • When the number of images of the subject does not reach a certain number. • When the 3D data is intended for use in applications where the "appearance" of the subject is important.

[0138] In other words, System 1 can automatically determine and execute the optimal generation flow from among multiple generation flows, depending on the purpose of generating the 3D image or the conditions of the image used for generation, based on user instructions or pre-set selection conditions. The procedure for generating 3D data by System 1, which selectively executes these analysis processes, is described below.

[0139] As shown in Figure 8, in step S801, the generation server 300 first acquires source images in response to the input of a 3D data generation instruction from the management server 200's generation instruction module 213. Specifically, the acquisition module 311 of the generation server 300, in response to receiving the 3D data generation instruction input from the management server 200, acquires still images stored in the external storage 600 or a series of frames captured by a camera as source images.

[0140] Next, in step S802, the generation server 300 performs preprocessing on the source image. Specifically, the preprocessing module 312 of the generation server 300 performs processing on the source image, such as noise reduction, edge enhancement, image normalization, and resolution adjustment. The preprocessing module 312 may combine multiple filtering methods, or it may perform high-precision noise estimation using deep learning.

[0141] Next, in step S803, the management server 200 selects an analysis method. Specifically, the selection module 212 of the management server 200 selects the analysis method to be used in generating 3D data. At this time, the selection module 212 uses at least one of the following pieces of information as selection conditions to determine the subsequent flow branch: • Selection instructions entered by the user • Automatic determination logic defined by the system

[0142] The determination based on user-inputted selection instructions corresponds to the case where the user inputs a selection of an analysis method during 3D data generation. In this case, the selection module 212 accepts a selection instruction from the user for either one or both of three-dimensional spatial domain analysis and ray spatial domain analysis as selection conditions, and selects the analysis process corresponding to the selected instruction. Alternatively, the user may select the attributes of the 3D data to be generated, such as the intended use of the generated 3D data, instead of selecting an analysis method.

[0143] The automatic determination logic defined by the system is the automatic determination process performed by the selection module 212. The selection module 212 determines the suitability of either one or both of the three-dimensional spatial domain analysis and the ray spatial domain analysis based on at least one of the following pieces of information as selection conditions. In other words, the selection module 212 functions as a determination means. • Number of acquired images • Number of terminals that took the images • Image attributes of the images

[0144] In the selection based on the number of images, the selection module 212 compares the number of source images with a pre-set threshold number. For example, full generation requires a certain number of source images. Therefore, if the number of source images does not exceed the threshold, the selection module 212 selects instant generation instead of full generation.

[0145] In the determination based on the number of devices that captured images, the selection module 212 compares the number of devices that captured the source images with a pre-set threshold number of devices. For example, full generation requires source images captured from a certain number of locations. Therefore, if the number of devices that captured the source images does not exceed the threshold, the selection module 212 selects instant generation instead of full generation.

[0146] In the determination based on the image attributes of the image in question, the selection module 212 uses one or more data items stored in the image attribute table as image attribute information 321 for the determination. Specifically, the selection module 212 selects an appropriate generation flow for the attributes of the image group by comparing the image attribute values ​​(collective values) of each image in the group of source images with pre-set determination conditions. Multiple determination conditions may be set. Then, the selection module 212 selects either one or both of three-dimensional spatial domain analysis and ray spatial domain analysis according to the determination result of the determination means.

[0147] Furthermore, the selection module 212 may perform the selection of the generation flow in combination with the determination based on the selection instructions entered by the user as described above. In this embodiment, the determination of selection condition 1 (step S804) is performed by automatic determination based on the number of images. Then, for selection condition 2 (step S805), the determination is performed based on the selection instructions from the user, as will be described later in Figure 17.

[0148] In other words, in step S804, where the decision is made based on selection condition 1, if the source image exceeds the threshold (Yes in step S804), the full generation flow is selected. In this case, after parameter estimation (step S900), processing proceeds to connector A. Parameter estimation (step S900) will be described later.

[0149] On the other hand, in step S804, if the source image does not exceed the threshold (No. in step S804), instant generation is selected. Next, in step S805, which is determined by selection condition 2, either three-dimensional spatial domain analysis or ray spatial domain analysis is selected depending on the intended use of the 3D data selected by the user. In the case of three-dimensional spatial domain analysis, processing proceeds to connector B, and in the case of ray spatial domain analysis, processing proceeds to connector C. With this, the processing in Figure 8 is completed.

[0150] In other words, in this embodiment, System 1 selects the richest full generation process when there are many images, and a relatively lightweight process when there are few images. Furthermore, depending on the intended use of the generated 3D data, it is also possible to perform analyses specialized for scene brightness distribution and viewpoint synthesis. In any branching path, after completing the predetermined analysis steps described later, a 3D model, viewpoint synthesis image, etc., will ultimately be generated.

[0151] (9) Parameter Estimation Process Next, we will explain the parameter estimation process 900 shown in Figure 8. Figure 9 is a diagram showing an example of the flow of the parameter estimation process 900.

[0152] As shown in Figure 9, in the parameter estimation process, feature points are first extracted from the source image in step S901. Specifically, the parameter estimation module 313 of the generation server 300 can use the following methods for extracting feature points: • Methods such as corner detection and edge detection • Extraction methods such as SIFT and SURF • Methods using deep neural networks such as CNN and Transformer

[0153] Next, in step S902, the correspondence between feature points is estimated. Specifically, pairs corresponding to the same feature point on the same object are searched for between consecutive frames or between multiple different viewpoint images. When matching pairs, the similarity of feature vectors and geometric constraints are considered. Specific methods for estimating pixels with similar features include, for example: • A method that searches for feature points that are close to each other in a feature vector space where each feature is represented as a vector. • A method that calculates the probability of how close each feature point is through a trained deep neural network. This yields feature point matching information 323, and a new record is recorded in the feature point matching table. In other words, matched feature points correspond to corresponding points.

[0154] Next, in step S903, camera parameters are estimated. Methods for estimating the initial intrinsic parameters include: • Calculating the lens focal length based on the size of the source image (for example, a constant multiple of the larger of the image's width and height) and assuming no lens distortion. • Estimating the lens focal length and distortion information through a trained deep neural network. • In the case of full generation where the number of images exceeds a certain limit, estimating the lens focal length and distortion information based on the correspondence information of feature points. The estimated initial intrinsic parameters are recorded in the image attribute table as image attribute information 321.

[0155] Furthermore, when estimating external parameters among the camera parameters, the parameter estimation module 313 geometrically calculates the external parameters based on the feature point matching information 323 between the source images and the internal parameters. As a result of this process, a new record is recorded in the external parameter table as external parameter information 324.

[0156] Next, in step S904, the generation server 300 estimates a 3D point for each corresponding point. Specifically, the point cloud estimation module 314 of the generation server 300 calculates a 3D point triangulated based on the feature point matching information 323 between images and known camera parameters. This process records a new record in the point table as point cloud information 325. This generates a sparse point cloud.

[0157] (10) Full generation process Next, the full generation process of System 1 will be described. Figure 10 is a diagram showing the flow 1000 of the 3D data generation process. This process follows connector A in Figure 8.

[0158] The processes shown in Figure 10 can be broadly divided into two categories: three-dimensional spatial domain analysis (S1002-S1005) and ray spatial domain analysis (S1006-S1010).

[0159] As shown in Figure 10, the full generation flow is divided into two parts: 3D spatial domain analysis and ray spatial domain analysis. First, in step S1001, the acquisition module 311 of the generation server 300 acquires the parameter information and point cloud information estimated in the previous processing. The acquisition module 311 also acquires the source image.

[0160] Furthermore, the parameter estimation module 313 of the generation server 300 pre-calculates camera parameter information for the virtual camera to be used in the subsequent ray space domain analysis. The virtual camera differs depending on whether an orthogonal ray space or a polar coordinate ray space is constructed. The processing in each coordinate system will be explained in order.

[0161] When constructing an orthogonal ray space, the parameter estimation module 313 sets up N virtual cameras for each virtual camera, with the orientation of the virtual camera remaining unchanged but translated horizontally in the camera's coordinate system. Then, for each virtual camera, it sets up N cameras, with the orientation of the virtual camera remaining unchanged but translated vertically in the camera's coordinate system. The information of the set virtual cameras is recorded as a new record in the basic spatial information.

[0162] On the other hand, when constructing a polar coordinate ray space, the parameter estimation module 313 sets up to M camera pairs for each camera that have a high degree of overlap of subjects captured in the image (in this case, each camera pair consists of two cameras: the camera of interest and the reference camera). Then, for each pair, it estimates the point on the optical axis of the camera of interest that is closest to the optical axis of the reference camera.

[0163] The parameter estimation module 313 sets a sphere with the estimated point as its center and the distance from that point to the camera of interest as its radius. It then calculates the point closest to the reference camera among the intersections of this sphere and the optical axis of the reference camera (this point is called the reference point). Furthermore, it interpolates the position and orientation of a virtual camera between the start point and the end point, using the position of the camera of interest as the start point and the reference point as the end point, so that the virtual camera continues to look at the center of the previously set sphere. The information of the set virtual camera is recorded as a new record in the spatial basic information.

[0164] (10-1) Three-dimensional spatial domain analysis in full generation Next, in step S1002, the first analysis module 315 of the generation server 300 initializes spatial data (3D data) based on the acquired point cloud data as a three-dimensional spatial domain analysis. That is, the first analysis module 315 of the generation server 300 performs the 3DGS method by referring to the 3DGS information 327 and adds 3D Gaussian information (transparency, rotation angle, scale, spherical harmonics, etc.) to the sparse point cloud, which already only has color and position information. As a result, the point cloud data is updated to a 3DGS point cloud.

[0165] Next, in step S1003, the first analysis module 315 of the generation server 300 selects an arbitrary viewpoint (hereinafter referred to as the "viewpoint of interest") from among the input viewpoints in the 3DGS point cloud.

[0166] Next, in step S1004, the first analysis module 315 of the generation server 300 creates a virtual image (generated image) at the viewpoint of interest, while referring to the 3DGS point cloud generated in step S1002. Specifically, the first analysis module 315 synthesizes the image as seen from the viewpoint of interest and acquires its pixel values ​​and depth information.

[0167] Next, in step S1005, the first analysis module 315 of the generation server 300 calculates the error between the input image of the point of interest and the generated image that was generated earlier. Specifically, the first analysis module 315 calculates a weighted sum (ratio 15:85) of the L1 norm (absolute difference in image values) of the difference between the two images and the SSIM (Structural Similarity Index Measure: structural and visual similarity of the entire image) of both images. Note that any method that can calculate the gradient of the "difference" between the input image and the generated image may be used to calculate the error. - A method that compares the brightness difference of each pixel or the similarity of edges. - A method that calculates the similarity between images based on the features output by a trained neural network, such as LPIPS (Learned Perceptual Image Patch Similarity). This obtains "image error information" to be used in subsequent processing. The image error information is stored as error evaluation information. This completes the 3D spatial domain analysis. Thus, in three-dimensional spatial domain analysis, the first analysis module 315 generates an input viewpoint image based on the corrected point cloud data and calculates the error between the generated input viewpoint image and the image.

[0168] (10-2) Ray Space Domain Analysis in Full Generation Next, in step S1006, as a ray space domain analysis, the second analysis module 316 of the generation server 300 constructs a ray space in the vicinity of the viewpoint of interest. Specifically, the second analysis module 316 constructs a ray space relating to rays from the subject using the image and camera parameters of a virtual camera in the ray space calculated based on camera parameters. That is, the second analysis module 316 aggregates ray information that can be obtained from nearby viewpoints and represents the three-dimensional space as a ray space domain. In this embodiment, the trajectory and intensity of rays are centrally managed by combining captured images from multiple viewpoints and their calibration information. The information of the constructed ray space is recorded as a new record of the basic spatial information.

[0169] Next, in step S1007, the second analysis module 316 of the generation server 300 estimates the structural likelihood based on the constructed ray space. Structural likelihood is an index that indicates the consistency of the object surface and shape within the ray space, and is used to evaluate which three-dimensional structure is most consistent with the ray distribution. A high likelihood value indicates a more plausible structure, and a low likelihood indicates the opposite. However, when updating the spatial data (3DGS point cloud), the loss function is optimized to be minimized, so the output is ultimately the reciprocal of the likelihood.

[0170] The second analysis module 316 calculates the extent to which the ray space constructed from spatial data (3DGS point cloud) possesses structural features (linear or waveform structures) unique to the ray space. For example, a filter bank with kernels representing various linear or waveform shapes may be used to evaluate the variance of how concentrated the values ​​are in each direction or waveform. Alternatively, the second analysis module 316 may train a neural network classifier capable of discriminating the structure of the ray space using adversarial learning or similar methods, and use it to evaluate the structure likelihood.

[0171] Next, in step S1008, the second analysis module 316 of the generation server 300 estimates depth information (first depth) for the viewpoint of interest based on the ray space. The second analysis module 316 may estimate the depth of the viewpoint of interest in a multi-view stereo manner for the constructed ray space (virtual camera). Alternatively, focusing on the structure of the ray space, the depth may be determined by the following methods: - If the ray space has a linear structure, the depth may be determined by estimating the slope of the line. - If the ray space has a waveform structure, the depth may be determined by estimating the amplitude of the waveform. Note that each depth estimation approach may be estimated using a trained neural network or using a pre-set method. The depth information estimated in this way is recorded as a new record as depth information.

[0172] Next, in step S1009, the second analysis module 316 of the generation server 300 estimates depth information (second depth) for the viewpoint of interest based on the already generated spatial data (3DGS point cloud). The second analysis module 316 projects the 3DGS point cloud onto the image plane of interest and generates depth based on that. In other words, the second analysis module 316 performs depth estimation in a three-dimensional spatial domain different from the ray spatial domain. The estimated depth information is recorded as a new record as depth information.

[0173] Next, in step S1010, the second analysis module 316 of the generation server 300 calculates the difference (depth error) between the first depth estimated in the ray spatial domain and the second depth generated in the spatial data domain. The second analysis module 316 calculates the error using, for example, the L1 error.

[0174] Next, in step S1011, the second analysis module 316 of the generation server 300 updates the spatial data based on the image error, depth error, and structure likelihood obtained up to this point. Specifically, the second analysis module 316 updates the spatial data (3DGS point cloud) using backpropagation based on this error information. In updating the spatial data, various parameters included in the 3DGS information 327 are adjusted and optimized. Note that at least one of the image error, depth error, and structure likelihood may be used to update the spatial data.

[0175] Next, in step S1012, the second analysis module 316 of the generation server 300 repeatedly executes the processes from steps S1003 to S1011 until a certain number of loops have been completed. The loop process here refers to the process being repeated until a threshold for error convergence is reached, or until a predetermined maximum number of iterations have been reached. If the specified number of iterations has not been reached, the process returns to step S1003 and updates the analysis results. This completes the optical spatial domain analysis.

[0176] Finally, in step S1013, the generation server 300 outputs the generated 3D data based on the analysis results obtained up to this point. In this embodiment, the output module 317 of the generation server 300 outputs 3D data. The 3D data is generated based on the camera parameters estimated by the parameter estimation module 313 and the analysis results by the first analysis module 315, and is subsequently updated based on the analysis results by the second analysis module 316. That is, in this generation flow, the output module 317 outputs 3D data as a 3DGS point cloud generated using the results of the analysis processing by the first analysis module 315. Alternatively, the output module 317 may output 3D data as information representing the ray space generated using the results of the analysis processing by the second analysis module 316.

[0177] The output module 317 then outputs the generated 3D data to the user terminal 700. The output 3D data includes updated 3D structural information, depth information, and additional attribute information obtained from ray spatial domain analysis. This allows the user to obtain a highly accurate 3D model and viewpoint composite image. With this, the full generation flow is completed.

[0178] Through the above process, 3D data is generated and stored in the external storage 600. Additionally, the generation server 300 records a new record in the model attribute table for the attribute information of the newly created 3D data.

[0179] (11) Flow of 3D spatial analysis as instant generation Next, the flow of the 3D spatial analysis process as instant generation of System 1 will be described. Figure 11 shows the flow 1100 of the 3D spatial analysis process as instant generation. This process follows the connector B in Figure 8.

[0180] The flow shown in Figure 11 broadly includes three processes: parameter estimation (S1101-S1106), point cloud estimation (S1107, S1116), and 3D spatial domain analysis (S1108-S1115).

[0181] (11-1) Parameter Estimation First, in step S1101, the acquisition module 311 of the generation server 300 acquires the source image and internal parameters. Only known information is acquired for the internal parameters. If the shooting terminal is equipped with a depth sensor such as a LiDAR sensor, the depth image obtained at the same time may be acquired. Also, if the shooting terminal is equipped with a sensor that can obtain position and orientation information at the time of shooting, such as an IMU sensor, the position and orientation information at the time of shooting may be acquired.

[0182] Next, in step S1102, the preprocessing module 312 of the generation server 300 selects images to be analyzed based on the image's capture location and time. For example, the preprocessing module 312 can select the following images for analysis: • Images with a capture location within a predetermined range • Images with a capture time within a predetermined range By selecting the images to be analyzed in this way, it is possible to prevent unrelated images from being included in the analysis.

[0183] In other words, in System 1, the preprocessing module 312 can select images to be analyzed from the source images according to pre-set selection conditions. The selection conditions can be set arbitrarily. In this embodiment, the shooting position and shooting time are used as selection conditions, but this is not limited to them. The selection conditions can be set based on any of the following information associated with the image: - Any item included in the image attribute information 321 - Any item included in the external parameter information 324 Alternatively, the selection conditions may be set by combining these items.

[0184] Next, in step S1103, the generation server 300 estimates corresponding points for pixels with similar feature quantities between each image. Specifically, the parameter estimation module 313 of the generation server 300 first calculates feature quantities for each source image. Feature quantities include, for example, the following: ・Manually designed feature quantities, such as those designed by humans, such as SIFT and SURF ・Machine learning feature quantities automatically extracted through deep neural networks such as CNN and Transformer This yields feature point information 322, and a new record is recorded in the feature point table.

[0185] The parameter estimation module 313 then estimates corresponding points for each image, where the feature quantities of the pixels are similar. Specific methods for estimating pixels with similar feature quantities include, for example: • A method that searches for feature points that are close to each other in a feature vector space where each feature quantity is represented as a vector. • A method that calculates the probability of how close each feature point is through a trained deep neural network. This yields feature point matching information 323, and a new record is added to the feature point matching table. In other words, matched feature points correspond to corresponding points.

[0186] Next, the parameter estimation module 313 of the generation server 300 performs estimation of initial internal parameters (step S1105), except when internal parameters have already been obtained (Yes in step S1104).

[0187] Methods for estimating the initial intrinsic parameters include, for example, the following: - Calculating the lens focal length based on the size of the source image (for example, a constant multiple of the larger of the image's width and height) and assuming there is no lens distortion. - Estimating the lens focal length and distortion information through a trained deep neural network. The estimated initial intrinsic parameters are recorded in the image attribute table as image attribute information 321.

[0188] Next, in step S1106, the generation server 300 estimates the external parameters (step S1006). Specifically, the parameter estimation module 313 of the generation server 300 geometrically calculates the external parameters based on the feature point matching information 323 between the source images and the internal parameters. As a result of this process, a new record is recorded in the external parameter table as external parameter information 324. This completes the parameter estimation process.

[0189] (11-2) Sparse Point Cloud Estimation Next, in step S1107, the generation server 300 estimates a 3D point for each corresponding point. Specifically, the point cloud estimation module 314 of the generation server 300 calculates a 3D point triangulated based on the feature point matching information 323 between images and known camera parameters. This process records a new record in the point table as point cloud information 325. This generates sparse point cloud data.

[0190] (11-3) Three-dimensional spatial domain analysis Next, in step S1108, the first analysis module 315 of the generation server 300 estimates a depth image for each image as a three-dimensional spatial domain analysis. Depth image estimation is performed in order to unfold a dense point cloud from sparse point cloud data. Depth estimation is performed so that a depth value can be obtained for all pixels in the image. The first analysis module 315 performs the estimation process using a deep neural network such as a CNN or a transformer.

[0191] When estimating depth images, either a monocular depth estimation network structure that estimates depth from a single image or a network structure that takes multiple view images and their camera parameters as input and uses those as input for the depth image is acceptable. However, if the depth image is estimated monocularly, it is necessary to adjust the depth image based on 3D points as described later.

[0192] In other words, while monocular depth images estimated through neural networks provide a reasonable estimate, the relative distances between subjects differ from reality. Furthermore, even if monocular depth images appear reasonable on an individual image basis, comparing the monocular depth estimation results across different images reveals inconsistencies in distance perception. Therefore, it is necessary to roughly equalize the distance consistency between images by adjusting the mean and variance of the overall monocular depth images for each image, using 3D points related to corresponding points as a reference.

[0193] Therefore, in step S1109, the first analysis module 315 of the generation server 300 determines whether the depth image estimation performed in the previous step was for a monocular image or not. Then, except when the depth estimation is not for a monocular image (Yes in step S1109), the first analysis module 315 adjusts the depth image based on the 3D points (step S1110).

[0194] Specifically, the first analysis module 315 adjusts the depth value of the monocular depth image for pixels where a 3D point has been estimated, using the depth value of the corresponding pixel in the monocular depth image and the depth value calculated from the 3D point corresponding to the corresponding pixel, so that the mean and variance are in line with the 3D point.

[0195] Next, in step S1111, the first analysis module 315 of the generation server 300 projects the corresponding pixels on each image into three-dimensional space based on the obtained depth images. If the depth value of each image and the camera's intrinsic parameters are available, the three-dimensional point in the camera coordinate system can be calculated, and if the external parameters of other cameras are available, it becomes possible to project pixels from one camera to another. Therefore, it is possible to calculate which pixel in another camera corresponds to a pixel in one camera, and to calculate the position of the pixel at the projection destination to a different camera.

[0196] Next, in step S1112, the first analysis module 315 of the generation server 300 calculates the error between the projected image and the corresponding image. Specifically, it calculates the error between the corresponding pixel position obtained by projection based on depth and camera parameters and the corresponding pixel position obtained based on feature quantities. The image error information is recorded as a new record as error evaluation information.

[0197] Next, in step S1113, the first analysis module 315 of the generation server 300 calculates the gradient of the error between the initial depth and the target depth. For example, while observing the spatial distribution and continuity of the depth, it identifies areas with shape inconsistencies and quantifies them as error indicators. The calculated error information is recorded as a new record as error evaluation information.

[0198] Next, in step S1114, the first analysis module 315 of the generation server 300 updates the camera parameters and depth from the error. Specifically, the first analysis module 315 updates the known camera parameters and depth from the calculated error. For example, the first analysis module 315 updates the camera parameters and depth based on the gradient of the calculated error. In this case, well-known stochastic gradient descent methods or improved methods such as Adam Optimizer can be used. Alternatively, the camera parameters and depth may be updated from the error calculated using a well-known error propagation method.

[0199] Next, in step S1115, the first analysis module 315 of the generation server 300 repeatedly executes the processes from steps S1108 to S1114 until a certain number of loops are reached. The number of loops here refers to the number of iterations until the error convergence threshold is reached, or up to a predetermined maximum number of iterations. If the specified number of iterations has not been reached, the process returns to step S1003 and updates the analysis results. This completes the 3D spatial domain analysis.

[0200] (11-4) Dense Point Cloud Estimation Next, in step S1116, the point cloud estimation module 314 of the generation server 300 complements the final obtained depth information by generating point cloud data and reconstructs the three-dimensional structure of the entire scene. This makes it possible to obtain a dense three-dimensional point cloud that takes depth information into account.

[0201] In this embodiment, the three-dimensional spatial domain analysis involves the first analysis module 315 estimating depth information from individual images of the source images being analyzed, and then using the estimated depth information to complement the point cloud data. By using the depth information estimated from the images to complement the point cloud data, it is possible to obtain sufficiently dense point cloud data with high accuracy, even when, for example, there are only a few source images.

[0202] Next, in step S1117, the first analysis module 315 of the generation server 300 generates 3D data from the camera parameters and point cloud. Specific methods for generating 3D data include, for example: • A method for generating a mesh model, point cloud model, or wireframe using photogrammetry. • A method for outputting a 3D Gaussian point cloud or mesh model using 3DGS.

[0203] Furthermore, the method for generating 3D data is not limited to the methods described above; any generation method that displays a 3D space or 3D objects can be arbitrarily selected. In addition, the data generated by each of these generation methods may be a mesh model, a point cloud model (including 3D Gaussian point clouds), a wireframe, or an arbitrary viewpoint image.

[0204] Then, in step S1118, the output module 317 of the generation server 300 outputs the generated 3D data to the user terminal 700. The output 3D data includes updated 3D structural information and additional attribute information obtained from depth information. This allows the user to obtain a highly accurate 3D model. With this, the 3D spatial analysis process as instant generation is completed.

[0205] Through the above process, 3D data is generated and stored in the external storage 600. Additionally, the generation server 300 records a new record in the model attribute table for the attribute information of the newly created 3D data.

[0206] (12) Flowchart of ray space domain analysis as instant generation Next, the ray space domain analysis process of System 1 as instant generation will be described. Figure 12 shows the flowchart 1200 of the ray space domain analysis process as instant generation. This process follows the connector C in Figure 8.

[0207] The flow chart 1200 shown in Figure 12 broadly includes three processes: parameter estimation processing 900, ray spatial domain analysis processing 1300 or 1400, and 3D data generation / output processing. Of these, the parameter estimation processing 900 is the same as that described in Figure 9, so a repeated explanation will be omitted.

[0208] In Figure 12, the ray spatial domain analysis performed after the parameter estimation process 900 is either the first example, flow 1300 (see Figure 13), or the second example, flow 1400 (see Figure 14). Details of each of these processes will be explained with reference to Figures 13 and 14, respectively.

[0209] (13) First Example of Ray Spatial Domain Analysis Next, we will describe the first example of ray spatial domain analysis shown in Figure 12. Figure 13 is a diagram showing the flow 1300 of the first example of ray spatial domain analysis shown in Figure 12. In this process, the model is iteratively updated by ray color estimation and error feedback using a neural network.

[0210] Specifically, as shown in Figure 13, first, in step S1301, the second analysis module 316 of the generation server 300 initializes the ray space construction model that represents the ray space. The ray space construction model is constructed using a neural network generated by machine learning. Here, the parameters of the neural network are set to their initial values.

[0211] The ray space construction model is trained to output ray density and luminance (or RGB values) in response to inputs of ray space coordinates and ray direction. By obtaining ray density and luminance, it is possible to represent what color and at what intensity light is emitted when a specific point in a scene, including a subject, is viewed from a specific direction.

[0212] Next, in step S1302, the second analysis module 316 of the generation server 300 estimates the color corresponding to each ray using a neural network. Specifically, the first analysis module 315 inputs the ray space coordinates and direction of some of the rays from the group of rays contained in the source image, which is a large viewpoint image, to the ray space construction model, thereby obtaining the output of ray density and brightness. This allows the acquisition of predicted values ​​for the color information of each ray. As a result, the density and brightness of rays passing through the entire scene including the subject are output, and information representing the ray space for that scene is obtained. Based on this, it becomes possible to synthesize an image viewed from any camera viewpoint.

[0213] Next, in step S1303, the second analysis module 316 of the generation server 300 calculates the error between the pixel value corresponding to each ray. Specifically, the second analysis module 316 uses the color information actually observed in the source image and the predicted color information of each ray output by the second analysis module 316 to calculate the difference between the predicted value and the actual image pixel value as the error.

[0214] Next, in step S1304, the second analysis module 316 of the generation server 300 updates the ray space construction model based on the calculated error. Specifically, the second analysis module 316 modifies the parameters of the ray space construction model using methods such as backpropagation, and adjusts them to obtain a more accurate ray representation in the next estimation.

[0215] Next, in step S1305, the second analysis module 316 of the generation server 300 repeats the processing from step S1302 to step S1304 until a certain number of loop iterations have been completed. The second analysis module 316 terminates the loop processing when the number of loop iterations has reached a predetermined number, or when the output from the ray space construction model has reached sufficient accuracy.

[0216] Finally, in step S1306, the output module 317 of the generation server 300 outputs 3D data generated using the ray space construction model. Specifically, the output module 317 synthesizes (renders) a 3D image from a specified arbitrary camera viewpoint, using the information representing the ray space for the entire scene including the subject obtained from the updated ray space construction model, and outputs it.

[0217] The output module 317 then outputs the generated 3D data to the user terminal 700. The output 3D data includes additional attribute information obtained from the ray spatial domain analysis. This allows the user to obtain a viewpoint composite image. Once this flow is complete, the results of the ray spatial domain analysis, generated instantly, can be used for scene visualization and subsequent image processing. This concludes flow 1300.

[0218] (14) Second example of ray space domain analysis Next, we will describe the second example of ray space domain analysis shown in Figure 12. Figure 14 is a diagram showing an example of flow 1400 of the second example of ray space domain analysis shown in Figure 12. This process is characterized by preparing the camera parameters of the viewpoint necessary for the ray space to be constructed and generating all of them using a neural network.

[0219] As shown in Figure 14, in step S1401, the second analysis module 316 of the generation server 300 transforms the input information. The input information here includes the source image and previously estimated camera parameters. In this embodiment, coordinate system transformations and luminance normalization may be performed to ensure consistency with the light space to be constructed later.

[0220] Next, in step S1402, the second analysis module 316 of the generation server 300 constructs a ray space. Specifically, the second analysis module 316 integrates ray information extracted from multiple viewpoints and frames at different times, and converts it into a form that allows for the management of the brightness and color distribution for each direction of light rays in space. This makes it possible to estimate what pixel values ​​will be obtained when viewing from a specific position in a specific direction in subsequent analyses and viewpoint synthesis.

[0221] Finally, in step S1403, the second analysis module 316 of the generation server 300 generates 3D data using the ray space information. The output module 317 then outputs the generated 3D data to the user terminal 700. The output 3D data includes additional attribute information obtained from the ray space domain analysis. This allows the user to obtain a 3D image representing a scene containing subjects synthesized from any viewpoint as a ray space. Once this flow is complete, the results of the ray space domain analysis as instant generation can be referenced and used for scene visualization and subsequent image processing. With this, flow 1400 is completed.

[0222] Furthermore, other methods may be used to construct the ray space. For example, the DIBR (Depth Image Based Rendering) method, which estimates the depth from the camera using the multi-view stereo method from multi-view images and then synthesizes an arbitrary viewpoint from that depth and the multi-view images, or the IBR (Image Based Rendering) method, which directly creates an image of an arbitrary viewpoint from multi-view images without using depth, may be employed.

[0223] (15) Screen Examples Next, we will describe an example of a screen on the user terminal 700 in System 1. Figure 15 is an example of the analysis method selection screen 1500. The selection screen 1500 is displayed on the user terminal 700 by the display module 713. The selection screen 1500 is the screen that receives the user's instruction to select an analysis process in the analysis method selection (step S803) shown in Figure 8.

[0224] As shown in Figure 15, the selection screen 1500 is configured to allow the user to select a method (analysis method) for generating 3D data. In other words, System 1 supports multiple output formats for the generated 3D data, and by selecting one of the options presented here, the user can obtain 3D data in the desired format.

[0225] The upper section 1501 of the selection screen 1500 displays "Output as a collection of rays" as a generation method corresponding to ray spatial domain analysis, along with the suitable applications for this generation method of 3D data. Selecting button B1 corresponding to this option performs ray spatial domain analysis, generating 3D data with a data format that represents the scene including the subject as a collection of ray information. Because the generated 3D data retains brightness and color from multiple viewpoints, it is suitable for cases where photorealistic rendering is desired or when outputting to stereoscopic display devices such as light field displays. Potential uses for this 3D data include, for example, advertising, exhibitions, trade fairs, education, and remote communication.

[0226] Meanwhile, in the lower section 1502 of the selection screen 1500, the option "Output as 3D model" is displayed as a generation method corresponding to 3D spatial domain analysis, along with the suitable uses for this generation method of 3D data. Selecting button B2 corresponding to this option performs 3D spatial domain analysis, generating 3D data that represents the 3D space including the subject as model data in formats such as point cloud (including 3DGS point cloud), polygon mesh, and solid data. The 3D data generated in this way is suitable for editing and processing with other modeling software and rendering engines, and is expected to be used in fields such as architectural design and game development.

[0227] In this way, System 1 allows users to select an analysis method that suits their purpose, depending on how they intend to use the 3D data. For example, if a user prioritizes photorealistic viewpoint synthesis that emphasizes the "appearance" of a scene including the subject, or if they are using a special display environment such as a stereoscopic display, they can select ray spatial domain analysis. On the other hand, if a user intends to use the data with other 3D editing software or edit and process it with modeling software or a rendering engine, they can select three-dimensional ray domain analysis.

[0228] Furthermore, when a large number of source images are required and high-precision 3D data is needed, it is possible to select an analysis method that performs both ray spatial domain analysis and 3D ray domain analysis. Therefore, in System 1, the user can select the optimal generation method according to the intended use of the generated 3D data.

[0229] Furthermore, in System 1, by using 3D spatial domain analysis and ray spatial domain analysis in combination, the following effects can be obtained: 1) Refocusing / defocusing becomes easier. Ray spatial domain analysis makes it easier to control camera blur, so it becomes possible not only to generate high-quality 3D data from blurred images, but also to perform editing and assignment with high quality.

[0230] 2) It can prevent reflections and refractions from glass and other materials. For example, when photographing a subject in a glass display case, your own reflection may appear, or a different subject may be reflected. In contrast, ray spatial domain analysis makes it easier to separate multiple images reflected in a transparent object, enabling not only the generation of high-quality 3D data from data including reflections and refractions, but also high-quality editing and addition of data.

[0231] 3) Reproduction of highly glossy subjects: Originally, it was difficult to reproduce glossy or reflective subjects in 3D photography. However, ray spatial domain analysis, which can describe and represent the "appearance" of a subject from multiple directions, can suitably represent gloss. Therefore, not only can high-quality 3D data be generated from images of glossy subjects, but editing and addition can also be done with high quality.

[0232] 4) Output to a light field display is possible. Since a light field display (i.e., a glasses-free stereoscopic display) is a device that directly displays the light ray space, it can output 3D data generated by light ray space domain analysis in System 1.

[0233] 5) Direct conversion from light field camera to 3D model is possible. Data from a light field camera, which can directly capture the light space, can be converted into a 3D model via System 1, which performs light space domain analysis.

[0234] 6) Material estimation is possible. By performing ray spatial domain analysis, 3D data with dense "visual" information from multiple directions is generated, which can be used to estimate the material.

[0235] Furthermore, System 1 can selectively perform 3D spatial domain analysis and ray spatial domain analysis to generate 3D data. Therefore, the method of generating 3D data can be automatically selected or selected by the user, depending on whether the required 3D data is in a point cloud-based data format or in a data format represented as a collection of rays.

[0236] Therefore, you can make selections according to each situation, for example, as follows: ・When you want to edit the model of the subject in a 3D virtual space, select 3D spatial domain analysis; when you want to edit the appearance of the subject in ray space, select ray space domain analysis. ・When you want to output to an external 3D rendering engine, select ray space domain analysis; when you want to directly output ray space for a 3D display, etc., select ray space domain analysis.

[0237] (16) Other Embodiments Next, other embodiments will be described. In this embodiment, a three-dimensional spatial domain analysis is performed after the ray spatial domain analysis as instant generation. Figure 16 shows a processing flow 1600 in which a three-dimensional spatial domain analysis is performed after the ray spatial domain analysis as instant generation.

[0238] In other words, Figure 16 shows the processing flow when the connector C shown in Figure 8 is selected, which includes not only ray spatial domain analysis but also 3D spatial domain analysis. The parameter estimation process 900 is the same as that shown in Figure 9, so a repeated explanation is omitted. The ray spatial domain analysis is the same as the processing flow 1300 or 1400 shown in Figures 13 and 14, excluding the "generation / output of 3D data (S1306 or S1403)" at the end, so a repeated detailed explanation is omitted.

[0239] Regardless of which flow is adopted, the second analysis module 316 performs ray spatial domain analysis, integrating ray information obtained from multiple viewpoints to estimate the radiance distribution and color information in each direction within the scene. Subsequently, the three-dimensional spatial domain analysis 1700 is executed. This process will be explained with reference to Figure 17.

[0240] Next, we will explain the 3D spatial domain analysis process 1700 shown in Figure 16. Figure 17 shows the flow chart 1700 of the 3D spatial domain analysis process shown in Figure 16. This process analyzes the 3D spatial domain in more detail using previously constructed ray space information, etc. Specifically, it illustrates a method that starts with a sparse point cloud and iteratively performs coefficient derivation using spherical harmonics and optimization of scale and rotation to ultimately obtain a precise 3D structure.

[0241] As shown in Figure 17, first, in step S1701, the first analysis module 315 of the generation server 300 generates an initial point cloud. Specifically, the first analysis module 315 of the generation server 300 can use a previously obtained sparse point cloud as its initial value. The first analysis module 315 may also generate the initial point cloud by other methods.

[0242] For example, one method is to utilize depth information from input cameras calculated during the preceding ray space generation process. Specifically, if depth information from each camera is already available, a dense point cloud can be obtained by projecting it onto the geometric space. Therefore, the first analysis module 315 can obtain an initial point cloud by randomly sampling from this point cloud until a certain number required for the initial value is reached. One of the following sampling methods can be adopted: - A completely random sampling method - A sampling method such that the spacing between each point is constant - A method of dividing the space into voxels of arbitrary size and sampling from a probability distribution corresponding to the density of the voxel point cloud

[0243] Next, in step S1702, the first analysis module 315 of the generation server 300 derives the coefficients (parameters) of the spherical harmonics for each point among the Gaussian distribution parameters. This is one example of a method for representing the local shape and radiance distribution of an object surface, and by correlating it with ray spatial information, smoother and more accurate surface characteristics can be estimated.

[0244] In other words, if ray spatial domain analysis is not performed, it is necessary to optimize the parameters of spherical harmonics and other elements for the 3DGS point cloud obtained from the multi-view input images through iterative loop processing. On the other hand, as in this embodiment, if ray spatial domain analysis is performed first and the results are utilized, the colors of light rays in each direction in space passing through a certain point are already known, making it possible to uniquely estimate the optimal parameters without the need for iterative optimization. This significantly reduces computational costs and enables the generation of 3DGS point clouds that more faithfully reproduce the actual "view". In short, by performing ray spatial domain analysis, generating multi-view images suitable for the 3DGS method, and then performing the 3DGS method using these as input data, the quality of the 3DGS method can be improved.

[0245] Next, in step S1703, the first analysis module 315 of the generation server 300 optimizes the scale value and rotation angle (quaternion) at each point among the Gaussian distribution parameters. It is desirable to perform the optimization process repeatedly. However, only the scale and rotation are updated, and the position, spherical harmonics, etc. are not updated.

[0246] Next, in step S1704, the first analysis module 315 of the generation server 300 adds new points to the point cloud or deletes unnecessary points based on the error information and required resolution obtained during the analysis process. This aims to create a structure that can sufficiently represent surface details while suppressing excessive point cloud generation. The following are possible methods for adding and deleting points: - Adding or deleting points in the same way as the 3DGS method - Projecting the already obtained geometric space onto each input multi-view image, adding points intensively to areas with large errors, and conversely deleting points in areas with no errors.

[0247] The positions of points to be added can be determined based on the depth information of each camera obtained during the generation of the ray space. A specific example of an addition method is to sample several points from areas with large errors, project them onto the image, randomly select pixels around those points that have not been projected into space, and project those pixels into space. A specific example of a deletion method is to sample several points from areas with small errors, project them onto the image, and if there are pixels in the vicinity of those points that have already been projected into space, delete the sampled points with small errors.

[0248] Next, in step S1705, the first analysis module 315 of the generation server 300 repeatedly executes the processes from steps S1701 to S1704 until a certain number of loops have been completed. Here, loop processing refers to the process being repeated until an error convergence threshold is reached, or until a predetermined maximum number of iterations are reached. Once a predetermined number of optimization loops have been performed, or the error convergence criteria are met, the loop processing is terminated.

[0249] Finally, in step S1706, the first analysis module 315 of the generation server 300 generates 3D data based on the analysis results obtained up to this point. In this embodiment, the first analysis module 315 generates 3D data by updating the initial values ​​of the 3DGS point cloud using the following information: - Sparse point cloud data obtained by parameter estimation - Analysis results of ray space analysis by the second analysis module 316 - Each parameter derived and optimized by the first analysis module 315

[0250] The output module 317 then outputs the generated 3D data to the user terminal 700. The output 3D data includes additional attribute information obtained from the ray space domain analysis. Once this flow is complete, flow 1600, which further performs 3D spatial domain analysis after the ray space domain analysis as instant generation, is finished. In the above process, the process is carried out in accordance with the general processing of 3DGS, but in processes such as generating point clouds from ray space and scaling and rotating each point, it is also possible to optimize the 3DGS point cloud in a single process without loop processing using a trained neural network.

[0251] (17) Modifications Next, modifications of System 1 will be described. In the embodiments described above, the process by which System 1 generates 3D data based on images captured by the user terminal 700 was described, but this is not limited to this. That is, System 1 may generate 3D data based on video footage captured by the user terminal 700. In this case, the image frames constituting the video footage are treated as image information 611.

[0252] Furthermore, the method of displaying the generated 3D data can be arbitrarily configured. That is, the 3D data depicting the subject in a virtual space may be rendered, and the viewpoint may be changed by the user's operation, or the image of the three-dimensionally depicted subject may be rendered and output as needed according to the viewpoint desired by the user.

[0253] Furthermore, each data table shown in the embodiments described above may be managed by a device different from the illustrated example.

[0254] Furthermore, the following may be used as criteria for determining the analysis method: • Determination based on the type of output device that outputs the generated 3D data • Determination based on the number of users (number of accounts or devices) that took the source images • Determination based on the attributes of the subject

[0255] In the determination based on the type of output device, for example, for 3D data output from a stereoscopic display, the system may be configured to select 3D data generation by ray spatial domain analysis when a connection to the stereoscopic display is detected. In the determination based on the number of users, the system may be configured to select full generation when the number of users who took the source images exceeds a predetermined threshold. In the determination based on the attributes of the subject, the system may be configured to select 3D data generation by ray spatial domain analysis when the subject is presumed to be 3D data primarily intended for viewing, such as natural landscapes or historical buildings.

[0256] Furthermore, the output module 317 of the generation server 300 may convert the data format of the generated 3D data before outputting it. That is, the output module 317 can convert 3D data, which is represented as a 3D model including a 3DGS point cloud generated using the analysis results of the first analysis module 315, into 3D data in which a scene including a subject is represented as a ray space, which is generated using the analysis results of the second analysis module 316.

[0257] Specifically, if ray space domain analysis has already been performed in the series of processes, the output module 317 uses known analysis results to cause the second analysis module 316 to generate 3D data represented as a ray space corresponding to the already generated 3D model. This process is executed, for example, in response to a data format conversion instruction from the user.

[0258] On the other hand, if ray space domain analysis has not yet been performed in the series of processes, the output module 317 causes the second analysis module 316 to perform ray space domain analysis anew, generating 3D data represented as a ray space corresponding to the already generated 3D model. This process is performed, for example, in response to a data format conversion instruction from the user.

[0259] Furthermore, the output module 317 of the generation server 300 can also output 3D data that includes a 3DGS point cloud generated using the analysis results of the first analysis module 315, after converting the 3D data, which represents a scene including the subject generated using the analysis results of the second analysis module 316 as a ray space, into 3D data. This process is performed, for example, in response to a data format conversion instruction from the user.

[0260] Specifically, the output module 317, if a three-dimensional spatial domain analysis has already been performed in a series of 3D data generation processes on the 3D data already represented as a ray space, uses the analysis results of the three-dimensional spatial domain analysis to cause the first analysis module 315 to generate 3D data as a 3D model corresponding to the already generated ray space. This process is executed, for example, in response to a data format conversion instruction from the user.

[0261] On the other hand, if a three-dimensional spatial domain analysis has not yet been performed in the series of 3D data generation processes, the output module 317 causes the first analysis module 315 to perform a new three-dimensional spatial domain analysis and generates 3D data as a 3D model corresponding to the already generated ray space. This process is performed, for example, in response to a data format conversion instruction from the user.

[0262] In this way, user convenience can be enhanced by converting the data format of 3D data that has been generated in response to subsequent user requests. The user's operation to select the output format for such already generated 3D data may be displayed as an output selection screen, for example, using a screen similar to the generation method selection screen shown in Figure 15.

[0263] (18) Others The present disclosure is not limited to the embodiments described above and includes various modifications. For example, the embodiments described above are described in detail to make the present disclosure easier to understand and are not necessarily limited to those having all the configurations described. It is also possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. Furthermore, it is possible to add, delete, replace, or combine parts of the configuration of each embodiment with other configurations.

[0264] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above configurations and functions may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD (SolID State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0265] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes and do not necessarily represent all control lines and information lines in the actual product. In practice, it can be assumed that almost all components are interconnected. The above-described embodiment discloses at least the configuration described in the claims.

[0266] 1...3D data generation system, 200...Management server, 201...Main memory, 211...Image acquisition module, 212...Selection module, 213...Generation instruction module, 214...Output module, 202...Auxiliary memory, 221...User information, 222...Device information, 223...Shooting log information, 300...Generation server, 301...Main memory, 311...Acquisition module, 312...Preprocessing module, 313...Parameter estimation module, 314...Point cloud estimation module, 315...First Analysis module, 316... Second analysis module, 317... Output module, 302... Auxiliary storage device, 321... Image attribute information, 322... Feature point information, 323... Feature point matching information, 324... External parameter information, 325... Point cloud information, 326... Generative model attribute information, 327... 3DGS information, 328... Ray space information, 600... External storage, 601... Storage device, 611... Image information, 612... 3D information, 613... Photo tag information, 614... Comment information, 700... User terminal

Claims

1. A 3D data generation system comprising: an acquisition means for acquiring images of light rays from a subject; a parameter estimation means for performing image processing on a plurality of the acquired images and estimating camera parameters for each of the plurality of images; a first analysis means for performing a three-dimensional spatial domain analysis on the acquired images, using a three-dimensional coordinate system in the three-dimensional space in which the subject exists as a reference; a second analysis means for performing a ray spatial domain analysis on the acquired images, using a ray coordinate system in the ray space defined by the position and direction through which the light rays pass as a reference; and an output means for outputting 3D data generated for the subject using either or both of the analysis results from the first analysis means and the analysis results from the second analysis means.

2. The 3D data generation system according to claim 1, further comprising point cloud estimation means for estimating a point cloud constituting the subject based on the estimated camera parameters.

3. The 3D data generation system according to claim 1 or 2, wherein the 3D data is generated based on the camera parameters estimated by the parameter estimation means and the analysis results by the first analysis means, and is further updated based on the analysis results by the second analysis means.

4. The 3D data generation system according to claim 1 or 2, wherein the 3D data is generated based on the camera parameters estimated by the parameter estimation means and the analysis results of the first analysis means using the analysis results of the second analysis means.

5. The 3D data generation system according to claim 1 or 2, further comprising selection means for selecting the execution of analysis processing by either the three-dimensional spatial domain analysis or the spatial domain analysis, or both, according to predetermined selection conditions.

6. The 3D data generation system according to claim 5, wherein the selection means receives a selection instruction from the user for either one or both of the three-dimensional spatial domain analysis and the ray spatial domain analysis, and selects an analysis process corresponding to the selection instruction.

7. The 3D data generation system according to claim 5, further comprising determination means for determining the suitability of either one or both of the three-dimensional spatial domain analysis and ray spatial domain analysis as selection conditions, based on at least one of the number of images acquired, the number of terminals that took the images, and the image attributes of the images, wherein the selection means selects either one or both of the three-dimensional spatial domain analysis and ray spatial domain analysis according to the suitability determination result by the determination means.

8. The 3D data generation system according to claim 2, wherein the 3D data is generated based on dense point cloud data that is supplemented based on the analysis results of the first analysis means to sparse point cloud data generated to depict the subject based on the estimated camera parameters.

9. The 3D data generation system according to claim 8, wherein the first analysis means estimates depth information from the image and complements the point cloud data with the estimated depth information.

10. The 3D data generation system according to claim 9, wherein the first analysis means generates an input viewpoint image based on the corrected point cloud data, and updates the 3D data based on the error between the generated input viewpoint image and the image.

11. The 3D data generation system according to claim 10, wherein the second analysis means calculates camera parameters of a virtual camera in an orthogonal ray space defined by the position and direction through which the light ray passes in a plane defined in a predetermined space, or in a polar coordinate ray space defined by where in the plane orthogonal to the light ray that travels in any direction passes, based on the image and the camera parameters, and constructs a ray space relating to the light ray from the subject.

12. The 3D data generation system according to claim 11, wherein the second analysis means calculates the structural likelihood of the constructed ray space, estimates a first depth at a viewpoint of interest based on the ray space, estimates a second depth based on the 3D data generated by the first analysis means, calculates a depth error related to the viewpoint of interest from the first depth and the second depth, and updates the 3D data generated by the first analysis means based on the calculated structural likelihood and depth error.

13. The 3D data generation system according to claim 1, wherein the output means, based on instructions from the user, converts the generated 3D data from the 3D data generated using the analysis results of the first analysis means to the 3D data generated using the analysis results of the second analysis means, and then outputs it.

14. The 3D data generation system according to claim 1, wherein the output means, based on instructions from the user, converts the generated 3D data into 3D data generated using the analysis results of the second analysis means, and outputs it.

15. A method for generating 3D data, performed by a computer processor, wherein the processor performs: an acquisition step of acquiring an image of light rays from a subject; a parameter estimation step of performing image processing on a plurality of the acquired images and estimating camera parameters in each of the plurality of images; a first analysis step of performing a three-dimensional spatial domain analysis on the acquired image, with reference to a three-dimensional coordinate system in the three-dimensional space in which the subject exists; a second analysis step of performing a ray spatial domain analysis on the acquired image, with reference to a ray coordinate system in the ray space defined by the position and direction through which the light rays pass; and an output step of outputting 3D data generated for the subject using either or both of the analysis results from the first analysis step and the analysis results from the second analysis step.

16. A 3D data generation program that is executed by a computer processor, the program causing the processor to perform: an acquisition step of acquiring an image of light rays from a subject; a parameter estimation step of performing image processing on a plurality of the acquired images and estimating camera parameters for each of the plurality of images; a first analysis step of performing a three-dimensional spatial domain analysis on the acquired image, with reference to a three-dimensional coordinate system in the three-dimensional space in which the subject exists; a second analysis step of performing a ray spatial domain analysis on the acquired image, with reference to a ray coordinate system in the ray space defined by the position and direction through which the light rays pass; and an output step of outputting 3D data generated for the subject using either or both of the analysis results from the first analysis step and the analysis results from the second analysis step.

Images