Information processing device, information processing method, and program

Abstract

An information processing device (10) comprises a processor (11) and a memory (12) connected to the processor (11). The processor (11) acquires, using a user's input with respect to a reference image in which a three-dimensional model viewed from a first viewpoint is displayed, first identification information for identifying a partial region of the three-dimensional model, determines at least one second viewpoint on the basis of the acquired first identification information, and displays at least one additional image respectively corresponding to the at least one second viewpoint. In each of the at least one additional image, the region viewed from the second viewpoint corresponding to the additional image is displayed.

Description

Information processing apparatus, information processing method, and program

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

[0002] Three-dimensional models are used in information processing related to various fields. Software that displays three-dimensional models is referred to as a three-dimensional viewer.

[0003] Conventionally, techniques for reducing input errors in coordinate data input have been disclosed (see Patent Document 1).

[0004] Japanese Unexamined Patent Publication No. 5-225317

[0005] When a user selects a three-dimensional point (also simply referred to as a point) included in a three-dimensional model using a conventional three-dimensional viewer, there is a problem that a three-dimensional point different from the three-dimensional point that the user wants to designate (in other words, the three-dimensional point that the user intends to designate) is selected.

[0006] The present disclosure provides an information processing apparatus and the like that contribute to appropriate selection of three-dimensional points.

[0007] An information processing apparatus according to one aspect of the present invention includes a circuit and a memory connected to the circuit, wherein the circuit acquires first specifying information that specifies a partial region of the three-dimensional model using a user's input for a reference image on which the three-dimensional model viewed from a first viewpoint is displayed, determines at least one second viewpoint based on the acquired first specifying information, and displays at least one additional image respectively corresponding to the at least one second viewpoint, wherein each of the at least one additional image displays the region viewed from the second viewpoint corresponding to the additional image.

[0008] Note that these general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and may also be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0009] The present disclosure contributes to appropriate selection of three-dimensional points.

[0010] This is an explanatory diagram showing the rotation of a three-dimensional model in the related technology. This is an explanatory diagram showing the enlargement of a three-dimensional model in the related technology. This is an explanatory diagram showing the movement of a three-dimensional model in the related technology. This is an explanatory diagram showing the measurement of a three-dimensional model in the related technology. This is a first explanatory diagram showing the selection of a three-dimensional point in the related technology. This is a second explanatory diagram showing the selection of a three-dimensional point in the related technology. This is a first block diagram showing the configuration of the information processing device in the embodiment. This is a second block diagram showing the configuration of the information processing device in the embodiment. This is an explanatory diagram showing the selection of a region by the information processing device in the embodiment. This is a flowchart showing a first example of processing performed by the information processing device in the embodiment. This is a flowchart showing a second example of processing performed by the information processing device in the embodiment. This is a flowchart showing a third example of processing performed by the information processing device in the embodiment. This is an explanatory diagram showing a first example of a reference image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a first example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a second example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a third example of an additional image displayed by the information processing device in the embodiment. This is a flowchart showing a fourth example of processing performed by the information processing device in the embodiment. This is an explanatory diagram showing a second example of a reference image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a fourth example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a fifth example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a sixth example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a seventh example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing an eighth example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a ninth example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a tenth example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing an eleventh example of an additional image displayed by the information processing device in the embodiment. This is a flowchart showing a fifth example of a process executed by the information processing device in the embodiment. This is an explanatory diagram showing an example of a technique for generating a three-dimensional model.This is an explanatory diagram showing a twelfth example of an additional image displayed by the information processing device in the embodiment. This is an explanatory diagram showing a thirteenth example of an additional image displayed by the information processing device in the embodiment. This is a flowchart showing an example of a process executed by the information processing device in a modified example of the embodiment.

[0011] (Knowledge that formed the basis of the present invention) The inventors identified the following problems with the three-dimensional viewer described in the "Background Art" section.

[0012] Traditionally, three-dimensional models have been used to reproduce arbitrary three-dimensional shapes on a computer or to edit such three-dimensional shapes on a computer. Software for processing three-dimensional models on a computer (also called three-dimensional data processing software) is used. This software includes a three-dimensional viewer that displays the three-dimensional model as a two-dimensional image and presents it to the user.

[0013] Figures 1A, 1B, 1C, and 1D are explanatory diagrams illustrating the processing of three-dimensional models in related technologies. Specifically, the above processing is an example of processing performed by general three-dimensional data processing software.

[0014] For example, 3D data processing software can rotate a 3D model (see Figure 1A), enlarge a 3D model (see Figure 1B), move a 3D model (see Figure 1C), and perform measurements on a 3D model (see Figure 1D). Measurements on a 3D model include, for example, measuring the distance between two points in a 3D point model (e.g., the distance between point 9A and point 9B).

[0015] Users can view, edit, or measure three-dimensional models using three-dimensional data processing software.

[0016] Examples of three-dimensional models include three-dimensional models represented by point clouds, three-dimensional models represented by meshes, three-dimensional models based on Gaussian distributions, and three-dimensional models represented by NeRF (Neural Radiance Fields).

[0017] When viewing, editing, or measuring a three-dimensional model displayed in a 3D viewer, users may select a specific three-dimensional point from among multiple three-dimensional points displayed in the 3D viewer.

[0018] However, conventionally, the 3D point selected by the 3D viewer according to the user's selection operation may differ from the 3D point the user intended to specify. In other words, the 3D viewer may select a 3D point different from the one the user intended to specify.

[0019] Figures 2 and 3 are explanatory diagrams illustrating the selection of three-dimensional points in the related technology. In the related technology, the three-dimensional point selected by the 3D viewer according to the user's selection operation may differ from the three-dimensional point that the user intended to specify.

[0020] For example, if multiple three-dimensional points included in a three-dimensional model are located at the same position when viewed from a single viewpoint, these multiple three-dimensional points will appear to overlap in the image viewed from that viewpoint on the three-dimensional viewer. An example of this case is shown in Figure 2(a).

[0021] In this case, the user may not be able to select the desired 3D point. For example, if a user wants to specify a 3D point that is further away than a 3D point that is closer, even if the user attempts to select the 3D point they want to specify, the 3D viewer may select the 3D point that is closer.

[0022] To enable the user to select a specific three-dimensional point, the 3D viewer needs to display an image from a different viewpoint (for example, Figure 2(b)). When this image is displayed, the user can select the desired three-dimensional point within that image.

[0023] Furthermore, for example, if multiple three-dimensional points included in a three-dimensional model are located close together when viewed from a single viewpoint, these multiple three-dimensional points will be displayed close together in the image viewed from that viewpoint on the three-dimensional viewer. An example of this case is shown in Figure 3.

[0024] In this case, the user may not be able to select the desired 3D point. For example, if a user wants to specify one of several adjacent 3D points, even if the user attempts to select the target 3D point, the 3D viewer may select a different 3D point from the aforementioned multiple 3D points.

[0025] To enable users to select specific three-dimensional points, the 3D viewer needs to display, for example, an image with adjusted dimensions of the displayed three-dimensional points, or an image showing an enlarged three-dimensional model. When such an image is displayed, the user can select the desired three-dimensional point within that image.

[0026] This disclosure provides an information processing device, etc., that contributes to the appropriate selection of three-dimensional points.

[0027] The following describes examples of inventions that can be obtained from the disclosures in this specification, and explains the effects and other benefits that can be obtained from such inventions.

[0028] (1) An information processing device comprising a circuit and a memory connected to the circuit, wherein the circuit acquires first identifying information that identifies a part of the three-dimensional model using user input to a reference image on which a three-dimensional model viewed from a first viewpoint is displayed, determines at least one second viewpoint based on the acquired first identifying information, displays at least one additional image corresponding to each of the at least one second viewpoint, and each of the at least one additional image displays the region as viewed from the second viewpoint corresponding to the additional image.

[0029] According to the above embodiment, the information processing device can display an additional image of a specific region identified based on user specification, viewed from a different viewpoint, thereby allowing the user to view the specific region from a viewpoint different from that of the reference image. When it is difficult for a user to select a desired three-dimensional point on the reference image, the user who has viewed the additional image can appropriately select the three-dimensional point included in the specific region on the additional image. In this way, the information processing device contributes to the appropriate selection of three-dimensional points.

[0030] (2) The information processing apparatus according to (1), wherein the circuit further acquires second identification information that identifies three-dimensional points included in the three-dimensional model using user input for at least one additional image, and performs information processing regarding the three-dimensional model using the acquired second identification information.

[0031] According to the above embodiment, the information processing device can more appropriately perform information processing related to the three-dimensional model by using information obtained using user input for additional images (corresponding to second specific information). Therefore, the information processing device can contribute to the appropriate selection of three-dimensional points and further perform information processing related to the selected three-dimensional points on the three-dimensional model.

[0032] (3) The information processing apparatus according to (2), wherein the information processing relating to the three-dimensional model includes a process for rotating the three-dimensional model, a process for moving the three-dimensional model, a process for measuring the three-dimensional model, or a process for changing the position of the viewpoint from which the three-dimensional model is viewed.

[0033] According to the above embodiment, the information processing device, as an information processing device for a three-dimensional model, can perform processes such as rotating the three-dimensional model, moving the three-dimensional model, measuring the three-dimensional model, or changing the position of the viewpoint from which the three-dimensional model is viewed. Therefore, the information processing device can contribute to the appropriate selection of three-dimensional points and can further perform information processing on the three-dimensional model related to the selected three-dimensional points.

[0034] (4) The information processing apparatus according to (1), wherein the at least one second viewpoint includes a second viewpoint in which the region of the three-dimensional model is displayed without being obscured in an additional image in which the three-dimensional model is displayed as seen from the second viewpoint.

[0035] According to the above embodiment, the information processing device displays an additional image in which a specific region is not obscured, so that the user can more reliably see the specific region displayed in the additional image and appropriately select the three-dimensional points included in the specific region on the additional image. Therefore, the information processing device contributes to the appropriate selection of three-dimensional points.

[0036] (5) The information processing apparatus according to (1), which includes a second viewpoint, wherein at least one second viewpoint is located on the normal of the plane defined by the region.

[0037] According to the above embodiment, the information processing device displays an additional image viewed from a position on the normal of a surface defined by a specific region. This allows the user to more reliably visualize the specific region displayed in the additional image and to appropriately select three-dimensional points included in the specific region on the additional image. Therefore, the information processing device contributes to the appropriate selection of three-dimensional points.

[0038] (6) The information processing apparatus according to (1), wherein the at least one second viewpoint includes two second viewpoints, the direction from each of the two second viewpoints toward the region is perpendicular to the direction from the first viewpoint toward the region, and the directions from the two second viewpoints toward the region are perpendicular to each other.

[0039] According to the above embodiment, the information processing device displays images of a specific region viewed from three orthogonal directions as a reference image and two additional images. By viewing the reference image and the two additional images, the user can more reliably visualize the specific region and appropriately select three-dimensional points included in the specific region on the additional images. Therefore, the information processing device contributes to the appropriate selection of three-dimensional points.

[0040] (7) The information processing apparatus according to (1), wherein the three-dimensional model has a predetermined upward direction, a rightward direction perpendicular to the upward direction, and a leftward direction opposite to the rightward direction, and the at least one second viewpoint includes a second viewpoint from which the three-dimensional model is viewed from above, a second viewpoint from which the three-dimensional model is viewed from the right, and a second viewpoint from which the three-dimensional model is viewed from the left.

[0041] According to the above aspect, the information processing device displays, as additional images, images of a specific region viewed from each of the upward direction, right direction, and left direction defined for the three-dimensional model. By visually recognizing the additional images, the user can more reliably visually recognize the specific region and appropriately select three-dimensional points included in the specific region on the additional images. Therefore, the information processing device contributes to appropriate selection of three-dimensional points.

[0042] (8) The information processing apparatus according to (1), wherein the three-dimensional model is a three-dimensional model generated by a learning process that receives a plurality of images in which an object viewed from a plurality of viewpoints is displayed, and the at least one second viewpoint includes a second viewpoint that is the same as one of the plurality of viewpoints.

[0043] According to the above aspect, the information processing device displays, as an additional image, an image of the specific region viewed from the same viewpoint as that of the image used in the learning process. Characteristic portions of the three-dimensional model may be displayed in the images used for the learning process. By visually recognizing the additional image of the specific region viewed from the same viewpoint as that of the image used in the learning process, the user can more reliably visually recognize the specific region and appropriately select three-dimensional points included in the specific region on the additional image. Therefore, the information processing device contributes to appropriate selection of three-dimensional points.

[0044] (9) The information processing apparatus according to (1), wherein the at least one second viewpoint includes a second viewpoint specified by a user who has visually recognized the reference image.

[0045] According to the above aspect, the information processing device displays, as an additional image, an image of the specific region viewed from a position specified by the user. The user can more reliably visually recognize the specific region from the position specified by the user themself, and can appropriately select three-dimensional points included in the specific region on the additional image. Therefore, the information processing device contributes to appropriate selection of three-dimensional points.

[0046] (10) The information processing apparatus according to (1), wherein the circuit further generates the at least one additional image to be displayed before displaying the at least one additional image, and generating one additional image included in the at least one additional image comprises: determining an angle of view of the one additional image according to a density degree of a plurality of three-dimensional points included in the region of the three-dimensional model, and generating the one additional image having the determined angle of view.

[0047] According to the above aspect, the information processing apparatus displays an additional image having an angle of view corresponding to the density degree of three-dimensional points in the specific region. A user can more reliably visually recognize the specific region and appropriately select three-dimensional points included in the specific region on the additional image by visually recognizing the additional image adjusted (for example, enlarged or reduced) to have an angle of view corresponding to the density degree. Therefore, the information processing apparatus contributes to appropriate selection of three-dimensional points.

[0048] (11) The information processing apparatus according to (10), wherein determining the angle of view of the one additional image comprises: determining a larger angle of view as the angle of view of the one additional image when a plurality of three-dimensional points included in the region of the three-dimensional model are denser.

[0049] According to the above aspect, the information processing apparatus displays an additional image having a larger angle of view when three-dimensional points are denser, according to the density degree of three-dimensional points in the specific region. A user can more reliably visually recognize the specific region and appropriately select three-dimensional points included in the specific region on the additional image by visually recognizing the additional image more appropriately adjusted (for example, enlarged or reduced) to have an angle of view corresponding to the density degree. Therefore, the information processing apparatus contributes to appropriate selection of three-dimensional points.

[0050] (12) The information processing apparatus according to (2), wherein acquiring the second specifying information comprises acquiring second specifying information that specifies the three-dimensional point, the second specifying information being input by an operation performed by a user who has visually recognized the at least one additional image.

[0051] According to the above embodiment, the information processing device can more reliably select a three-dimensional point specified by the user based on the user's operation after viewing the additional image, and can perform information processing on the three-dimensional model using the selected three-dimensional point. Therefore, the information processing device can contribute to the appropriate selection of three-dimensional points and perform information processing on the three-dimensional model.

[0052] (13) The information processing apparatus according to (12), wherein one additional image included in the at least one additional image includes a figure showing feature points included in the region.

[0053] According to the above embodiment, the information processing device includes a figure that shows feature points included in a specific region, which helps a user who has viewed an additional image to easily select feature points using the figure as a guide. Therefore, the information processing device contributes to the appropriate and easy selection of three-dimensional points.

[0054] (14) The information processing apparatus according to (1), wherein one additional image included in the at least one additional image is a three-dimensional point included in the three-dimensional model, and includes a figure indicating a position that is symmetric with respect to a predetermined reference, and a three-dimensional point identified by the second specific information.

[0055] According to the above embodiment, the information processing device includes a figure indicating a position within a specific area that is symmetrical to a three-dimensional point selected by the user. This contributes to the user easily selecting a three-dimensional point by using the figure as a guide after viewing the additional image. Therefore, the information processing device contributes to the appropriate and easy selection of a three-dimensional point.

[0056] (15) The information processing apparatus according to (1), wherein the at least one additional image includes a first additional image and a second additional image, the circuit further includes acquiring second specific information that is input to the first additional image and specifies a three-dimensional point included in the three-dimensional model, and displaying the at least one additional image includes, after acquiring the second specific information, displaying a figure showing the three-dimensional point specified by the second specific information together with the second additional image.

[0057] According to the above embodiment, the information processing device displays a figure representing the three-dimensional point selected by the user in the first additional image in the second additional image. Therefore, the user who views the second additional image can easily confirm the selected three-dimensional point. Thus, the information processing device contributes to the appropriate selection of the three-dimensional point.

[0058] (16) The information processing apparatus according to (2), wherein the circuit further updates the displayed reference image or the at least one additional image after performing the information processing to reflect the result of the information processing.

[0059] According to the above embodiment, the information processing device updates the three-dimensional model displayed on the reference image or additional image to reflect the results of the information processing. Therefore, the user can intuitively grasp the impact or effect of the information processing by viewing the updated reference image or additional image. Thus, the information processing device contributes to the appropriate selection of three-dimensional points, and furthermore, the user can intuitively grasp the results of the information processing related to the selected three-dimensional points.

[0060] (17) The information processing apparatus according to (1), wherein the circuit further acquires second identification information that identifies a three-dimensional point included in the three-dimensional model, which is input to one of the at least one additional images; displays a new additional image in which the three-dimensional point specified by the second identification information and other three-dimensional points are displayed, as viewed from a third viewpoint different from the second viewpoint corresponding to the one additional image; and acquires third identification information that identifies one of the three-dimensional points among the three-dimensional point and the other three-dimensional point, which is input to the new additional image.

[0061] According to the above embodiment, the information processing device displays a new additional image from a different viewpoint showing the three-dimensional point selected by the user and other three-dimensional points. The user can then select a three-dimensional point from among the three-dimensional point and the other three-dimensional points in the new additional image. Therefore, the information processing device contributes to the appropriate selection of the three-dimensional point.

[0062] (18) An information processing method performed by an information processing device using a circuit and a memory connected to the circuit, wherein the device acquires first specific information that identifies a part of the three-dimensional model using user input to a reference image on which a three-dimensional model viewed from a first viewpoint is displayed, determines at least one second viewpoint based on the acquired first specific information, displays at least one additional image corresponding to each of the at least one second viewpoint, and each of the at least one additional image displays the region as viewed from the second viewpoint corresponding to the additional image.

[0063] According to the above embodiment, the same effects as those of the above-described information processing device are achieved.

[0064] A program that causes an information processing device to execute the information processing methods described in (19) and (18).

[0065] According to the above embodiment, the same effects as those of the above-described information processing device are achieved.

[0066] These comprehensive or specific embodiments may be implemented as a system, device, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, integrated circuit, computer program, or recording medium.

[0067] The embodiments will be described in detail below with reference to the drawings.

[0068] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0069] (Embodiment) In this embodiment, an information processing device and an information processing method that contribute to the appropriate selection of three-dimensional points will be described.

[0070] Figure 4A is a first block diagram showing the configuration of the information processing device 10 in this embodiment. Figure 4A shows the hardware configuration of the information processing device 10.

[0071] As shown in Figure 4A, the information processing device 10 comprises at least a processor 11 and a memory 12 connected to the processor 11.

[0072] The processor 11 is an electronic circuit (also simply called a circuit) that performs information processing. The processor 11 is a circuit that can access the memory 12. The processor 11 is, for example, a CPU (Central Processing Unit), and this will be explained as an example, but it is not limited to this. The processor 11 may also include a GPU (Graphics Processing Unit). The processor 11 may also be a collection of multiple electronic circuits.

[0073] The processor 11 may be a dedicated or general-purpose electronic circuit for selecting three-dimensional points. For example, the processor 11 may play the role of multiple components (described later) of the information processing device 10, excluding the component for storing information.

[0074] The processor 11 can implement the functional unit described later by executing a program using the memory 12.

[0075] Memory 12 is a storage device that stores information. Memory 12 is connected to the processor 11. Memory 12 may be a volatile storage device (such as DRAM (Dynamic Random Access Memory)) or a non-volatile storage device (such as SSD (Solid State Drive) or HDD (Hard Disk Drive)).

[0076] The memory 12 may be an electronic circuit, or it may be connected to the processor 11. The memory 12 may also be included in the processor 11. Furthermore, the memory 12 may be a collection of multiple electronic circuits. Also, the memory 12 may be a magnetic disk or an optical disk, or it may be referred to as storage or a recording medium.

[0077] Memory 12 is a dedicated or general-purpose memory that stores information for information processing performed by the processor 11 (for example, information processing for selecting three-dimensional points). For example, memory 12 may store three-dimensional point cloud data or a stream. Memory 12 may also store a program for the processor 11 to decode the stream.

[0078] Furthermore, for example, the memory 12 may function as one of the components of the information processing device 10 that stores information. Specifically, the memory 12 may function as the storage unit 101 of the information processing device 10. More specifically, three-dimensional point cloud data may be stored in the memory 12.

[0079] Furthermore, the information processing device 10 does not necessarily have to implement all of the multiple components of the information processing device 10 described later, nor does it have to perform all of the multiple processes described above. Some of the multiple components of the information processing device 10 described later may be included in other devices, and some of the multiple processes described above may be executed by other devices.

[0080] Figure 4B is a second block diagram showing the configuration of the information processing device 10 in this embodiment. Figure 4B shows the functional configuration of the information processing device 10.

[0081] As shown in Figure 4B, the information processing device 10 includes a storage unit 101, an acquisition unit 102, a generation unit 103, a UI unit 104, a region identification unit 105, a target point identification unit 106, and a processing unit 107 as functional units. These functional units can be realized by the processor 11 executing a program using the memory 12.

[0082] The memory unit 101 is a memory unit that stores information. The memory unit 101 stores at least three-dimensional data that represents a three-dimensional model. Specifically, the three-dimensional data includes at least positional information indicating the position of each of a plurality of three-dimensional points in three-dimensional space. The three-dimensional data may also include connection information indicating the connections between a plurality of three-dimensional points, or attribute information indicating the attributes of the three-dimensional points. The three-dimensional data stored in the memory unit 101 may be three-dimensional data acquired by the acquisition unit 102 from an external information processing device or the like and stored in the memory unit 101.

[0083] The acquisition unit 102 acquires three-dimensional data from the storage unit 101. The acquisition unit 102 can acquire the data by reading the three-dimensional data stored in the storage unit 101. The acquisition unit 102 may also have a communication interface, in which case it can acquire three-dimensional data from an external information processing device, etc., using the communication interface and store it in the storage unit 101, and then acquire the data by reading the stored three-dimensional data from the storage unit 101.

[0084] The generation unit 103 determines the viewpoint and line of sight direction for the three-dimensional model represented by the three-dimensional data acquired by the acquisition unit 102, and generates a two-dimensional image of the three-dimensional model as viewed from the determined viewpoint and in the determined line of sight direction.

[0085] Specifically, the generation unit 103 determines a predetermined viewpoint (also called a reference viewpoint) and a predetermined line of sight direction (also called a reference line of sight direction) for the three-dimensional model represented by the three-dimensional data acquired by the acquisition unit 102. The generation unit 103 also generates a two-dimensional image (also called a reference image) of the three-dimensional model as viewed from the reference viewpoint in the reference line of sight direction.

[0086] Furthermore, when the generation unit 103 obtains first identification information from the region identification unit 105, it determines at least one viewpoint (also called a second viewpoint or additional viewpoint) based on the obtained first identification information. The additional viewpoint is, for example, a viewpoint that views the specific region identified by the region identification unit 105 in the three-dimensional model obtained by the acquisition unit 102. Note that the additional viewpoint is not limited to the above.

[0087] If the generation unit 103 determines at least one additional viewpoint, it determines the line of sight direction corresponding to each of the determined additional viewpoints. The line of sight direction is the direction in which the three-dimensional model is viewed from the additional viewpoint corresponding to that line of sight direction.

[0088] The generation unit 103 then generates a two-dimensional image (also called an additional image) of a specific region viewed from each of the additional viewpoints in the direction of the additional line of sight. In this way, the generation unit 103 generates one or more additional images.

[0089] For example, at least one second viewpoint may include a second viewpoint in which a specific region of the three-dimensional model is displayed without being obscured in an additional image showing the three-dimensional model as seen from that second viewpoint.

[0090] Furthermore, for example, at least one second viewpoint may be located on the normal of a surface defined by a specific region in the three-dimensional model. The normal of a surface defined by a specific region refers, for example, to the normal of a surface formed by the specific region, or to the normal of a surface passing through the vicinity of multiple three-dimensional points included in the specific region. The same applies hereafter.

[0091] Furthermore, for example, at least one second viewpoint may include two second viewpoints. The direction from each of the two second viewpoints toward a specific region may be perpendicular to the direction from the first viewpoint toward the specific region, and the directions from the two second viewpoints toward the specific region may be perpendicular to each other.

[0092] Furthermore, for example, at least one second viewpoint may include a second viewpoint that views the three-dimensional model from above, a second viewpoint that views the three-dimensional model from the right, and a second viewpoint that views the three-dimensional model from the left. This assumes that a predetermined upward direction, a rightward direction perpendicular to the upward direction, and a leftward direction opposite to the rightward direction are defined for the three-dimensional model.

[0093] Furthermore, for example, at least one second viewpoint may include a second viewpoint specified by the user who viewed the reference image. In this case, the receiving unit 104B receives the position of the second viewpoint specified by the user who viewed the reference image. The receiving unit 104B, having received the position of the second viewpoint, provides that position to the generating unit 103, and the generating unit 103 can use the provided position of the second viewpoint.

[0094] Furthermore, when generating an additional image, the generation unit 103 may determine the field of view of the additional image according to the degree of density of the multiple three-dimensional points included in a specific region of the three-dimensional model, and generate an additional image having the determined field of view. When determining the field of view, the generation unit 103 may determine the field of view so that the field of view becomes larger the more densely packed the multiple three-dimensional points included in the specific region of the three-dimensional model are.

[0095] The UI unit 104 provides a user interface to the user. Specifically, the UI unit 104 includes a display unit 104A and a reception unit 104B.

[0096] The display unit 104A displays an image. The display unit 104A is implemented by an image display device (for example, a liquid crystal display or an OLED (Organic Light Emitting Diode) display).

[0097] The display unit 104A displays the reference image generated by the generation unit 103. The display unit 104A also displays at least one additional image generated by the generation unit 103. The display unit 104A may display the reference image and at least one additional image on a single screen, or it may display the reference image first, then stop displaying the reference image, and then display the additional image.

[0098] Furthermore, when displaying a reference image or an additional image, the display unit 104A may display a cursor that serves as a guide for the position where user operation is accepted. The shape of the cursor may be a figure with a width, or it may be a dot. The shape of the cursor may be changed each time an operation is performed depending on the target of the operation.

[0099] The reception unit 104B accepts user operations. The reception unit 104B is implemented by a device such as a mouse, keyboard, touchpad, or touch panel.

[0100] The reception unit 104B receives user input on the image displayed by the display unit 104A. When the reception unit 104B receives user input, it provides information indicating that input to the area identification unit 105 or the target point identification unit 106.

[0101] Specifically, the reception unit 104B receives an operation from the user to specify a portion of a three-dimensional model in relation to the reference image displayed by the display unit 104A. When the above operation is received, the UI unit 104 generates information indicating the above operation (also called first operation information) and provides it to the region identification unit 105.

[0102] Furthermore, the reception unit 104B receives an operation from the user to specify a three-dimensional point included in the three-dimensional model for the additional image displayed by the display unit 104A. When the above operation is received, the reception unit 104B acquires information indicating the above operation (corresponding to second operation information) and provides it to the target point identification unit 106.

[0103] The UI unit 104 can be implemented using a touch panel-equipped liquid crystal display. In that case, the display unit 104A may be the liquid crystal display portion of the touch panel-equipped liquid crystal display of the UI unit 104, and the reception unit 104B may be the touch panel portion of the touch panel-equipped liquid crystal display of the UI unit 104.

[0104] When the UI unit 104 provides first operation information, the region identification unit 105 identifies a portion of the three-dimensional model (also called a specific region) that is the target of the operation indicated in the first operation information. The region identification unit 105 generates specific information indicating the specific region and provides it to the generation unit 103.

[0105] Specifically, the region identification unit 105 uses first operation information to identify a specific region within the three-dimensional model that is the target of user operation on the reference image. If there are multiple regions that can be identified as specific regions using the first operation information, the region identification unit 105 can select a specific region from among the multiple specific regions that is displayed on the reference image (in other words, a specific region visible to the user). The above selection method will be explained with reference to Figure 5.

[0106] Figure 5 is an explanatory diagram showing the selection of a region by the information processing device 10 (specifically, the region identification unit 105) in this embodiment.

[0107] Figure 5 schematically shows the three-dimensional model M and the user's line of sight.

[0108] In Figure 5, there can be multiple candidates for the region included in the three-dimensional model that corresponds to the region selected on the reference image. Specifically, the candidate regions included in the three-dimensional model that correspond to the region selected on the reference image are region A1 near three-dimensional point P1 and region A2 near three-dimensional point P2. In this case, the region identification unit 105 can select region A1, which is the closest to the user, by the method described below.

[0109] In other words, the region identification unit 105 can identify a specific region within the three-dimensional model that corresponds to the region selected on the reference image, using the line-of-sight vector indicating the user's line of sight direction and the normal vector of the three-dimensional point. Note that when referring to the normal vector of a three-dimensional point, it means the normal vector indicating the normal of the surface defined by that three-dimensional point (or that three-dimensional point and neighboring three-dimensional points). The same applies hereafter.

[0110] For example, the region identification unit 105 can calculate the dot product of the line-of-sight vector indicating the direction of the line of sight and the normal vector of a three-dimensional point included in the region specified on the reference image, and select the three-dimensional point for which the dot product is negative.

[0111] In the example shown in Figure 5, the angle θ1 between the line of sight vector V and the normal vector N1 is greater than 90° and less than 180°. Also, the angle θ2 between the line of sight vector V and the normal vector N2 is greater than 0° and less than 90°. The region identification unit 105 can select a three-dimensional point P1 (in other words, region A1) based on its determination that the dot product of the line of sight vector V and the normal vector N1 is negative.

[0112] Furthermore, the region identification unit 105 can also select a three-dimensional point based on the angular range of the angle between the line-of-sight vector and the normal vector. In other words, the region identification unit 105 can select a three-dimensional point where the angle between the line-of-sight vector indicating the direction of the line of sight and the normal vector of a three-dimensional point included in the region specified on the reference image is greater than 90° and less than 180°. In the example in Figure 5, the region identification unit 105 can select the three-dimensional point P1 (in other words, region A1) based on its determination that the angle θ1 between the line-of-sight vector V and the normal vector N1 is greater than 90°.

[0113] When the UI unit 104 provides second operation information, the target point identification unit 106 identifies a three-dimensional point included in the three-dimensional model that is the target of the operation indicated in the second operation information. The target point identification unit 106 obtains the identification information (also called second identification information) that identifies the three-dimensional point by generating it and provides it to the processing unit 107.

[0114] Specifically, when acquiring the second identification information, the target point identification unit 106 may acquire second identification information that specifies a three-dimensional point, which is input by an operation performed by a user who has viewed at least one additional image.

[0115] The processing unit 107 performs information processing related to the three-dimensional model. Specifically, when the processing unit 107 obtains second identification information provided by the target point identification unit 106, it uses the obtained second identification information to perform information processing related to the three-dimensional model.

[0116] The information processing may include processes for updating the three-dimensional model, specifically processes for rotating the three-dimensional model, moving the three-dimensional model, or taking measurements of the three-dimensional model. The information processing may also include processes for changing the position of the viewpoint from which the three-dimensional model is viewed. Furthermore, the information processing may include processes for updating the image displayed by the display unit 104A (specifically, a reference image or an additional image). For example, the process for updating the image may include superimposing a figure representing a three-dimensional point selected by the user onto the additional image displayed by the display unit 104A.

[0117] The processing unit 107 can update the three-dimensional data through the above information processing, and in that case, the updated data can be stored in the storage unit 101. This allows the processing unit 107 to ensure that when the acquisition unit 102 subsequently acquires the three-dimensional data, the updated three-dimensional data is acquired.

[0118] Furthermore, if the processing unit 107 changes the position of the viewpoint from which the three-dimensional model is viewed (specifically, the reference viewpoint or additional viewpoint) through the above information processing, it can provide the generation unit 103 with information indicating the changed viewpoint. This allows the processing unit 107 to change the position of the viewpoint from which the three-dimensional model is viewed in the image generated by the generation unit 103.

[0119] Furthermore, if the processing unit 107 updates the image displayed by the UI unit 104 as a result of the above information processing, it can provide the generation unit 103 with information indicating the content of the update. This allows the processing unit 107 to update the image generated by the generation unit 103 and to update the image displayed by the display unit 104A.

[0120] In this way, after the processing unit 107 has performed information processing, the image displayed by the UI unit 104 (specifically, the reference image or additional image) can be updated to reflect the results of the information processing.

[0121] Furthermore, of the functional units of the information processing device 10, at least the UI unit 104 may be located in another device. In that case, the information processing device 10 and the other device are connected by a communication line, and information is exchanged between the generation unit 103, the area identification unit 105, the target point identification unit 106 and the UI unit 104 via the communication line. In this case, at least the UI unit 104 may be located in a user terminal (for example, a personal computer or smartphone). The information processing device 10 may also be referred to as an information processing system.

[0122] The information processing performed by the information processing device 10 configured as described above will now be explained.

[0123] Figure 6 is a flowchart showing a first example of the processing performed by the information processing device 10 in this embodiment.

[0124] In step S101, the information processing device 10 acquires three-dimensional data. Specifically, the acquisition unit 102 acquires three-dimensional data by reading it from the storage unit 101.

[0125] In step S102, the information processing device 10 displays a reference image. Specifically, the generation unit 103 generates a reference image and provides it to the UI unit 104. The UI unit 104 displays the reference image provided by the generation unit 103 on the display unit 104A. An example of a reference image will be explained with reference to Figure 9.

[0126] Figure 9 is an explanatory diagram showing a first example of a reference image displayed by the information processing device 10 in this embodiment. In Figure 9, the coordinates or directions of the three-dimensional model may be represented using the x, y, and z axes shown in the figure. The same applies to the subsequent reference images and additional images.

[0127] The reference image shown in Figure 9 shows a three-dimensional model of a stylized quadrupedal animal (e.g., a cow). The reference image shown in Figure 9 is an example of a two-dimensional image of the quadrupedal animal viewed from the right front of the animal. In other words, the reference image shown in Figure 9 is an example of a two-dimensional image of the three-dimensional model viewed from a viewpoint where the x-coordinate is larger, the y-coordinate is smaller, and the z-coordinate is approximately the same as the set of three-dimensional points representing the three-dimensional model. The generation unit 103 can generate a reference image by processing information about existing three-dimensional data.

[0128] In step S103, the information processing device 10 acquires first specific information. Specifically, the UI unit 104 receives user operations on the reference image displayed in step S102 via the reception unit 104B and provides first operation information indicating the operation to the area identification unit 105. The area identification unit 105 acquires first specific information by generating information that identifies the area that is the target of the operation indicated in the provided first operation information. The first specific information may be information that identifies the area using a mathematical formula (e.g., an inequality) using coordinates indicating a position in three-dimensional space, or it may be information that identifies the area using a surface (e.g., a boundary surface) defined in three-dimensional space. The definition of the surface may be by any method, for example, it may be defined by a mathematical formula or by geometry.

[0129] The UI unit 104 may also display a frame 31 (see Figure 9) indicating the area selected by the user, along with the reference image. The frame 31 illustrated in Figure 9 is a rectangular frame with width in the vertical and horizontal directions for selecting a portion of the reference image. For example, the position and shape of the frame 31 are determined by the user specifying two points (e.g., the upper left corner and the lower right corner of the rectangular frame). Once the frame 31 is determined, the area located inside the frame 31 is selected. The shape of the frame 31 is not limited to the above, and may be a polygon, a circle or an ellipse, or a closed line of any shape.

[0130] In step S104, the information processing device 10 displays an additional image using the first specific information. Specifically, the information processing device 10 acquires the first specific information obtained by the region identification unit 105 in step S103, generates and displays at least one additional image. The detailed processing included in step S104 will be explained in detail later.

[0131] Here, at least one additional image generated by the generation unit 103 will be described with reference to Figures 10A, 10B, and 10C.

[0132] Figure 10A is an explanatory diagram showing a first example of an additional image displayed by the information processing device 10 in this embodiment. The additional image shown in Figure 10A is a two-dimensional image in which the viewpoint is a position with a z-coordinate smaller than the area selected by the frame 31 in the three-dimensional model (see Figure 9), and the z-axis positive direction is the line of sight.

[0133] Figure 10B is an explanatory diagram showing a second example of an additional image displayed by the information processing device 10 in this embodiment. The additional image shown in Figure 10B is a two-dimensional image in which the viewpoint is a position with a y-coordinate smaller than the area selected by the frame 31 in the three-dimensional model (see Figure 9), and the positive y-axis direction is the line of sight.

[0134] Figure 10C is an explanatory diagram showing a third example of an additional image displayed by the information processing device 10 in this embodiment. The additional image shown in Figure 10C is a two-dimensional image in which the viewpoint is a position where the y-coordinate is greater than the area selected by the frame 31 in the three-dimensional model (see Figure 9), and the negative y-axis direction is the line of sight.

[0135] In step S105, the information processing device 10 acquires second identification information. Specifically, the UI unit 104 receives user operations on the additional image displayed in step S104 via the reception unit 104B and provides second operation information indicating the operation to the target point identification unit 106. The target point identification unit 106 acquires second identification information by generating information that identifies the three-dimensional point that is the target of the operation indicated in the provided second operation information. The second identification information is information that can uniquely identify a three-dimensional point included in the three-dimensional model.

[0136] The UI unit 104 may also display a point indicating a three-dimensional point selected by the user (specifically, point 32 (see Figure 10A), point 33 (see Figure 10B), or point 34 (see Figure 10C)) along with an additional image.

[0137] In step S106, the target point identification unit 106 uses the second identification information acquired in step S105 to identify a three-dimensional point included in the three-dimensional model. The identified three-dimensional point is a three-dimensional point that the user wants to specify.

[0138] In step S107, the information processing device 10 (processing unit 107) performs information processing related to the three-dimensional model using the three-dimensional points identified in step S106. This information processing may include updating the three-dimensional data related to the three-dimensional model, changing the position of the viewpoint from which the three-dimensional data is viewed, and updating the image displayed by the display unit 104A.

[0139] When the processing unit 107 updates the three-dimensional data through the above information processing, it can store the updated data in the storage unit 101. When the processing unit 107 changes the viewpoint from which the three-dimensional model is viewed through the above information processing, it can provide the generation unit 103 with information indicating the changed viewpoint. When the processing unit 107 updates the image displayed by the display unit 104A through the above information processing, it can provide the generation unit 103 with information indicating the content of the update.

[0140] Figure 7 is a flowchart showing a second example of the processing performed by the information processing device 10 in this embodiment. The flowchart shown in Figure 7 shows a detailed example of the processing included in step S104 of Figure 6.

[0141] In step S201, the generation unit 103 identifies three-dimensional points among the three-dimensional points included in the three-dimensional model that are located within the range specified by the first identification information. The range specified by the first identification information is the range of coordinates of the three-dimensional points specified by the first identification information.

[0142] In step S202, the generation unit 103 determines the viewpoint of the additional image. The generation unit 103 also determines the line of sight direction (corresponding to the additional line of sight direction) from which the three-dimensional model is viewed from the determined viewpoint.

[0143] In step S203, the generation unit 103 generates a two-dimensional image as an additional image, which is a view of the three-dimensional model from the viewpoint and in the direction of the line of sight determined in step S202.

[0144] In step S204, the generation unit 103 provides the generated additional image to the UI unit 104. The UI unit 104 displays the provided additional image using the display unit 104A.

[0145] Figure 8 is a flowchart showing a third example of the processing performed by the information processing device 10 in this embodiment. The flowchart shown in Figure 8 shows a detailed example of the processing included in step S201 in Figure 7.

[0146] In step S301, the generation unit 103 obtains range conditions for a three-dimensional point specified by the first specific information. The range conditions for a three-dimensional point are conditions for the range of coordinates of the three-dimensional point specified by the first specific information. The coordinate range specified by the first specific information is obtained.

[0147] In step S302, the generation unit 103 identifies one or more three-dimensional points among the three-dimensional points included in the three-dimensional model that satisfy the range condition. The range condition is the range condition obtained in step S301.

[0148] Through the series of processes shown in Figure 8, the generation unit 103 can identify a three-dimensional point located within the range specified by the first specific information.

[0149] Figure 11 is a flowchart showing a fourth example of the processing performed by the information processing device 10 in this embodiment. The flowchart shown in Figure 11 shows a detailed processing included in step S201 of Figure 7, and is a different example from the series of processing shown in Figure 8.

[0150] In step S321, the generation unit 103 acquires range conditions for the three-dimensional point identified by the first identification information. Step S321 is the same as step S301 shown in Figure 8.

[0151] In step S322, the generation unit 103 determines whether there is one object that satisfies the range condition obtained in step S321. Here, an object refers to a part of the three-dimensional model that satisfies the range condition.

[0152] If it is determined that there is only one object (Yes in step S322), proceed to step S325; otherwise (No in step S322), proceed to step S323.

[0153] In step S323, the information processing device 10 generates and displays an additional image in which all objects that satisfy the range condition are visible. At this time, the generation unit 103 determines the viewpoint and line of sight direction in which all objects that satisfy the range condition are visible, and can generate an additional image as an image of the three-dimensional model viewed from the determined viewpoint in the line of sight direction. The UI unit 104 displays the additional image generated by the generation unit 103 on the display unit 104A.

[0154] In step S324, the UI unit 104 acquires additional range conditions. Specifically, the UI unit 104 receives user operations on the additional image displayed in step S323 via the reception unit 104B and provides operation information indicating those operations to the area identification unit 105. The area identification unit 105 acquires additional range conditions based on the operations indicated in the provided operation information.

[0155] In step S325, the generation unit 103 identifies one or more three-dimensional points among the three-dimensional points included in the three-dimensional model that satisfy the range condition. The range condition is the range condition obtained in step S321 if it is determined in step S322 that there is one object (Yes in step S322). If it is determined in step S322 that there is more than one object (No in step S322), the range condition is the range condition obtained in step S321 plus the range condition obtained in step S324.

[0156] Through the series of processes shown in Figure 11, the generation unit 103 can identify a three-dimensional point located within the range specified by the first specific information.

[0157] The series of processes shown in Figure 11 will be explained with reference to Figures 12, 13, and 14.

[0158] Figure 12 is an explanatory diagram showing a second example of a reference image displayed by the information processing device 10 in this embodiment.

[0159] The reference image shown in Figure 12 shows a three-dimensional model of a stylized quadrupedal animal (e.g., a cow). The reference image shown in Figure 12 is an example of a two-dimensional image of the quadrupedal animal viewed from the right side of the animal. In other words, the reference image shown in Figure 12 is an example of a two-dimensional image of the three-dimensional model viewed from a viewpoint located at a position where the y-coordinate is larger than that of the set of three-dimensional points representing the three-dimensional model. The UI unit 104 also displays a frame 41 indicating the region selected by the user, along with the reference image.

[0160] In this case, the range condition is "the portion located within frame 41 in the reference image." The objects that satisfy the range condition are the left foreleg and the right foreleg of a quadrupedal animal. The generation unit 103 determines the viewpoint and line of sight direction from which the left foreleg and the right foreleg are visible as all objects that satisfy the range condition, and generates an additional image of the three-dimensional model viewed from the determined viewpoint in the line of sight direction (step S323). An example of an additional image generated by the generation unit 103 is shown in Figure 13.

[0161] Figure 13 is an explanatory diagram showing a fourth example of an additional image displayed by the information processing device 10 in this embodiment. The additional image shown in Figure 13 is an image generated by the generation unit 103 in step S323 and is an example of an image displayed by the UI unit 104.

[0162] The additional image shown in Figure 13 shows the left and right foreleg portions as all objects that satisfy the range conditions.

[0163] Furthermore, the UI unit 104 displays a linear cursor 42, along with an additional image, indicating the position selected by the user. The cursor 42 can be moved left and right on the screen (in other words, in the positive and negative y-axis directions in the three-dimensional model) by user operation.

[0164] If the user intends to specify, for example, the right foreleg, they can perform the selection operation while the cursor 42 is overlapping the right foreleg.

[0165] The reception unit 104B receives the above operation from the user and provides the area identification unit 105 with information indicating that the right foreleg has been selected as operation information indicating the above operation. The area identification unit 105 can acquire the condition "part of the right foreleg" as an additional range condition.

[0166] Figure 14 is an explanatory diagram showing a fifth example of an additional image displayed by the information processing device 10 in this embodiment. The additional image shown in Figure 14 is an image generated by the generation unit 103 after the additional range conditions are acquired in step S324, and is an example of an image displayed by the UI unit 104 (steps S203, S204).

[0167] In the additional image shown in Figure 14, the right foreleg is shown, which is one of the two forelegs that may fall within the area selected by the user (i.e., frame 41 in Figure 12). This is because the additional range conditions obtained when the user specified the right foreleg in the additional image shown in Figure 13 are reflected.

[0168] The following section will explain the process when a three-dimensional point is selected in an additional image.

[0169] For example, when the information processing device 10 generates and displays multiple additional images, if a three-dimensional point is selected in one of the additional images, the selected three-dimensional point may be displayed in the other additional images.

[0170] Specifically, when the reception unit 104B receives second specific information that specifies a three-dimensional point included in the three-dimensional model, which has been input for the first additional image, it may display a figure indicating the three-dimensional point specified by the second specific information along with the second additional image.

[0171] Figure 15 is an explanatory diagram showing a sixth example of an additional image displayed by the information processing device 10 in this embodiment. Figure 16 is an explanatory diagram showing a seventh example of an additional image displayed by the information processing device 10 in this embodiment.

[0172] Figures 15(a) and 15(b) are examples of additional images similar to those in Figures 10A and 10B, respectively, and are images before user selection. Since user selection has not yet been made, points 32 shown in Figure 10A and point 33 shown in Figure 10B are not shown in Figures 15(a) and 15(b).

[0173] In a state where the display unit 104A is displaying the additional image shown in Figure 15(a) and the additional image shown in Figure 15(b) (step S104), suppose the user selects point 51 in the additional image shown in Figure 15(a).

[0174] In this case, the UI unit 104 receives user operations on the displayed additional image via the reception unit 104B and provides second operation information indicating the operation to the target point identification unit 106. The target point identification unit 106 acquires second identification information that identifies the three-dimensional point that is the target of the operation indicated in the provided second operation information. The processing unit 107 can update the additional image displayed on the UI unit 104 by providing information indicating the content of the update of the additional image to the generation unit 103. An example of the updated additional image will be explained with reference to Figure 16.

[0175] Figures 16(a) and 16(b) are examples of additional images showing the same three-dimensional model as Figures 10A and 10B, respectively, and are images after user selection.

[0176] Figure 16(a) shows point 52, which represents a three-dimensional point selected by the user, superimposed on a figure representing a three-dimensional model. Figure 16(b) shows point 53, which represents a three-dimensional point selected by the user, superimposed on a figure representing a three-dimensional model.

[0177] Furthermore, if an operation such as rotation or movement is performed on the three-dimensional model displayed in one of the multiple additional images, the generation unit 103 may rotate or move the three-dimensional model displayed in the other additional images in the multiple additional images in conjunction with that rotation or movement.

[0178] The following section will explain the process when a three-dimensional point is selected in an additional image.

[0179] For example, when the information processing device 10 generates and displays multiple additional images, if a three-dimensional point is selected in one of the additional images, it may display additional images viewed from other viewpoints.

[0180] Specifically, when the reception unit 104B receives second identification information that specifies a three-dimensional point included in the three-dimensional model, which has been input for one additional image, it may display a new additional image viewed from a viewpoint different from the second viewpoint corresponding to the first additional image (also called a third viewpoint). The new additional image displays the three-dimensional point specified by the second identification information and other three-dimensional points. The reception unit 104B may then receive third identification information that specifies one of the three-dimensional points among the three-dimensional point and the other three-dimensional points, which has been input for the new additional image.

[0181] Figure 17 is an explanatory diagram showing an eighth example of an additional image displayed by the information processing device 10 in this embodiment.

[0182] Figure 17(a) is an example of an additional image. The additional image shown in Figure 17(a) includes point 61, which is a three-dimensional point. Point 61 is a three-dimensional point included in the three-dimensional model.

[0183] In the additional image shown in Figure 17(a), if point 61 is selected by user operation, point 61 will be selected in principle.

[0184] In this case, when point 61 is viewed in the line of sight in the three-dimensional model, other three-dimensional points may be located behind point 61. In this case, in the line of sight of the additional image shown in Figure 17(a), the other three-dimensional points located behind point 61 are hidden behind point 61 and are not displayed in the additional image.

[0185] It is difficult for the user to select other three-dimensional points located behind point 61 simply by using the additional image shown in Figure 17(a). In other words, it is difficult for the information processing device 10 to receive a selection from the user for other three-dimensional points located behind point 61 simply by using the additional image shown in Figure 17(a).

[0186] In this case, the generation unit 103 can generate the additional image shown in Figure 17(b).

[0187] Figure 17(b) is an example of an additional image. The additional image shown in Figure 17(b) is an example of an additional image generated by the generation unit 103 when, in the three-dimensional model, point 61 is viewed in the line of sight and other three-dimensional points (points 62 and 63) are located behind point 61.

[0188] In a three-dimensional model, when point 61 is viewed in the line of sight, points 62 and 63 are located behind point 61. If point 61, as shown in Figure 17(a), is selected by the user, the target point identification unit 106 identifies the selected point 61, and the processing unit 107 performs a process to change the position of the viewpoint from which the three-dimensional model is viewed. The processing unit 107 determines a viewpoint and line of sight from which all of point 61 and the other three-dimensional points, points 62 and 63, located behind it, are visible, and can provide information indicating that viewpoint and line of sight to the generation unit 103. Using the viewpoint and line of sight provided by the processing unit 107, the generation unit 103 generates an additional image from that viewpoint in that line of sight, viewing points 61, 62, and 63, thereby generating the additional image shown in Figure 17(b).

[0189] The processing unit 107 may also instruct the generation unit 103 to determine a viewpoint and line of sight that allows all points 61, 62, and 63 to be seen. In this case, the processing unit 107 provides the generation unit 103 with control information to determine a viewpoint and line of sight that allows all points 61, 62, and 63 to be seen, and the generation unit 103 can determine the viewpoint and line of sight according to that control information.

[0190] Figure 17(b) shows points 61, 62, and 63. Point 61 shown in Figure 17(b) is the same three-dimensional point as point 61 shown in Figure 17(a).

[0191] Furthermore, arrow 65 is shown in Figure 17(b). Arrow 65 is an arrow extending from point 61 in the direction of line of sight of the additional image in Figure 17(a). Points 62 and 63 are located on arrow 65.

[0192] The user can select points 61, 62, or 63 in the additional image shown in Figure 17(b). In other words, the information processing device 10 can receive a selection from the user for other three-dimensional points located behind point 61 using the additional image shown in Figure 17(b).

[0193] In this way, the information processing device 10 enables the user to select a specific three-dimensional point from among multiple three-dimensional points located at the same position in an additional image.

[0194] In the following sections, we will explain the process of displaying additional images, including candidate points that the user may select.

[0195] For example, the information processing device 10 may display the additional image including a figure that shows the feature points included in the specific region. Alternatively, the information processing device 10 may display the additional image including a figure that shows a three-dimensional point already selected by the user and a position that is symmetrical with respect to a predetermined reference.

[0196] Figure 18A is an explanatory diagram showing a ninth example of an additional image displayed by the information processing device 10 in this embodiment.

[0197] When generating an additional image, the generation unit 103 may superimpose a figure indicating candidate points that may be candidates for selection by the user onto the three-dimensional model. Candidate points are, for example, feature points included in the region of the three-dimensional model. Feature points are three-dimensional points included in the three-dimensional model, and may be three-dimensional points located in convex or concave parts of the three-dimensional model, or three-dimensional points with a different color from their surroundings. Also, if the three-dimensional model is a three-dimensional model that simulates an animal, the feature points may be three-dimensional points located in positions corresponding to parts of that animal (eyes, nose, mouth, jaw, ears, horns, etc.).

[0198] In Figure 18A, candidate points 71, 72, 73, 74, and 75 are superimposed on the three-dimensional model.

[0199] Candidate point 71 is a three-dimensional point located at the tip of the left ear of a quadrupedal animal. Candidate point 72 is a three-dimensional point located at the left corner of the jaw of a quadrupedal animal. Candidate point 73 is a three-dimensional point located at the tip of the jaw of a quadrupedal animal. Candidate point 74 is a three-dimensional point located at the right corner of the jaw of a quadrupedal animal. Candidate point 75 is a three-dimensional point located at the tip of the right ear of a quadrupedal animal.

[0200] By visualizing candidate points 71-75, users can understand the location of characteristic points in the three-dimensional model and easily select those characteristic points.

[0201] Figure 18B is an explanatory diagram showing a tenth example of an additional image displayed by the information processing device 10 in this embodiment.

[0202] In Figure 18B, candidate points 76, 77, 78, and 79 are superimposed on the three-dimensional model.

[0203] Candidate point 76 is a three-dimensional point located at the tip (in other words, the lower end) of the convex part that protrudes downward from the jaw of a quadrupedal animal. Candidate point 77 is a three-dimensional point located at the tip (in other words, the front end) of the convex part that protrudes forward from the jaw of a quadrupedal animal. Candidate point 78 is a three-dimensional point located at the tip of the right horn of a quadrupedal animal. Candidate point 79 is a three-dimensional point located at the tip of the right ear of a quadrupedal animal.

[0204] By visualizing candidate points 76-79, users can understand the location of characteristic points in the three-dimensional model and easily select those characteristic points.

[0205] Figure 19 is an explanatory diagram showing an eleventh example of an additional image displayed by the information processing device 10 in this embodiment.

[0206] Figure 19 shows the user-selected point 81 and candidate point 82 superimposed on the three-dimensional model.

[0207] Point 81 is a three-dimensional point located at the tip of the left ear of a quadrupedal animal. Point 81 is a point selected by the user in other additional images.

[0208] Candidate point 82 is a three-dimensional point located at the tip of the right ear of a quadrupedal animal. Candidate point 82 is located at the tip of the right ear (i.e., symmetrical to point 81 selected by the user (i.e., the tip of the left ear) with respect to the centerline of the quadrupedal animal's body.

[0209] By visually inspecting the candidate points 82, the user can understand the location of points symmetrical to the point 81 they have already selected, and can easily select those symmetrical points.

[0210] Figure 20 is a flowchart showing a fifth example of the processing performed by the information processing device 10 in this embodiment. The flowchart shown in Figure 20 shows the detailed processing included in step S105 (see Figure 6).

[0211] In step S401, the information processing device 10 displays the candidate points superimposed on the additional image. Specifically, the generation unit 103 generates an image with the candidate points superimposed on the additional image and provides it to the UI unit 104. The UI unit 104 displays the provided additional image on the display unit 104A.

[0212] In step S402, the UI unit 104 determines whether the user has performed an operation to select a candidate point or an operation to select another three-dimensional point. If the user has performed an operation to select a candidate point ("candidate point selection operation" in step S402), the unit proceeds to step S403. If the user has performed an operation to select another three-dimensional point ("another three-dimensional point selection operation" in step S402), the unit proceeds to step S411.

[0213] In step S403, the information processing device 10 acquires second identification information indicating the candidate point selected in step S402. Specifically, the UI unit 104 receives user operations on the additional image via the reception unit 104B and provides second operation information indicating that operation to the target point identification unit 106. The target point identification unit 106 acquires second identification information that identifies the three-dimensional point that is the target of the operation indicated in the provided second operation information.

[0214] In step S411, the information processing device 10 acquires second identification information indicating the three-dimensional point selected in step S402. Specifically, the UI unit 104 receives user operations on the additional image via the reception unit 104B and provides second operation information indicating that operation to the target point identification unit 106. The target point identification unit 106 acquires second identification information that identifies the three-dimensional point that is the target of the operation indicated in the provided second operation information.

[0215] In the following sections, we will explain the selection of three-dimensional points for a three-dimensional model generated by a learning process from multiple two-dimensional images.

[0216] The information processing device 10 can generate an additional image using the same second viewpoint as one of the multiple viewpoints when the three-dimensional model is a three-dimensional model generated by a learning process that takes multiple images as input, in which the object is displayed from multiple viewpoints.

[0217] Figure 21 is an explanatory diagram illustrating an example of a technique for generating a three-dimensional model.

[0218] Figure 21 shows several images 91, 92, and 93 as examples of two-dimensional images of an object taken from multiple different viewpoints.

[0219] By using a technique to generate a three-dimensional model from multiple two-dimensional images, a three-dimensional model 95 corresponding to an object is generated from multiple images 91, 92, and 93 taken from multiple viewpoints of the object. An example of a technique for generating a three-dimensional model from multiple two-dimensional images is 3DGS (3D Gaussian Platting).

[0220] When a user selects a three-dimensional point included in a three-dimensional model generated by the three-dimensional model generation technology described above, it is assumed that the user will select a point in the two-dimensional image used to generate the three-dimensional model, thereby selecting the corresponding three-dimensional point included in the three-dimensional model. In this case, the user may only be able to select from points that appear in multiple two-dimensional images; in other words, the points that the user can select may be limited to points that appear in multiple two-dimensional images.

[0221] The information processing device 10 can generate and display additional images when selecting a three-dimensional point included in the three-dimensional model generated by the three-dimensional model generation technology described above, thereby enabling the user to select any three-dimensional point included in the three-dimensional model.

[0222] Figure 22 is an explanatory diagram showing a twelfth example of an additional image displayed by the information processing device 10 in this embodiment. The additional image shown in Figure 22 is an example of an additional image that the information processing device 10 generates and displays when, for example, the abdomen of a quadrupedal animal is identified as a part of the three-dimensional model 95 shown in Figure 21.

[0223] The additional image shown in Figure 22 is a two-dimensional image with a viewpoint at a position where the z-coordinate is smaller than that of a portion of the three-dimensional model 95 shown in Figure 21 (i.e., the abdomen), and the line of sight is in the positive z-axis direction.

[0224] Furthermore, if the user selects the center of the abdomen for the additional image shown in Figure 22, and the selected location differs from the location of the three-dimensional point, the information processing device 10 can select point 97, which is the three-dimensional point closest to the selected location. If the three-dimensional model is represented by 3DGS, the selected three-dimensional point is the center point of the Gaussian.

[0225] Furthermore, if the user selects the center of the abdomen for the additional image shown in Figure 22, and the selected position is different from the position of the three-dimensional point, the information processing device 10 may perform interpolation in the three-dimensional model to generate a new three-dimensional point at the selected position (or a position relatively close to it), and then select the generated three-dimensional point.

[0226] Furthermore, if a three-dimensional point selected in one additional image is not displayed in another additional image because, from the viewpoint, it is located in a position hidden by an opaque surface included in the three-dimensional model (in other words, it is located on the opposite side of the opaque surface from the viewpoint), the selected three-dimensional point may be displayed in the additional image by making the opaque surface semi-transparent.

[0227] Figure 23 is an explanatory diagram showing a thirteenth example of an additional image displayed by the information processing device 10 in this embodiment. The additional image shown in Figure 23 is an example of an additional image that the information processing device 10 generates and displays when, for example, the abdomen of a quadrupedal animal is identified as a part of the three-dimensional model 95 shown in Figure 21.

[0228] The additional image shown in Figure 23 is a two-dimensional image with a viewpoint at a position where the z-coordinate is greater than that of a portion of the three-dimensional model 95 shown in Figure 21 (i.e., the abdomen), and the line of sight is in the negative z-axis direction.

[0229] In the additional image shown in Figure 23, point 97 shown in Figure 22 is shown as point 98. Point 98 is displayed semi-transparently. This is because point 98 is located on the abdomen of a quadrupedal animal and is therefore hidden from the viewpoint of the additional image shown in Figure 23.

[0230] (Modification of the Embodiment) In this modification, another example of an information processing device and information processing method that contribute to the appropriate selection of three-dimensional points will be described.

[0231] Figure 24 is a flowchart showing an example of processing performed by the information processing device 10 in a modified example of this embodiment. The flowchart shown in Figure 24 illustrates an information processing method performed by the information processing device 10 in a modified example of this embodiment.

[0232] The information processing device 10 comprises a circuit and a memory connected to the circuit, and the circuit performs a series of processes shown in Figure 24.

[0233] In step S501, the circuit acquires first identification information that identifies a part of the three-dimensional model using user input on a reference image in which the three-dimensional model viewed from a first viewpoint is displayed.

[0234] In step S502, the circuit determines at least one second viewpoint based on the first specific information acquired in step S501.

[0235] In step S503, the circuit displays at least one additional image corresponding to each of the at least one second viewpoint determined in step S502. Each of the at least one additional image displays the region as seen from the second viewpoint corresponding to that additional image.

[0236] Through the series of processes shown in Figure 24, the information processing device 10 contributes to the appropriate selection of three-dimensional points.

[0237] In the above embodiment, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Here, the software that implements the information processing device, etc., of the above embodiment is the following program.

[0238] In other words, this program causes a computer to execute an information processing method in which an information processing device uses a circuit and a memory connected to the circuit, the method being to acquire first specific information that identifies a part of the three-dimensional model using user input to a reference image displaying a three-dimensional model viewed from a first viewpoint, determine at least one second viewpoint based on the acquired first specific information, display at least one additional image corresponding to each of the at least one second viewpoint, and display in each of the at least one additional image the region viewed from the second viewpoint corresponding to the additional image.

[0239] Although the present invention has been described above based on embodiments of one or more information processing devices, the present invention is not limited to these embodiments. Without departing from the spirit of the present invention, various modifications that a person skilled in the art can conceive of may be applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments.

[0240] This disclosure is applicable to information processing devices or software that perform information processing on three-dimensional models.

[0241] 9A, 9B, 32, 33, 34, 51, 52, 53, 61, 62, 63, 81, 97, 98 points 10 Information processing device 11 Processor 12 Memory 31, 41 Frame 42 Cursor 65 Arrow 71, 72, 73, 74, 75, 76, 77, 78, 79, 82 Candidate points 91, 92, 93 Image 95 Three-dimensional model 101 Storage unit 102 Acquisition unit 103 Generation unit 104 UI ​​unit 104A Display unit 104B Reception unit 105 Area identification unit 106 Target point identification unit 107 Processing unit

Claims

1. An information processing device comprising a circuit and a memory connected to the circuit, wherein the circuit acquires first identifying information that identifies a part of the three-dimensional model using user input to a reference image on which a three-dimensional model viewed from a first viewpoint is displayed, determines at least one second viewpoint based on the acquired first identifying information, displays at least one additional image corresponding to each of the at least one second viewpoint, and displays the region as viewed from the second viewpoint corresponding to the additional image.

2. The information processing apparatus according to claim 1, wherein the circuit further acquires second identifying information that identifies three-dimensional points included in the three-dimensional model using user input for at least one additional image, and performs information processing on the three-dimensional model using the acquired second identifying information.

3. The information processing apparatus according to claim 2, wherein the information processing relating to the three-dimensional model includes a process of rotating the three-dimensional model, a process of moving the three-dimensional model, a process of measuring the three-dimensional model, or a process of changing the position of the viewpoint from which the three-dimensional model is viewed.

4. The information processing apparatus according to claim 1, wherein the at least one second viewpoint is a second viewpoint, wherein the region of the three-dimensional model is displayed without being obscured in an additional image showing the three-dimensional model as seen from the second viewpoint.

5. The information processing apparatus according to claim 1, which includes a second viewpoint, wherein the at least one second viewpoint is located on the normal of the surface defined by the region.

6. The information processing apparatus according to claim 1, wherein the at least one second viewpoint includes two second viewpoints, the direction from each of the two second viewpoints toward the region is perpendicular to the direction from the first viewpoint toward the region, and the directions from the two second viewpoints toward the region are perpendicular to each other.

7. The information processing apparatus according to claim 1, wherein the three-dimensional model has a predetermined upward direction, a rightward direction perpendicular to the upward direction, and a leftward direction opposite to the rightward direction, and the at least one second viewpoint includes a second viewpoint from which the three-dimensional model is viewed from above, a second viewpoint from which the three-dimensional model is viewed from the right, and a second viewpoint from which the three-dimensional model is viewed from the left.

8. The information processing apparatus according to claim 1, wherein the three-dimensional model is a three-dimensional model generated by a learning process that takes as input multiple images, each displaying an object viewed from multiple viewpoints, and the at least one second viewpoint includes the same second viewpoint as one of the multiple viewpoints.

9. The information processing apparatus according to claim 1, wherein the at least one second viewpoint includes a second viewpoint specified by a user who has viewed the reference image.

10. The information processing apparatus according to claim 1, further comprising: generating the at least one additional image to be displayed before displaying the at least one additional image; generating one additional image included in the at least one additional image, determining the field of view of the one additional image according to the degree of density of the plurality of three-dimensional points included in the region of the three-dimensional model; and generating the one additional image having the determined field of view.

11. The information processing apparatus according to claim 10, wherein determining the field of view of the first additional image is such that the more densely packed the plurality of three-dimensional points included in the region of the three-dimensional model, the larger the field of view of the first additional image.

12. The information processing apparatus according to claim 2, wherein acquiring the second specific information includes acquiring the second specific information specifying the three-dimensional point, which is input by an operation performed by a user who has viewed the at least one additional image.

13. The information processing apparatus according to claim 12, wherein one additional image included in the at least one additional image includes a figure showing feature points included in the region.

14. The information processing apparatus according to claim 1, wherein one additional image included in the at least one additional image includes a three-dimensional point included in the three-dimensional model, the three-dimensional point identified by the second specific information, and a figure indicating a position symmetric with respect to a predetermined reference.

15. The information processing apparatus according to claim 1, wherein the at least one additional image includes a first additional image and a second additional image, the circuit further acquires second specific information that specifies a three-dimensional point included in the three-dimensional model, which is input to the first additional image, and the display of the at least one additional image includes, after acquiring the second specific information, displaying a figure showing the three-dimensional point specified by the second specific information together with the second additional image.

16. The information processing apparatus according to claim 2, wherein the circuit further updates the displayed reference image or the at least one additional image after performing the information processing, in accordance with the results of the information processing.

17. The information processing apparatus according to claim 1, wherein the circuit further acquires second identification information that identifies a three-dimensional point included in the three-dimensional model, which is input to one of the at least one additional images; displays a new additional image in which the three-dimensional point specified by the second identification information and other three-dimensional points are displayed, as viewed from a third viewpoint different from the second viewpoint corresponding to the one additional image; and acquires third identification information that identifies one of the three-dimensional points among the three-dimensional point and the other three-dimensional point, which is input to the new additional image.

18. An information processing method performed by an information processing device using a circuit and a memory connected to the circuit, comprising: acquiring first specific information that identifies a part of the three-dimensional model using user input to a reference image on which a three-dimensional model viewed from a first viewpoint is displayed; determining at least one second viewpoint based on the acquired first specific information; displaying at least one additional image corresponding to each of the at least one second viewpoint; and displaying the region as viewed from the second viewpoint corresponding to the additional image in each of the at least one additional image.

19. A program that causes an information processing device to execute the information processing method described in claim 18.

Images