Model training device, interpolation device, and model training interpolation apparatus

Abstract

Provided is a model training device capable of training a model flexible to the number of vertices and the structure of data. This model training device comprises: a training data generation unit that generates, for each of a plurality of rooms having different shapes, training data comprising a graph structure including vertices arranged at equal intervals so as to correspond to the shape of the room, and sound pressure at each vertex of the graph structure; and a model training unit that trains, on the basis of the training data, a GNN model, which receives, as inputs, the graph structure and the sound pressure at one or more vertices selected from the graph structure, and outputs the sound pressure at each vertex of the graph structure.

Description

Model learning device, interpolation device, model learning and interpolation device

[0001] The present disclosure relates to a model learning device that learns a model for use in sound field interpolation, an interpolation device that performs sound field interpolation based on a model, and a model learning and interpolation device that learns a model and performs interpolation.

[0002] The research field related to estimating the distribution of physical quantities of sound such as sound pressure, that is, spatial acoustics, is called sound field estimation, and it forms the foundation of various applied technologies related to spatial acoustic processing.

[0003] A technique for estimating sound field information at unobserved positions based on sound field information obtained from limited observation positions is called sound field interpolation.

[0004] For example, Non-Patent Document 1 is known as a CNN-based sound field interpolation method.

[0005] Francesc Lluis, Pablo Martinez-Nuevo, Martin Bo Moller, Sven Ewan Shepstone, “Sound field reconstruction in rooms: inpainting meets super-resolution", [online], submitted on 30 Jan 2020 (v1), last revised 6 Aug 2020(v2), [searched February 27, 2025], Internet<URL: https: / / arxiv.org / pdf / 2001.11263 >

[0006] In the CNN-based sound field interpolation method, there are constraints on the structure of input data, and the number and arrangement of microphone arrays could not be changed.

[0007] Accordingly, an object of the present disclosure is to provide a model learning device capable of learning a model that is flexible with respect to the number of vertices and the structure of data.

[0008] The model learning device of the present disclosure includes a learning data generation unit and a model learning unit.

[0009] The learning data generation unit generates learning data for each of several rooms of different shapes, consisting of a graph structure containing vertices arranged at equal intervals corresponding to the shape of the room, and the sound pressure at each vertex of the graph structure.

[0010] The model learning unit, based on the training data, takes a graph structure and the sound pressure at one or more vertices selected from the graph structure as inputs, and learns a GNN model that outputs the sound pressure at each vertex of the graph structure.

[0011] The model learning device described herein can learn models that are flexible in terms of the number of vertices and the structure of the data.

[0012] A block diagram showing the functional configuration of the model learning interpolation device (model learning device, interpolation device) of Example 1. A flowchart showing the operation of the model learning device of Example 1. A diagram illustrating the sound pressure distribution of multiple rooms of different shapes. A diagram illustrating a graph structure including vertices arranged according to the shape of the room. A diagram showing the characteristics of the GNN model to be learned. A diagram showing the edge characteristics of the GNN model to be learned. A diagram showing the edge characteristics of the GNN model to be learned. A flowchart showing the operation of the interpolation device of Example 1. A diagram showing an example of the functional configuration of a computer.

[0013] The embodiments of this disclosure will be described in detail below. Components having the same function will be numbered the same, and redundant explanations will be omitted.

[0014] The functional configuration of the model learning interpolation device 1 of Embodiment 1 will be described below with reference to Figure 1. As shown in the figure, the model learning interpolation device 1 of this embodiment includes a learning data generation unit 111, a model learning unit 112, a model storage unit 120, and an interpolation unit 121. The model learning interpolation device 1 of this embodiment performs two phases of operation. The first phase is the learning phase processing performed by the learning data generation unit 111 and the model learning unit 112, and the second phase is the interpolation phase processing performed by the interpolation unit 121.

[0015] The processing of the first phase and the processing of the second phase may be separated into processing of separate devices. As shown in the figure, the model learning interpolation device 1 may be separated into two devices, the model learning device 11 and the interpolation device 12, with the model learning device 11 including a learning data generation unit 111 and a model learning unit 112, and the interpolation device 12 including a model storage unit 120 and an interpolation unit 121.

[0016] The operation and characteristics of each component will be explained in detail below with reference to Figures 2-8.

[0017] <Learning Data Generation Unit 111> The learning data generation unit 111 generates learning data for each of several rooms of different shapes, consisting of a graph structure of a GNN (Graph Neural Network) where vertices (also called nodes) are arranged at equal intervals corresponding to the shape of the room, and the relationships between each vertex are called edges, and sound pressure at each vertex of the graph structure (S111, Figure 2).

[0018] The shape of the room used as training data can be anything. Figure 3 shows examples of room shapes and sound pressure distributions in each room. The sound pressure distribution can be measured values ​​or simulation data. When preparing simulation data, in addition to the room shape, parameters such as the acoustic reflectivity of the walls, the coordinates of the sound source, and the signal to be reproduced from the sound source are required. There are no particular frequency restrictions on the signal to be reproduced from the sound source, but it is preferable not to use frequencies above the frequency at which spatial aliasing occurs.

[0019] As mentioned above, in step S111, a graph structure containing vertices and edges is assigned to these rooms. Figure 4 shows an example of assigning a graph structure to each room. The white circles in the figure indicate the positions of vertices. For the sake of clarity, the display of edges has been omitted in the figure. Edges will be discussed later.

[0020] The vertices represent the positions where sound pressure is observed or interpolated. The spacing between vertices is set to be uniform. Maintaining consistent vertex spacing across all training data is preferable because it prevents spatial aliasing.

[0021] <Model Learning Unit 112> Based on the training data, the model learning unit 112 takes a graph structure and the sound pressure at one or more vertices selected from the graph structure as inputs and learns a GNN model that outputs the sound pressure at each vertex of the graph structure (S112).

[0022] Figure 5 shows the characteristics of the GNN model learned in step S112. In the figure, the black circles represent selected vertices (also called observation points), and the white circles represent vertices where the sound pressure is unknown. From the viewpoint of interpolation accuracy, it is preferable to have multiple observation points. In the example shown in the figure, there are three observation points.

[0023] The GNN model is characterized by taking a graph structure containing vertices and edges (not shown in the diagram) and the sound pressure at the vertices indicated by black circles as input, interpolating the sound pressure at vertices other than the observation point as shown by the dot-hatched circles, and outputting the sound pressure at each vertex of the graph structure.

[0024] While the position of the observation points on the graph structure may be unknown, inputting the position of the observation points on the graph structure into the model improves the accuracy of interpolation and is therefore preferable.

[0025] <<Structure of GNN>> The following variations are possible in terms of types of GNNs, depending on the differences in the intermediate network and input features.

[0026] [1. Graph Convolution Network (GCN)] This model uses a graph convolution network for its intermediate layer. It aggregates messages weighted by edge weights. [2. Graph Attention Network (GAT)] At the aggregation stage, it calculates the weights of messages from neighboring vertices from vertex features and aggregates them. [3. Graph Attention Network + Coordinates of Each Vertex (GAT (+coordinate))] This model adds 3D coordinates (the difference for edges) as input for vertex and edge features, in addition to the model in 2. There is no need to set edge weights.

[0027] ≪Graph Structure≫ Since GNNs have a structure where information is exchanged only from adjacent vertices in each layer, in a multi-layer GNN (L layers, where L is a natural number greater than or equal to 2), the information that can be used to predict a given vertex is limited to vertices whose path length from that vertex is L or less. Therefore, if the graph structure only has edges connecting nearby vertices, information will not be transmitted. Thus, it is preferable to connect distant vertices with edges as well.

[0028] In this embodiment, the GNN used for training connects adjacent vertices vertically and horizontally with edges, as illustrated in Figure 6, and also connects distant vertices with edges, as illustrated in Figure 7.

[0029] In the example shown in Figure 7, there are 2, 4, ..., 2 in the vertical and horizontal directions. ｎ The vertices, separated by a number of spaces (n is a natural number), are connected by edges. To distinguish between nearby and distant edges, it is preferable to assign a larger weight to nearby edges and a smaller weight to distant edges.

[0030] <Model storage unit 120> The model storage unit 120 stores the GNN model learned in step S112.

[0031] <Interpolation Unit 121> The interpolation unit 121 inputs the graph structure and the sound pressure at one or more vertices selected from the graph structure into the GNN model and obtains the sound pressure at each vertex of the graph structure (S121, Figure 8).

[0032] <Effects> According to the model learning interpolation device 1 of this embodiment, it is possible to interpolate the sound pressure at many vertices using sound pressure information obtained from a small number of observation points, thereby obtaining the sound field (sound pressure distribution) for the entire space.

[0033] Furthermore, according to the model learning interpolation device 1 of this embodiment, since the entire graph is not referenced when updating each vertex, the GNN, which has flexible properties with respect to the number of vertices and graph structure, can be used to deform the graph structure to match the room shape, allowing the model to be trained using sound pressure distributions of various room shapes as training data, and furthermore, sound field interpolation can be performed for rooms of various shapes using the trained model.

[0034] Furthermore, according to the model learning interpolation device 1 of this embodiment, since the room can be divided into smaller sections by utilizing a GNN, it is easier to suppress the occurrence of spatial aliasing.

[0035] Furthermore, the model learning interpolation device 1 of this embodiment has the advantage of using a GNN, which allows for a relatively small number of parameters and easy sound field interpolation. For example, in a single-layer GCN, the parameters are W ∈ R^(d ｉｎ ×d ｏｕｔ ) is the only exception (however, d ｉｎ d ｏｕｔ (where is the dimension of the vertex features in the input and output).

[0036] The "device for use (terminal)" may also include devices (terminals) for using the device, system, or method of disclosure via a network (telecommunication line). The "device for use (terminal)" may be equipped with functions necessary to obtain the effects of implementing the device, system, or method of disclosure (e.g., control functions, decoding functions, restoration functions, input / output functions, etc.).

[0037] [Processors, Programs, Recording Media] The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.

[0038] In this specification, circuitry, units, and means refer to hardware programmed to or configured to implement the functions described. Said hardware may be any hardware disclosed in the present specification, or any hardware known to be programmed to or configured to implement the described functions.

[0039] Where said hardware is a processor considered to be a type of circuitry, said circuitry, means, or unit is a combination of hardware and software used to configure said hardware and / or processor.

[0040] The various processes described above can be implemented by causing a recording unit 10020 of a computer 10000 shown in FIG. 9 to load a program that executes each step of the above method, and causing a control unit 10010, an input unit 10030, an output unit 10040, a display unit 10050 and the like to operate in accordance with the program.

[0041] A program describing the content of this processing can be recorded on a computer-readable recording medium. Any type of computer-readable recording medium may be used, for example, including a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, and the like.

[0042] A program describing the content of this processing may be included in a computer program product.

[0043] Furthermore, distribution of this program may be performed, for example, by selling, assigning, or lending portable recording media such as DVDs or CD-ROMs having the program recorded thereon. Alternatively, the program may be distributed by storing the program in a storage device of a server computer, and transferring the program from the server computer to another computer via a network.

[0044] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0045] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

Claims

1. A model learning device comprising: a learning data generation unit that generates learning data consisting of a graph structure including vertices arranged at equal intervals corresponding to the shape of each of several rooms of different shapes, and sound pressure at each vertex of the graph structure; and a model learning unit that learns a GNN model based on the learning data, taking the graph structure and the sound pressure at one or more vertices selected from the graph structure as inputs and outputting the sound pressure at each vertex of the graph structure.

2. An interpolation device comprising a graph structure for each of several rooms of different shapes, including vertices arranged at equal intervals corresponding to the shape of the room, and a GNN model learned based on learning data consisting of sound pressure at each vertex of the graph structure, wherein the interpolation device includes a model storage unit that stores the graph structure and a GNN model that takes the sound pressure at one or more vertices selected from the graph structure as input and outputs the sound pressure at each vertex of the graph structure, and an interpolation unit that inputs the graph structure and the sound pressure at one or more vertices selected from the graph structure to the GNN model and obtains the sound pressure at each vertex of the graph structure.

3. A model learning interpolation device comprising: a learning data generation unit that generates learning data consisting of a graph structure including vertices arranged at equal intervals corresponding to the shape of each of several rooms of different shapes, and sound pressure at each vertex of the graph structure; a model learning unit that learns a GNN model based on the learning data, taking the graph structure and the sound pressure at one or more vertices selected from the graph structure as inputs, and outputting the sound pressure at each vertex of the graph structure; and an interpolation unit that inputs the graph structure and the sound pressure at one or more vertices selected from the graph structure to the GNN model and obtains the sound pressure at each vertex of the graph structure.

4. A model learning device according to claim 1, wherein the model learning unit learns a GNN model that takes the graph structure, the sound pressure at one or more vertices selected from the graph structure, and the position of one or more vertices selected from the graph structure on the graph structure as inputs, and outputs the sound pressure at each vertex of the graph structure.

Images