Information processing device, information processing method, and recording medium

Abstract

An information processing device according to the present disclosure comprises a preprocessing unit that has a sorting unit and a parameter allocation unit. The sorting unit sorts multidimensional tensors. The parameter allocation unit allocates learnable parameters to the sorted multidimensional tensors, and generates and outputs converted multidimensional tensors obtained by converting histograms of the multidimensional tensors.

Description

Information processing device, information processing method, and recording medium

[0001] This disclosure relates to an information processing device, an information processing method, and a recording medium.

[0002] There are data correction technologies such as image correction. Patent document 1 discloses a technology for constructing a learning model that accepts image input and outputs a medical image with improved resolution while reducing noise.

[0003] Japanese Patent Publication No. 2022-70035

[0004] One example of the purpose of this disclosure is to advance data correction techniques.

[0005] According to one aspect of this disclosure, an information processing device is provided that has a preprocessing means comprising: sorting means for sorting a multidimensional tensor; parameter assignment means for assigning learnable parameters to the sorted multidimensional tensor and generating and outputting a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor.

[0006] Furthermore, according to one aspect of this disclosure, an information processing method is provided in which one or more computers sort a multidimensional tensor, assign learnable parameters to the sorted multidimensional tensor, and generate and output a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor.

[0007] Furthermore, according to one aspect of this disclosure, a recording medium is provided which records a program that causes a computer to perform a sorting step of sorting a multidimensional tensor, and a parameter assignment step of assigning learnable parameters to the sorted multidimensional tensor, and generating and outputting a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor.

[0008] According to this example of disclosure, data correction techniques can be advanced.

[0009] FIG. 1 is a diagram showing an example of a functional block diagram of an information processing apparatus. FIG. 2 is a flowchart showing an example of the processing flow of the information processing apparatus. FIG. 3 is a diagram showing an example of the hardware configuration of the information processing apparatus. FIG. 4 is a diagram showing an example of the processing executed by the information processing apparatus. FIG. 5 is a flowchart showing another example of the processing flow of the information processing apparatus. FIG. 6 is a diagram showing another example of the processing executed by the information processing apparatus. FIG. 7 is a diagram showing another example of the functional block diagram of the information processing apparatus. FIG. 8 is a diagram showing another example of the functional block diagram of the information processing apparatus. FIG. 9 is a diagram showing another example of the functional block diagram of the information processing apparatus. FIG. 10 is a diagram showing another example of the functional block diagram of the information processing apparatus. FIG. 11 is a flowchart showing another example of the processing flow of the information processing apparatus. FIG. 12 is a diagram showing another example of the functional block diagram of the information processing apparatus. FIG. 13 is a diagram showing another example of the functional block diagram of the information processing apparatus. FIG. 14 is a flowchart showing another example of the processing flow of the information processing apparatus.

[0010] Hereinafter, embodiments of this disclosure will be described with reference to the drawings. In this disclosure, the drawings are associated with one or more embodiments. Also, in all the drawings, the same components are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0011] <<First Embodiment>> FIG. 1 is a functional block diagram showing an overview of an information processing apparatus 10. FIG. 2 is a flowchart showing an example of the processing flow executed by the information processing apparatus 10.

[0012] As shown in FIG. 1, the information processing apparatus 10 includes a preprocessing unit 11. The preprocessing unit 11 includes a sorting unit 111 and a parameter assignment unit 112. By these functional units, the processing shown in the flowchart of FIG. 2 is executed.

[0013] In S10, the sorting unit 111 sorts the multi-dimensional tensor. In S11, the parameter assignment unit 112 assigns learnable parameters to the sorted multi-dimensional tensor, generates and outputs a transformed multi-dimensional tensor obtained by transforming the histogram of the multi-dimensional tensor.

[0014] Incidentally, as one of the data correction techniques, an image correction technique called histogram equalization is known. In this technique, an image is corrected so that the histogram of pixel values becomes overall flat, that is, the pixel values become overall uniform distribution (flat). By such histogram equalization, the contrast of the image is enhanced and the image becomes easier to recognize.

[0015] However, it is unclear whether making the histogram of pixel values overall "flat" is optimal for making the image easier to recognize. That is, there is a possibility that a non-flat state is optimal.

[0016] Also, the "flat" state has a certain spread. The "flat state of the histogram of image values" that can be obtained by histogram equalization does not have to be completely flat, and it is said that it is sufficient if local concentration is eliminated and it becomes overall flat to some extent. If the optimal state among such "flat" states with spread can be achieved, it is considered that the image becomes easier to recognize.

[0017] As described above, in the image correction technique, in order to further improve the visibility of the corrected image, it is desired to make the histogram of pixel values of the corrected image in a more optimal state. As a means to solve this, it is conceivable to use machine learning to optimize the state of the histogram of pixel values. However, histogram equalization is not suitable for machine learning because the processes of creating a histogram and stretching the interval of the histogram, etc. in its calculation process are non-differentiable and error backpropagation is not possible. In histogram equalization, the histogram of pixel values is calculated, the histogram is converted into a normalized cumulative histogram (tone curve), and the interval of the histogram is stretched so that the histogram becomes a uniform distribution by multiplying the tone curve by 255.

[0018] The information processing apparatus 10 is configured to be able to reduce such problems. The information processing apparatus 10 generates a transformed multi-dimensional tensor obtained by transforming the histogram of the multi-dimensional tensor to be processed by assigning "learnable parameters" to each element of the multi-dimensional tensor to be processed and updating the value of each element of the multi-dimensional tensor to be processed.

[0019] Learnable parameters can be optimized using machine learning because they are learnable. Learnable parameters are, for example, composed of a one-dimensional tensor, with each element having a parameter. Each of these element parameters has a gradient value in addition to the parameter value, and this gradient value can be calculated by backpropagation of errors from the parameters of a subsequent classifier or the like. Furthermore, the parameter values ​​can be updated using the gradient value and the learning rate. According to the information processing device 10 that uses such learnable parameters to transform the histogram of the multidimensional tensor to be processed and generate a transformed multidimensional tensor, the state of the histogram of the multidimensional tensor to be processed can be optimized using machine learning.

[0020] Furthermore, the information processing device 10 sorts the multidimensional tensor and then assigns learnable parameters to the sorted multidimensional tensor. By assigning learnable parameters to a multidimensional tensor sorted in a predetermined order, it becomes possible to update the values ​​of each element while maintaining the predetermined order relationship between multiple elements.

[0021] For example, when a multidimensional tensor is sorted in descending order of its values, it is possible to update the values ​​of multiple elements while maintaining that order. This can be achieved by preparing learnable parameters sorted in descending order of their values, and then updating the value of the nth element of the sorted multidimensional tensor with the value of the nth element of the learnable parameters sorted in descending order. With this process, even after updating the values ​​of each element of the sorted multidimensional tensor with the learnable parameters, the values ​​of each element of the sorted multidimensional tensor remain sorted in descending order. In other words, the relative values ​​of the values ​​among multiple elements are maintained while the values ​​of each element are updated.

[0022] As explained above, the information processing device 10 enables the development of data correction technology.

[0023] <<Second Embodiment>> <Overview> The information processing device 10 of the second embodiment is a concrete implementation of the configuration of the information processing device 10 of the first embodiment. In the second embodiment, the configuration of the preprocessing unit 11, which performs processing in both the learning stage and the estimation stage, will be described in detail. The specific processing of the learning stage and the estimation stage will be described in a separate embodiment.

[0024] <Hardware Configuration> First, an example of the hardware configuration of the information processing device 10 will be described. Each functional unit of the information processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the implementation method and the device. The software includes programs that are pre-installed at the time of shipment of the device, as well as programs downloaded from recording media such as CDs (Compact Discs) or from servers on the Internet.

[0025] Figure 3 is a block diagram illustrating the hardware configuration of the information processing device 10. As shown in Figure 3, the information processing device 10 includes a processor 1A, memory 2A, input / output interface 3A, peripheral circuitry 4A, and bus 5A. The peripheral circuitry 4A includes various modules. The information processing device 10 does not necessarily have peripheral circuitry 4A. The information processing device 10 may also be composed of multiple physically and / or logically separated devices. In this case, each of the multiple devices may have the above hardware configuration.

[0026] Bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to send and receive data to and from each other. The processor 1A is a processing unit such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit). Memory 2A is a memory such as RAM (Random Access Memory) or ROM (Read Only Memory). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes interfaces for connecting to a communication network such as the Internet. Input devices include, for example, a keyboard, mouse, microphone, physical buttons, touch panel, etc. Output devices include, for example, a display, projection device, speaker, printer, mailer, etc. The processor 1A can issue commands to each module and perform calculations based on their calculation results.

[0027] <Functional Configuration> Next, the functional configuration of the information processing device 10 will be described in detail. Figure 1 is an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 has a preprocessing unit 11. The preprocessing unit 11 has a sorting unit 111 and a parameter assignment unit 112.

[0028] The preprocessing unit 11 is configured to be differentiable. This means that the sorting unit 111 and the parameter assignment unit 112 of the preprocessing unit 11 are defined / designed to perform differentiable processing. The processing performed by such a sorting unit 111 and parameter assignment unit 112 will be described below.

[0029] The sorting unit 111 sorts the multidimensional tensor to be processed. Hereafter, the multidimensional tensor to be processed will be referred to as the "multidimensional tensor to be processed".

[0030] The "multidimensional tensor to be processed" may be a 3D tensor, a 2D tensor, or a tensor of any other dimension. A 3D tensor represents, for example, the data set for each pixel of a color image (e.g., an RGB image). In this case, the three dimensions represent width, height, and RGB channels. A 2D tensor represents, for example, the data set for each pixel of a grayscale image (black and white image). In this case, the two dimensions represent width and height.

[0031] The type of multidimensional tensor that becomes the target multidimensional tensor differs between the learning stage and the estimation stage after learning. The "learning stage" is the stage in which the values ​​of learnable parameters are optimized using machine learning. The "estimation stage" is the stage in which the target multidimensional tensor is appropriately corrected using the learned learnable parameters that have been trained. Processing by the preprocessing unit 11 is performed in both the learning stage and the estimation stage.

[0032] During the learning phase, the multidimensional tensor extracted from a pre-prepared training dataset becomes the multidimensional tensor to be processed. During the estimation phase, the multidimensional tensor that is subject to correction becomes the multidimensional tensor to be processed.

[0033] The information processing device 10 can obtain the above-described multidimensional tensor to be processed by utilizing widely known techniques.

[0034] "Acquisition" includes at least one of the following: the device retrieving data or information stored in another device or storage medium (active acquisition), and the device inputting data or information output from another device into its own device (passive acquisition). Examples of active acquisition include making a request to another device and receiving a reply, and accessing and reading data from another device or storage medium. Examples of passive acquisition include receiving information that is delivered (or transmitted, push notification, etc.). Furthermore, acquisition may also involve selecting and acquiring data or information from among the received data or information, or selecting and receiving data or information that has been delivered.

[0035] The sorting unit 111 can perform the following operations as the operation to "sort the multidimensional tensor to be processed": • Operation to convert the multidimensional tensor to be processed into a one-dimensional vector • Operation to sort the one-dimensional vector according to a predetermined rule and generate a sorted one-dimensional vector

[0036] "Process to convert a multidimensional tensor to be processed into a one-dimensional vector" In the process to convert a multidimensional tensor to be processed into a one-dimensional vector, the sorting unit 111 generates a one-dimensional vector by arranging multiple elements that make up the multidimensional tensor to be processed in a predetermined order.

[0037] Figure 4 shows an overview of the process. Figure 4 shows how the "multidimensional tensor to be processed" is converted into a "one-dimensional vector". As shown in Figure 4, the multidimensional tensor to be processed contains multiple elements arranged in two directions: width and height. In the example in the figure, the multidimensional tensor to be processed is x １ 〜x Ｎ It contains N elements. In the example shown in the figure, the sorting unit 111 performs a scan from left to right and then sequentially from top to bottom, and generates a one-dimensional vector by arranging the elements in the order they were scanned. Note that the arrangement of multiple elements of the multidimensional tensor being processed in one direction is not limited to the example given here.

[0038] The elements that make up a one-dimensional vector are "x ｐ This is represented by `x`. `p` is the sequential number assigned to multiple elements in the order of the one-dimensional vector. `p` is an integer between 1 and N (inclusive). ｐ This is the value of the p-th element, for example, a pixel value.

[0039] "Process of sorting a one-dimensional vector according to predetermined rules to generate a sorted one-dimensional vector" After obtaining a one-dimensional vector as described above, the sorting unit 111 sorts the one-dimensional vector according to predetermined rules to generate a sorted one-dimensional vector. The sorting unit 111 can sort multiple elements of a one-dimensional vector based on the magnitude of the values ​​of each element to generate a sorted one-dimensional vector.

[0040] In the sorting rule, it may be defined that a plurality of elements of a one-dimensional vector are sorted in descending order of the value of each element. In this case, the sorting unit 111 sorts a plurality of elements of the one-dimensional vector in descending order of the value of each element. Alternatively, in the sorting rule, it may be defined that a plurality of elements of a one-dimensional vector are sorted in ascending order of the value of each element. In this case, the sorting unit 111 sorts a plurality of elements of the one-dimensional vector in ascending order of the value of each element.

[0041] The outline of the process is shown in FIG. 4. FIG. 4 shows a state in which a "one-dimensional vector" is sorted to generate a "sorted one-dimensional vector". As shown in FIG. 4, in the one-dimensional vector, N elements x １ to x Ｎ are arranged in the order of serial number p. And in the sorted one-dimensional vector, N elements x １ to x Ｎ are arranged in an order different from the order of serial number p. In the sorted one-dimensional vector, for example, N elements x １ to x Ｎ are arranged in descending order of the value of each element. Or, in the sorted one-dimensional vector, for example, N elements x １ to x Ｎ are arranged in ascending order of the value of each element.

[0042] Returning to FIG. 1, the parameter assignment unit 112 assigns learnable parameters to the sorted multi-dimensional tensor to be processed, and generates and outputs a converted multi-dimensional tensor obtained by converting the histogram of the multi-dimensional tensor to be processed. The parameter assignment unit 112 is configured as a module, for example.

[0043] The term "sorted multidimensional tensor to be processed" refers to the output obtained by performing the "sorting process of the multidimensional tensor to be processed" described above on the "multidimensional tensor to be processed". Specifically, the sorted multidimensional tensor to be processed refers to the sorted one-dimensional vector described above. In other words, "assigning learnable parameters to the sorted multidimensional tensor to be processed" means assigning learnable parameters to the sorted one-dimensional vector. The parameter assignment unit 112 assigns learnable parameters to each of the "multiple elements arranged in a predetermined order" that constitute the sorted one-dimensional vector.

[0044] A "learnable parameter" is a parameter whose value can be optimized using machine learning. In other words, a learnable parameter is a parameter whose value can be updated (optimized) through machine learning.

[0045] In the second embodiment, the learnable parameters are constructed by arranging the same number of elements as the sorted one-dimensional vector. That is, the sorted one-dimensional vector and the learnable parameters are constructed by arranging the same number of elements.

[0046] Figure 4 shows an example of learnable parameters. In the example shown, the learnable parameters are y １ or y Ｎ It contains N elements. The elements that make up the learnable parameters are "y ｑ This is represented by `q`. q is a sequential number of multiple elements assigned to the order of the learnable parameters. q is an integer between 1 and N (inclusive). y ｑ This indicates the value of the q-th element.

[0047] It is preferable that the learnable parameters are sorted based on the magnitude of each element's value. However, the learnable parameters do not necessarily have to be sorted. The sorting rule is the same as the sorting rule used when generating a sorted one-dimensional vector from the one-dimensional vector described above. That is, if the sorting rule when generating a sorted one-dimensional vector from a one-dimensional vector is, for example, "sort in descending order of each element's value," then the sorting rule for the learnable parameters will be the same. Also, if the sorting rule when generating a sorted one-dimensional vector from a one-dimensional vector is, for example, "sort in ascending order of each element's value," then the sorting rule for the learnable parameters will be the same.

[0048] The values ​​of the learnable parameters may differ between the training phase and the post-training estimation phase. During the training phase, the values ​​of each element of the learnable parameters are not yet optimized. During the training phase, the values ​​of each element of the learnable parameters are either initial values ​​or values ​​updated by iterative machine learning. During the estimation phase, the values ​​of each element of the learnable parameters are values ​​optimized by prior machine learning.

[0049] The parameter assignment unit 112 can perform the following operations as "the process of assigning learnable parameters to a sorted multidimensional tensor to be processed and generating a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor to be processed": • The process of determining learnable parameters to be applied to the sorted one-dimensional vector (the sorted multidimensional tensor to be processed) • The process of updating the sorted one-dimensional vector with the learnable parameters to generate an updated one-dimensional vector • The process of inverse sorting the updated one-dimensional vector to generate an inversely sorted one-dimensional vector • The process of generating a transformed multidimensional tensor using the inversely sorted one-dimensional vector

[0050] "Process to determine learnable parameters to apply to the sorted one-dimensional vector (the sorted multidimensional tensor to be processed)" First, the parameter assignment unit 112 can determine learnable parameters to apply to the sorted one-dimensional vector (the sorted multidimensional tensor to be processed).

[0051] During the learning phase, the values ​​of each element of the learnable parameters determined here are either initial values ​​or values ​​updated by repeated machine learning. During the estimation phase, the values ​​of each element of the learnable parameters determined here are values ​​optimized by prior machine learning.

[0052] "Process to update a sorted one-dimensional vector with learnable parameters to generate an updated one-dimensional vector" Next, the parameter assignment unit 112 updates the value of the nth element in the sorted one-dimensional vector using the value of the nth element in the learnable parameters to generate an updated one-dimensional vector.

[0053] Figure 4 shows how the value of the nth element in the sorted one-dimensional vector is updated using the value of the nth element in the learnable parameters to generate an updated one-dimensional vector. In the example shown, the first element in the sorted one-dimensional vector is x １８ Therefore, the second element in the sequence is x １９ Therefore, the third element in the sequence is x ３７ That is the case.

[0054] In the example shown, the first element x is the one whose position in the sorted one-dimensional vector is determined by the order of the elements. １８ The first element y in the learnable parameters is the element whose order of sorting is... １ Using z １８ It has been updated. The elements that make up the one-dimensional vector after the update are "z ｐ It is indicated by "x ｐ ga z ｐ It will be updated.

[0055] The parameter assignment unit 112 can update the value of the nth element in the sorted one-dimensional vector by replacing it with one of the following values: • The value of the nth element in the learnable parameters (Example 1) • A calculated value derived from the value of the nth element in the sorted one-dimensional vector and the value of the nth element in the learnable parameters (Example 2)

[0056] In Example 1, the order of the updated one-dimensional vector shown in Figure 4 is such that the first element z １８ The first element y in the learnable parameters is the element whose order of sorting is... １ (z) will have the same value. １８ = y １ ).

[0057] In Example 2, the order of the updated one-dimensional vector shown in Figure 4 is such that the first element z １８ This is the first element x in the sorted one-dimensional vector. １８ And the order of the learnable parameters is the first element y １ This is the calculated value derived from z. １８ is, x １８ and y １ The product of (z) is also acceptable. １８ = x １８ ×y １ ). In addition, z １８ is, x １８ and y １ The average value of z is also acceptable ( １８ = (x １８ +y １ ) / 2). In addition, z １８ is, x １８ and y １ A weighted average of (z １８ = (x １８ α + y １ (β) / (α+β)). In the case of a weighted average, x １８ Rather than the weight α of y １ The weight β may be increased (α < β).

[0058] "Process to reverse sort the updated one-dimensional vector and generate a reverse-sorted one-dimensional vector" Next, the parameter assignment unit 112 returns multiple elements of the updated one-dimensional vector to their original order before sorting the one-dimensional vector to generate a reverse-sorted one-dimensional vector. "Sorting the one-dimensional vector" is the sorting performed when generating a sorted one-dimensional vector from the one-dimensional vector.

[0059] Figure 4 shows how multiple elements of the updated one-dimensional vector are returned to their original order before sorting to generate a reverse-sorted one-dimensional vector. As shown in Figure 4, "returning to the original order before sorting the one-dimensional vector" involves N elements z １ ~z Ｎ This means rearranging them in the order of serial number p.

[0060] "Process to generate a transformed multidimensional tensor using a reverse-sorted one-dimensional vector" Next, the parameter assignment unit 112 generates a transformed multidimensional tensor based on the values ​​of each of the multiple elements of the reverse-sorted one-dimensional vector.

[0061] Figure 4 shows how a transformed multidimensional tensor is generated based on the values ​​of each of the multiple elements of the inversely sorted one-dimensional vector. As shown in Figure 4, the parameter assignment unit 112 generates a transformed multidimensional tensor by arranging the values ​​of the multiple elements of the inversely sorted one-dimensional vector in two dimensions according to predetermined rules. In the transformed multidimensional tensor, the parameter assignment unit 112 assigns the element z ｎ The element x in the multidimensional tensor to be processed ｎ Place them in the same position.

[0062] The "transformed multidimensional tensor" may be a 3D tensor, a 2D tensor, or a tensor of any other dimension. A 3D tensor represents, for example, the data set for each pixel of a color image (e.g., an RGB image). In this case, the three dimensions represent width, height, and RGB channels. A 2D tensor represents, for example, the data set for each pixel of a grayscale image (black and white image). In this case, the two dimensions represent width and height.

[0063] The transformed multidimensional tensor is a multidimensional tensor generated by performing a predetermined differentiable operation on the target multidimensional tensor, as shown in Figure 4. The histograms of the target multidimensional tensor and the transformed multidimensional tensor have different shapes. By performing a predetermined differentiable operation on the target multidimensional tensor, the shape of the histogram of the target multidimensional tensor is transformed to the shape of the histogram of the transformed multidimensional tensor.

[0064] The "histogram of the multidimensional tensor to be processed" is a histogram of the values ​​of multiple elements (e.g., pixel values) of the multidimensional tensor to be processed. The "histogram of the transformed multidimensional tensor" is a histogram of the values ​​of multiple elements (e.g., pixel values) of the transformed multidimensional tensor.

[0065] During the learning phase, the transformed multidimensional tensor becomes an intermediate product used to learn the learnable parameters. That is, the learnable parameters are learned by processing this transformed multidimensional tensor. The details of the learning process will be explained in the ninth embodiment.

[0066] In the estimation stage, the transformed multidimensional tensor is the target object obtained by applying appropriate corrections to the multidimensional image to be processed. That is, this transformed multidimensional tensor is presented to the user, stored in a predetermined memory device, or used for subsequent processing such as image recognition.

[0067] Next, an example of the processing flow performed by the information processing device 10 will be explained using the flowchart in Figure 5. Note that the purpose here is to explain the processing flow. Details of each process have been described above, so explanations will be omitted here as appropriate.

[0068] "Learning Phase" First, the information processing device 10 initializes the learnable parameters (S20). In the learning phase, the information processing device 10 sets each element of the learnable parameters to a predetermined value in this initialization process. For example, the information processing device 10 may set random values ​​generated using a random number generator to the values ​​of each element of the learnable parameters. Alternatively, the information processing device 10 may set values ​​prepared in advance by the user to the values ​​of each element of the learnable parameters. Alternatively, the information processing device 10 may set pre-learned values ​​to the values ​​of each element of the learnable parameters.

[0069] Next, the information processing device 10 acquires the multidimensional tensor to be processed (S21). During the learning phase, the information processing device 10 extracts the multidimensional tensor to be processed from a pre-prepared training dataset. The training dataset consists of multiple pairs of multidimensional tensors and correct data. Since the training dataset requires correct data, it is generally acquired by reading data stored in storage.

[0070] Next, the information processing device 10 performs a differentiable preprocessing on the multidimensional tensor to be processed (S22). The differentiable preprocessing is the "process of generating a transformed multidimensional tensor from the multidimensional tensor to be processed" performed by the sorting unit 111 and the parameter assignment unit 112 described above.

[0071] During the learning phase, after S22, the transformed multidimensional tensor generated in the preprocessing is used to calculate the loss value and update the learnable parameters. Details of these processes will be explained in the ninth embodiment.

[0072] "Estimation Stage" First, the information processing device 10 initializes the learnable parameters (S20). In the estimation stage, the information processing device 10 can read the values ​​optimized by pre-training and set them as the values ​​of each element of the learnable parameters during this initialization process.

[0073] Next, the information processing device 10 acquires the multidimensional tensor to be processed (S21). In the estimation stage, the information processing device 10 acquires the multidimensional tensor that is subject to correction as the multidimensional tensor to be processed. For example, the information processing device 10 may read image data (multidimensional tensor to be processed) stored in data storage. Alternatively, the information processing device 10 may acquire image data (multidimensional tensor to be processed) captured by an image sensor such as a camera. Alternatively, the information processing device 10 may acquire image data (multidimensional tensor to be processed) input by the user.

[0074] Next, the information processing device 10 performs a differentiable preprocessing on the multidimensional tensor to be processed (S22).

[0075] In the estimation stage, after S22, the transformed multidimensional tensor generated in the preprocessing may be presented to the user. For example, the information processing device 10 may generate an image based on the transformed multidimensional tensor and display the image on a display or transmit it to an external device.

[0076] In addition, during the estimation stage, after S22, the information processing device 10 may store the converted multidimensional tensor generated in the preprocessing in a predetermined storage device.

[0077] In addition, during the estimation stage, after S22, the information processing device 10 may perform subsequent processing such as image recognition using the transformed multidimensional tensor generated in the preprocessing, and present the results of the subsequent processing to the user. For example, the information processing device 10 may display the results of the subsequent processing on a display or transmit them to an external device. Examples of subsequent processing will be described in the seventh and eighth embodiments.

[0078] Here, a modified version of this embodiment will be described. When the multidimensional tensor to be processed is a three-dimensional tensor (color image), the information processing device 10 can perform the differentiable preprocessing described above for each color. When the multidimensional tensor to be processed is a three-dimensional tensor, the multidimensional tensor to be processed may have multiple elements arranged in two directions, the width direction and the height direction, for each color, as shown in Figure 4. The information processing device 10 can generate a converted multidimensional tensor from the multidimensional tensor to be processed by performing the processing shown in Figure 4 on the multiple elements of each color.

[0079] In this modified example, the information processing device 10 can prepare learnable parameters for each color. It can then learn these learnable parameters for each color. The values ​​of each element in these learnable parameters may differ from one another.

[0080] <Effects and Effects> The information processing device 10 of the second embodiment can achieve the same effects and effects as the information processing device 10 of the first embodiment.

[0081] Furthermore, the information processing device 10 transforms the histogram of the values ​​(pixel values, etc.) of each element of the multidimensional tensor being processed into another state by performing the differentiable processing described above. Because the histogram is transformed using differentiable processing, backpropagation becomes possible, and as a result, it becomes possible to optimize the parameter values ​​using machine learning.

[0082] <<Third Embodiment>> In the second embodiment, the learnable parameters were composed of the same number of elements as the sorted one-dimensional vector. In the third embodiment, as shown in Figure 6, the learnable parameters are composed of fewer elements than the sorted one-dimensional vector. The information processing device 10 then upsamples these learnable parameters according to a predetermined rule to generate parameters composed of the same number of elements as the sorted one-dimensional vector, and updates the sorted one-dimensional vector using the generated parameters. This will be explained in detail below.

[0083] Figure 6 shows an overview of the processing performed by the information processing device 10 of the third embodiment. As described in the second embodiment, the sorted one-dimensional vector is constructed by arranging N elements. The definition of the sorted one-dimensional vector is as described in the second embodiment.

[0084] As shown in Figure 6, the learnable parameter is composed of M elements arranged in a sequence, where M is a value less than N. In other words, the learnable parameter is composed of fewer than N elements. Thus, the learnable parameter in the third embodiment differs from the learnable parameter in the second embodiment in the number of elements. The other configurations of the learnable parameter in the third embodiment can be the same as those of the learnable parameter in the second embodiment.

[0085] The parameter assignment unit 112 converts a learnable parameter consisting of M elements into data consisting of N elements. Hereinafter, this converted data consisting of N elements will be referred to as the upsampled parameter.

[0086] The upsampling method is not restricted, and widely known techniques can be employed. For example, the elements y of the learnable parameters ｑ and element y ｑ＋１ Upsampling can be performed by adding a predetermined number of elements in between. The values ​​of the elements to be added can be determined using widely known techniques. Furthermore, if the learnable parameters are arranged in a predetermined order (such as from largest to smallest or from largest to smallest value of each element), it is preferable that the upsampled parameters also be arranged in that predetermined order. In this case, the elements y of the learnable parameters ｑ and element y ｑ＋１ The value of the element added between them is "y ｑ That's all, Y ｑ＋１ The value can be within the following numerical range.

[0087] After generating the upsampled parameters, the parameter assignment unit 112 can generate an updated one-dimensional vector by assigning the upsampled parameters, which consist of N elements, to a sorted one-dimensional vector consisting of N elements. In the process of generating an updated one-dimensional vector using the "learnable parameters" described in the second embodiment, the parameter assignment unit 112 replaces the "learnable parameters" with the "upsampled parameters". Except for using the "upsampled parameters" instead of the "learnable parameters", the process can be the same as the process of generating an updated one-dimensional vector using the "learnable parameters" described in the second embodiment.

[0088] Other configurations of the information processing device 10 can be the same as those in the first and second embodiments.

[0089] The information processing device 10 of the third embodiment can achieve the same effects and advantages as the information processing device 10 of the first and second embodiments. Furthermore, the information processing device 10 of the third embodiment can reduce the number of learnable parameter elements (data volume). As a result, the processing load on the computer for learning learnable parameters and other processing using learnable parameters can be reduced. In addition, the time required for learning learnable parameters can be shortened.

[0090] <<Fourth Embodiment>> In the fourth embodiment, the information processing device 10 performs a color conversion process on the multidimensional tensor to be processed, and then performs the sorting unit 111 and parameter assignment unit 112 described in the first to third embodiments. A detailed explanation follows below.

[0091] Figure 7 is an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 has a preprocessing unit 11. The preprocessing unit 11 has a sorting unit 111, a parameter assignment unit 112, and a color conversion unit 113.

[0092] In the fourth embodiment, the multidimensional tensor to be processed represents the data of each pixel in a color image (e.g., an RGB image). In this case, the three dimensions represent width, height, and RGB channels.

[0093] The color conversion unit 113 performs a process to convert the color space of a color image to a multidimensional tensor to be processed. There are various methods for converting the color space. For example, the color conversion unit 113 may convert the order of colors in a color image from RGB to GBR, BGR, GRB, RBG, BRG, etc. Alternatively, the color conversion unit 113 may convert a color image to a gray image (grayscale image). Alternatively, the color conversion unit 113 may convert an RGB image to an image in another color space such as HSV, Lab, LUV, XYZ, YCrCb, etc. The color space conversion may be represented by a matrix or by other methods.

[0094] The preprocessing unit 11 performs the processing described in the first to third embodiments on the color image (multidimensional tensor to be processed) after the color space conversion. That is, the preprocessing unit 11 performs the processing shown in Figures 4 and 6 on the color image (multidimensional tensor to be processed) after the color space conversion to generate a converted multidimensional tensor.

[0095] More specifically, the sorting unit 111 sorts the color image (multidimensional tensor to be processed) after the color space transformation and generates a one-dimensional vector and a sorted one-dimensional vector. Next, the parameter assignment unit 112 replaces the sorted one-dimensional vector with learnable parameters to generate an updated one-dimensional vector, and then rearranges the updated one-dimensional vector to generate a reverse-sorted one-dimensional vector. Then, the parameter assignment unit 112 uses the reverse-sorted one-dimensional vector to generate a color image (transformed multidimensional tensor) with a histogram of pixel values.

[0096] The parameter assignment unit 112 may output the "color image (converted multidimensional tensor) after the conversion of the pixel value histogram" generated in this manner.

[0097] As a variation, the parameter assignment unit 112 may, after generating a color image (transformed multidimensional tensor) from the histogram of pixel values, perform a process to restore the color space of the color image to its original state. The parameter assignment unit 112 may then output a color image (transformed multidimensional tensor) after performing the process to restore the color space of the color image to its original state (inverse transformation).

[0098] In the process of converting the color space of a color image back to its original state (inverse transformation), the parameter assignment unit 112 may convert the order of colors in the color image from, for example, GBR, BGR, GRB, RBG, BRG, etc., to RGB. In addition, the parameter assignment unit 112 may convert a gray image (grayscale image) to a color image. In addition, the parameter assignment unit 112 may convert an image in a color space such as HSV, Lab, LUV, XYZ, YCrCb, etc., to an RGB image.

[0099] As described in the first to third embodiments, the information processing device 10 can perform various subsequent processes using the color image (converted multidimensional tensor) output from the parameter assignment unit 112. In the learning phase, the information processing device 10 can perform subsequent processes to update the learnable parameters using the color image. In the estimation phase, the information processing device 10 can present the color image to the user, save it to a predetermined storage device, or perform subsequent processes such as image recognition using the color image.

[0100] Other configurations of the information processing device 10 can be the same as those of the first to third embodiments.

[0101] The information processing device 10 of the fourth embodiment can achieve the same effects as the information processing device 10 of the first to third embodiments. Furthermore, since the information processing device 10 of the fourth embodiment performs image enhancement (image correction) in a different color space, it is expected that the image enhancement performance will be improved compared to processing RGB images as they are.

[0102] <<Fifth Embodiment>> In the fifth embodiment, the information processing device 10 performs downsampling on the multidimensional tensor to be processed, and then performs the sorting unit 111 and parameter assignment unit 112 described in the first to third embodiments. A detailed explanation follows below.

[0103] Figure 8 is an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 has a preprocessing unit 11. The preprocessing unit 11 has a sorting unit 111, a parameter assignment unit 112, and a reduction unit 114.

[0104] The multidimensional tensor to be processed in the fourth embodiment represents the data of each pixel in an image (color image or grayscale image).

[0105] The reduction unit 114 performs downsampling to reduce the number of pixels in the multidimensional tensor to be processed. The reduction unit 114 can perform this downsampling using widely known techniques.

[0106] The preprocessing unit 11 performs the processing described in the first to third embodiments on the downsampled image (multidimensional tensor to be processed). That is, the preprocessing unit 11 performs the processing shown in Figures 4 and 6 on the downsampled image (multidimensional tensor to be processed) to generate a transformed multidimensional tensor.

[0107] More specifically, the sorting unit 111 sorts the image (multidimensional tensor to be processed) after downsampling and generates a one-dimensional vector and a sorted one-dimensional vector. Next, the parameter assignment unit 112 replaces the sorted one-dimensional vector with learnable parameters to generate an updated one-dimensional vector, and then rearranges the updated one-dimensional vector to generate a reverse-sorted one-dimensional vector. Then, the parameter assignment unit 112 uses this reverse-sorted one-dimensional vector to generate a transformed image (transformed multidimensional tensor) from the histogram of pixel values.

[0108] The parameter assignment unit 112 may output the "image after transformation (transformed multidimensional tensor)" of the pixel value histogram generated in this manner.

[0109] As a variation, the parameter assignment unit 112 may, after generating a converted image (converted multidimensional tensor) from the histogram of pixel values, perform upsampling on the image to increase the number of pixels (for example, upsampling back to the number of pixels before downsampling). The parameter assignment unit 112 may then output the image (converted multidimensional tensor) after performing the upsampling.

[0110] As described in the first to third embodiments, the information processing device 10 can perform various subsequent processes using the image (transformed multidimensional tensor) output from the parameter assignment unit 112. In the learning phase, the information processing device 10 can perform subsequent processes to update the learnable parameters using the image. In the estimation phase, the information processing device 10 can present the image to the user, save it to a predetermined storage device, or perform subsequent processes such as image recognition using the image.

[0111] Other configurations of the information processing device 10 can be the same as those of the first to third embodiments.

[0112] Here, a modified version of the information processing device 10 of the fifth embodiment will be described. The information processing device 10 may incorporate the configuration of the fourth embodiment. That is, the preprocessing unit 11 may include a sorting unit 111, a parameter assignment unit 112, a color conversion unit 113, and a reduction unit 114.

[0113] Furthermore, the preprocessing unit 11 may perform the processing described in the first to third embodiments on the color image (multidimensional tensor to be processed) that has undergone downsampling and color space conversion. That is, the preprocessing unit 11 may perform the processing shown in Figures 4 and 6 on the color image (multidimensional tensor to be processed) that has undergone downsampling and color space conversion to generate a converted multidimensional tensor. The preprocessing unit 11 may perform color space conversion of the color image after downsampling. Alternatively, the preprocessing unit 11 may perform downsampling after converting the color space of the color image.

[0114] The information processing device 10 of the fifth embodiment can achieve the same effects as the information processing device 10 of the first to fourth embodiments. Furthermore, the information processing device 10 of the fifth embodiment can reduce the amount of data processed by the sorting unit 111 and the parameter assignment unit 112. Specifically, it can reduce the number of elements (data volume) of the one-dimensional vector, the sorted one-dimensional vector, the updated one-dimensional vector, the inversely sorted one-dimensional vector, and the learnable parameters. As a result, the processing burden on the computer in learning learnable parameters and other processing using learnable parameters can be reduced. In addition, the time required to learn learnable parameters can be shortened.

[0115] <<Sixth Embodiment>> In the sixth embodiment, the information processing device 10 generates a sorted one-dimensional reference vector from a reference multidimensional tensor using the same process as the process for generating a sorted one-dimensional vector from a multidimensional tensor to be processed. The information processing device 10 then uses this sorted one-dimensional reference vector as a learnable parameter to perform subsequent processing. This will be explained in detail below.

[0116] Figure 9 is an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 has a preprocessing unit 11 and a reference histogram extraction unit 12. The preprocessing unit 11 has a sorting unit 111 and a parameter assignment unit 112. The preprocessing unit 11 may further include at least one of a color conversion unit 113 and a reduction unit 114.

[0117] The reference histogram extraction unit 12 converts a multidimensional tensor for reference into a one-dimensional vector for reference, sorts the one-dimensional vector for reference, and generates a sorted one-dimensional vector for reference.

[0118] The "reference multidimensional tensor" may be a three-dimensional tensor, a two-dimensional tensor, or a tensor of any other dimension. A three-dimensional tensor represents, for example, the data set for each pixel of a color image (e.g., an RGB image). In this case, the three dimensions represent width, height, and RGB channels. A two-dimensional tensor represents, for example, the data set for each pixel of a grayscale image (black and white image). In this case, the two dimensions represent width and height.

[0119] The user can select a reference multidimensional tensor from a variety of multidimensional tensors and input it into the information processing device 10. For example, the user can select an image with high contrast and easy recognition (a color image or a grayscale image) as the reference multidimensional tensor.

[0120] The reference histogram extraction unit 12 generates a sorted one-dimensional reference vector from a reference multidimensional tensor using the same process as the process described above for generating a sorted one-dimensional vector from the multidimensional tensor to be processed.

[0121] The parameter assignment unit 112 uses the sorted one-dimensional reference vector generated by the reference histogram extraction unit 12 as learnable parameters to generate a transformed multidimensional tensor.

[0122] The parameter assignment unit 112 may compare the number of elements R in the sorted reference one-dimensional vector with the number of elements N in the sorted one-dimensional vector. If R is less than N, the parameter assignment unit 112 may upsample the sorted reference one-dimensional vector so that R = N. If R is greater than N, the parameter assignment unit 112 may downsample the sorted reference one-dimensional vector so that R = N. If R = N, the parameter assignment unit 112 does not perform these operations. Downsampling and upsampling here can be achieved using widely known techniques.

[0123] During the estimation phase, the multidimensional tensor to be processed is corrected using a sorted one-dimensional reference vector as a learnable parameter, thereby generating the transformed multidimensional tensor.

[0124] Other configurations of the information processing device 10 can be the same as those of the first to fifth embodiments.

[0125] The information processing device 10 of the sixth embodiment can achieve the same effects as the information processing device 10 of the first to fifth embodiments. Furthermore, the information processing device 10 of the sixth embodiment can use a sorted one-dimensional reference vector obtained from a multidimensional reference tensor as a learnable parameter.

[0126] In other words, during the estimation phase, the multidimensional tensor to be processed can be corrected by using a sorted one-dimensional reference vector obtained from a reference multidimensional tensor as a learnable parameter. For example, by using an appropriate reference multidimensional tensor such as a color image with high contrast that is easy to recognize, the multidimensional tensor to be processed can be enhanced (image corrected) to make it easier to recognize.

[0127] <<Seventh Embodiment>> In the seventh embodiment, an example of the estimation process performed by the information processing device 10 will be described. This will be explained in detail below.

[0128] Figure 10 is an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 has a pre-processing unit 11 and a recognition unit 14. The pre-processing unit 11 has a sorting unit 111 and a parameter assignment unit 112. The pre-processing unit 11 may further include at least one of a color conversion unit 113 and a reduction unit 114. The information processing device 10 may also include a reference histogram extraction unit 12.

[0129] The recognition unit 14 receives the transformed multidimensional tensor generated by the parameter assignment unit 112 as input and performs a process to recognize the transformed multidimensional tensor. The recognition unit 14 can perform a predetermined image recognition process on the image represented by the transformed multidimensional tensor and output the recognition result. The content of the image recognition process performed by the recognition unit 14 is not limited, and widely known image recognition processes can be employed. The recognition unit 14 is composed of, for example, a neural network for classification or object detection.

[0130] Next, an example of the processing flow performed by the information processing device 10 will be explained using the flowchart in Figure 11. The purpose here is simply to explain the processing flow. Details of each process have been described above, so further explanation will be omitted here as appropriate.

[0131] First, the information processing device 10 initializes the learnable parameters (S30). In this initialization process, the information processing device 10 can read values ​​optimized by pre-training and set them as the values ​​of each element of the learnable parameters. Alternatively, in this initialization process, the information processing device 10 can set a sorted one-dimensional reference vector obtained from a multi-dimensional reference tensor as the value of each element of the learnable parameters.

[0132] Next, the information processing device 10 acquires the multidimensional tensor to be processed (S31). The information processing device 10 acquires the multidimensional tensor that is subject to correction as the multidimensional tensor to be processed. For example, the information processing device 10 may read image data (multidimensional tensor to be processed) stored in data storage. Alternatively, the information processing device 10 may acquire image data (multidimensional tensor to be processed) captured by an image sensor such as a camera. Alternatively, the information processing device 10 may acquire image data (multidimensional tensor to be processed) input by a user.

[0133] Next, the information processing device 10 performs a differentiable preprocessing on the multidimensional tensor to be processed (S32). The details of the preprocessing are as described in the first to sixth embodiments.

[0134] Next, the information processing device 10 recognizes the transformed multidimensional tensor generated in the preprocessing in S32 and performs a process to estimate recognition information, and outputs the result (recognition information) (S33). For example, the information processing device 10 performs a class classification or object detection process and outputs the result.

[0135] Other configurations of the information processing device 10 can be the same as those of the first to sixth embodiments.

[0136] The information processing device 10 of the seventh embodiment can achieve the same effects as the information processing device 10 of the first to sixth embodiments. Furthermore, the information processing device 10 of the seventh embodiment can perform predetermined recognition processing after correcting a multidimensional tensor using learnable parameters to generate a transformed multidimensional tensor and then processing the transformed multidimensional tensor. By processing the transformed multidimensional tensor obtained by appropriately correcting the multidimensional tensor using learnable parameters and performing predetermined recognition processing, the accuracy of the recognition processing is improved.

[0137] <<Eighth Embodiment>> In the eighth embodiment, another example of the estimation process performed by the information processing device 10 will be described. In the eighth embodiment, a plurality of parameter assignment units 112 are prepared in advance. The information processing device 10 then selects one of the plurality of parameter assignment units 112 according to the multidimensional tensor to be processed, and corrects the multidimensional tensor to be processed using the selected parameter assignment unit 112. This will be explained in detail below.

[0138] Figure 12 is an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 has a pre-processing unit 11 and a selection unit 13. The pre-processing unit 11 has a sorting unit 111 and a plurality of parameter assignment units 112. The pre-processing unit 11 may further include at least one of a color conversion unit 113 and a reduction unit 114. The information processing device 10 may also include at least one of a reference histogram extraction unit 12 and a recognition unit 14.

[0139] Multiple parameter assignment units 112 generate a transformed multidimensional tensor from a multidimensional tensor to be processed using learnable parameters with different values ​​set for each unit. The multiple parameter assignment units 112 are configured, for example, as multiple modules.

[0140] The images represented by the multidimensional tensor to be processed can include a variety of images, such as dark images, images of heavy snow, and images of sandstorms. The values ​​of the learnable parameters suitable for image correction for each of these various images may differ from one another. Therefore, in the eighth embodiment, machine learning can be performed in advance using various types of training datasets, and the optimal values ​​of the learnable parameters can be determined for each dataset.

[0141] For example, one training dataset may consist of multiple pairs of "dark images (multidimensional tensors) and ground truth data (e.g., classes in classification)." Another training dataset may consist of multiple pairs of "heavy snow images (multidimensional tensors) and ground truth data (e.g., classes in classification)."

[0142] The selection unit 13 selects a parameter assignment unit 112 from among several parameter assignment units 112 that will process the target multidimensional tensor, based on the data of the target multidimensional tensor. The selected parameter assignment unit 112 processes the target multidimensional tensor and generates a transformed multidimensional tensor.

[0143] Here, we will explain a specific example of the process by which the selection unit 13 selects the parameter assignment unit 112.

[0144] In one example, the selection unit 13 may be composed of a simple classifier that uses statistical values ​​such as the mean and standard deviation of the pixel values ​​of the multidimensional tensor to be processed and a predetermined threshold. For example, a predetermined numerical range of the above statistical values ​​is pre-associated with each of the multiple parameter assignment units 112. After calculating the statistical values ​​such as the mean and standard deviation of the pixel values ​​of the multidimensional tensor to be processed, the selection unit 13 identifies which numerical range of the parameter assignment unit 112 the calculated statistical values ​​fall into. Then, the selection unit 13 selects the parameter assignment unit 112 corresponding to the numerical range in which the calculated statistical values ​​are included as the parameter assignment unit 112 to process the multidimensional tensor to be processed.

[0145] Here, a modified example of the eighth embodiment will be described. The selection unit 13 may be composed of a classifier trained from several images using unsupervised learning such as k-means or GMM (Gaussian Mixture Model).

[0146] Alternatively, the selection unit 13 may be a classifier composed of a neural network including a CNN (Convolutional Neural Network) or a Transformer, which selects the optimal parameter assignment unit 112 from the input image (multidimensional tensor to be processed). In this case as well, the selection unit 13 may be composed of a small-scale neural network and may be trained using training data consisting of a small number of images (multidimensional tensors) and the correct answer.

[0147] Alternatively, the selection unit 13 may be configured using a pre-trained network that has been trained on images and language, such as CLIP (Contrastive Language-Image Pre-training). In this case, the selection unit 13 may perform classification using multiple features extracted from language information (such as text related to images included in the training dataset) as the ground truth vector. The ground truth vector may be added along the way.

[0148] Each of the multiple parameter assignment units 112 has its own neural network, and each updates the learnable parameters by machine learning to optimize for each class set in the selection unit 13. For example, the learnable parameters processed by each of the multiple parameter assignment units 112 are optimized by machine learning based on images and ground truth data belonging to each class set in the selection unit 13. When using CLIP or similar, classes can be added midway through the process. When a new class is added, a new parameter assignment unit 112 that has been machine learning optimized for the newly added class is added.

[0149] Other configurations of the information processing device 10 can be the same as those of the first to seventh embodiments.

[0150] The information processing device 10 of the eighth embodiment can achieve the same effects as the information processing device 10 of the first to seventh embodiments. Furthermore, the information processing device 10 of the eighth embodiment can utilize the optimal parameter assignment unit 112 according to various input images (multidimensional tensors to be processed). As a result, the input image (multidimensional tensor to be processed) can be corrected (enhanced) more appropriately.

[0151] <<Ninth Embodiment>> In the ninth embodiment, an example of the learning stage processing performed by the information processing device 10 will be described. This will be explained in detail below.

[0152] Figure 13 is an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 includes a preprocessing unit 11, a recognition unit 14, and a parameter update unit 15. The preprocessing unit 11 includes a sorting unit 111 and a plurality of parameter assignment units 112. The parameter update unit 15 includes a loss calculation unit 151, a gradient calculation unit 152, and an update unit 153. The preprocessing unit 11 may further include at least one of a color conversion unit 113 and a reduction unit 114. The information processing device 10 may also include a reference histogram extraction unit 12.

[0153] The parameter update unit 15 updates the learnable parameters based on a loss function that calculates the difference between the output of the recognition unit 14 and the correct data. This process is carried out by the loss calculation unit 151, the gradient calculation unit 152, and the update unit 153.

[0154] The loss calculation unit 151 calculates the loss value based on a loss function that calculates the difference (loss value) between the output of the recognition unit 14 and the correct data.

[0155] The "output of the recognition unit 14" is the result of the process of recognizing the transformed multidimensional tensor. In the process of recognizing the transformed multidimensional tensor, image recognition processes such as class classification and object detection are performed. The output of the recognition unit 14 is the result of these image recognition processes.

[0156] The "ground truth data" is included in the training dataset beforehand. The training dataset consists of multiple "pairs of multidimensional tensors and ground truth data." The ground truth data indicates the correct image recognition for the paired multidimensional tensors.

[0157] The loss function calculates the difference (loss value) between the output of the recognition unit 14 and the correct data. The loss function is used in an optimization process called backpropagation. The loss calculation unit 151 can use widely known loss functions. For example, loss functions include, but are not limited to, the mean squared error, mean absolute error, the square root of the mean squared error, the mean squared logarithmic error, Huber loss, Poisson loss, hinge loss, and Kullback-Leibler divergence.

[0158] The gradient calculation unit 152 calculates the gradient of each parameter of the entire network, consisting of the preprocessing unit 11 and the recognition unit 14, by backpropagating the loss values ​​calculated by the loss calculation unit 151. The gradient calculation is achieved using widely known techniques. The gradient of each parameter is obtained by differentiating the objective function with respect to each parameter.

[0159] The update unit 153 updates the network parameters composed of the preprocessing unit 11 and the recognition unit 14 based on the gradient calculated by the gradient calculation unit 152. In other words, the update unit 153 updates the learnable parameters based on the gradient calculated by the gradient calculation unit 152.

[0160] For example, the update unit 153 can update the "parameter value" to "the value obtained by subtracting the "product of the learning rate and gradient" from the "original value of the parameter". In this case, the learning rate of the parameter assignment unit 112 can be set higher than the learning rate of the recognition unit 14. That is, the learning rate used in the process of updating the parameters of the parameter assignment unit 112 can be set higher than the learning rate used in the process of updating the parameters of the recognition unit 14. The learning rate used in the process of updating the parameters of the recognition unit 14 may be 0.

[0161] Next, an example of the processing flow performed by the information processing device 10 will be explained using the flowchart in Figure 14. The purpose here is simply to explain the processing flow. Details of each process have been described above, so further explanation will be omitted here as appropriate.

[0162] First, the information processing device 10 initializes the learning parameters (S30). The learning parameters include the parameters of the parameter assignment unit 112 and the recognition unit 14. The learning parameters include the learnable parameters described above. In this initialization process, the information processing device 10 sets initial values ​​for the learning parameters. For example, the information processing device 10 may set random values ​​generated using a random number generator as the values ​​of the learning parameters. Alternatively, the information processing device 10 may set values ​​prepared in advance by the user as the values ​​of the learning parameters. Alternatively, the information processing device 10 may set pre-learned values ​​as the values ​​of the learning parameters.

[0163] Next, the information processing device 10 acquires a training dataset (S31). The training dataset consists of multiple pairs of multidimensional tensors and ground truth data. Since the training dataset requires ground truth data, it is generally acquired by reading data stored in storage.

[0164] Next, the information processing device 10 acquires a training data batch (S32). For example, the information processing device 10 acquires a training data batch by randomly selecting a predetermined number (e.g., 32 pairs) from the training dataset.

[0165] Next, the information processing device 10 performs preprocessing on the multidimensional tensor to be processed (S33). That is, the information processing device 10 preprocesses the multidimensional tensor included in the training data batch as the multidimensional tensor to be processed. Preprocessing is the process performed by the preprocessing unit 11 described in the first to sixth embodiments. By performing preprocessing, a converted multidimensional tensor is generated from the multidimensional tensor to be processed.

[0166] Next, the information processing device 10 estimates recognition information by performing recognition processing on the transformed multidimensional tensor (S34). For example, the information processing device 10 performs image recognition processing such as class classification or object detection on the transformed multidimensional tensor and outputs the result (recognition information).

[0167] Next, the information processing device 10 calculates the loss value (S35) based on a loss function that calculates the difference (loss value) between the recognition information estimated in S34 and the correct data included in the training data batch.

[0168] Next, the information processing device 10 calculates the gradient of each learning parameter of the entire network, which is composed of the parameter assignment unit 112 and the recognition unit 14, by backpropagation of the loss value calculated in S35 (S36).

[0169] Next, the information processing device 10 updates the values ​​of each learning parameter based on the gradient of each learning parameter calculated in S36 and the predetermined learning rate of each learning parameter (S37).

[0170] The information processing device 10 repeatedly performs the learning loop described in S32 to S37. There are various conditions for terminating the learning loop. For example, the information processing device 10 may terminate the learning loop when the number of loops reaches a predetermined number of iterations.

[0171] The information processing device 10 saves the values ​​of the learning parameters at the time the learning loop is completed to a predetermined storage device (S38). The predetermined storage device may be provided within the information processing device 10, or it may be provided in an external device that is communicatively connected to the information processing device 10.

[0172] Other configurations of the information processing device 10 can be the same as those of the first to eighth embodiments.

[0173] The information processing device 10 of the ninth embodiment can achieve the same effects as the information processing device 10 of the first to eighth embodiments. Furthermore, the information processing device 10 of the ninth embodiment can update parameters that can be learned by machine learning. With such an information processing device 10, the values ​​of the learnable parameters can be optimized.

[0174] Furthermore, after performing recognition processing on the transformed multidimensional tensor, the information processing device 10 updates the learnable parameters based on the difference (loss value) between the recognition result and the correct data. For example, the information processing device 10 updates the learnable parameters to improve the visibility of an image, but it is difficult to quantify the visibility perceived by the human eye. Therefore, the information processing device 10 performs recognition processing on the transformed multidimensional tensor using a computer, and then updates the learnable parameters based on the results of that recognition processing. With such an information processing device 10, it is possible to update the learnable parameters after appropriately quantifying visibility. As a result, it is expected that the learnable parameters will be optimized with high accuracy.

[0175] <<Modifications>> Here, modifications applicable to the first to ninth embodiments are described. In the first to ninth embodiments, an example was described in which the multidimensional tensor to be processed represents the data of each pixel of an image. In the modifications, the multidimensional tensor to be processed can represent other data. For example, the multidimensional tensor to be processed can represent various features.

[0176] In this case, the sorting unit 111 sorts the elements of the multidimensional tensor to be processed, which represent features, based on the values ​​of the features. The parameter assignment unit 112 assigns learnable parameters to the sorted elements output from the sorting unit 111. The learnable parameters consist of parameters with the same shape as the features represented by the elements of the multidimensional tensor to be processed. The parameter assignment unit 112 updates the value of each of the sorted elements output from the sorting unit 111 with the pre-prepared values ​​of the learnable parameters to create output features, which are then output. The information processing device 10 may also reverse sort the output features and output the result.

[0177] In the modified example, the content of the multidimensional tensor to be processed differs from that of the embodiment described above, but the processing performed on the multidimensional tensor is the same as in the embodiment described above.

[0178] This modification allows features to be transformed into parameters in a differentiable form. Therefore, the training of the entire neural network, including this normalization, becomes more efficient.

[0179] Although this disclosure has been described above with reference to embodiments, this disclosure is not limited to the embodiments described above. Various modifications to the structure and details of this disclosure are possible, which can be understood by those skilled in the art within the scope of this disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0180] Furthermore, the flowchart used in the above explanation shows multiple steps (processes) in sequence. However, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impede the content.

[0181] Some or all of the above embodiments may also be described as follows, but are not limited to the following: 1. An information processing apparatus having a preprocessing means comprising: sorting means for sorting a multidimensional tensor; parameter assignment means for assigning learnable parameters to the sorted multidimensional tensor and generating and outputting a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor. 2. The information processing apparatus according to 1, wherein the sorting means converts the multidimensional tensor into a one-dimensional vector, sorts the one-dimensional vector to generate a sorted one-dimensional vector, and the parameter assignment means assigns the learnable parameters to the sorted one-dimensional vector. 3. The information processing apparatus according to 2, wherein the sorted one-dimensional vector is composed of N elements, the learnable parameters are composed of fewer than N elements, the parameter assignment means converts the learnable parameters into data with N elements, and assigns the learnable parameters, after being converted into data with N elements, to the sorted one-dimensional vector. 4. The information processing device according to 2, wherein the sorted one-dimensional vector and the learnable parameters are configured such that they each have the same number of elements. 5. The information processing device according to any one of 2 to 4, wherein the sorting means converts the multidimensional tensor into a one-dimensional vector with multiple elements arranged in a row, and sorts the multiple elements of the one-dimensional vector based on the magnitude of the values ​​of each element to generate the sorted one-dimensional vector. 6. The information processing device according to any one of 2 to 5, wherein the parameter assignment means updates the value of the nth element in the sorted one-dimensional vector using the value of the nth element in the learnable parameters to generate an updated one-dimensional vector, returns the multiple elements of the updated one-dimensional vector to their original order before sorting the one-dimensional vector to generate a reverse-sorted one-dimensional vector, and generates the converted multidimensional tensor based on the values ​​of each of the multiple elements of the reverse-sorted one-dimensional vector.7. The parameter assignment means is an information processing device according to 6, wherein the parameter assignment means replaces the value of the nth element in the sorted one-dimensional vector with the value of the nth element in the learnable parameter to generate the updated one-dimensional vector. 8. The parameter assignment means is an information processing device according to 6, wherein the parameter assignment means replaces the value of the nth element in the sorted one-dimensional vector with the product of the value of the nth element in the sorted one-dimensional vector and the value of the nth element in the learnable parameter to generate the updated one-dimensional vector. 9. The multidimensional tensor represents the pixel data of a color image, the preprocessing means further comprises a color conversion means that performs a process to convert the color space of the color image to the multidimensional tensor, and the sorting means sorts the multidimensional tensor after the color space conversion process. 10. The parameter assignment means is an information processing device according to 9, wherein after generating the converted multidimensional tensor, the converted multidimensional tensor performs a process to restore the color space of the color image to its original state. 11. An information processing apparatus according to any one of 1 to 10, wherein the multidimensional tensor represents the pixel data of an image, the preprocessing means further comprises a reduction means that performs downsampling to reduce the number of pixels on the multidimensional tensor, the sorting means sorts the multidimensional tensor after downsampling, and the parameter assignment means generates the transformed multidimensional tensor and then performs upsampling to increase the number of pixels on the transformed multidimensional tensor. 12. An information processing apparatus according to any one of 1 to 11, further comprising a reference histogram extraction means that converts a reference multidimensional tensor into a reference one-dimensional vector, sorts the reference one-dimensional vector to generate a sorted reference one-dimensional vector, and the parameter assignment means uses the sorted reference one-dimensional vector as the learnable parameter. 13. An information processing apparatus according to any one of 1 to 11, wherein the learnable parameter is composed of a plurality of elements arranged in a row, and the parameter assignment means determines the value of each element of the learnable parameter using a random number generator.14. An information processing device according to any one of 1 to 13, further comprising a plurality of parameter assignment means, and further comprising a selection means for selecting a parameter assignment means from the plurality of parameter assignment means for processing the multidimensional tensor based on the data of the multidimensional tensor. 15. An information processing device according to 14, wherein the selection means is configured using a trained network learned from images and language. 16. An information processing device according to any one of 1 to 15, further comprising a recognition means for receiving the input of the transformed multidimensional tensor and performing a process to recognize the transformed multidimensional tensor, wherein the preprocessing means is configured to be differentiable, and further comprising a parameter update means for updating the learnable parameters based on a loss function that calculates the difference between the output of the recognition means and the correct data. 17. An information processing device according to 16, wherein the learning rate of the parameter assignment means during learning to update the learnable parameters is higher than the learning rate of the recognition means. 18. An information processing device according to any one of 1 to 17, wherein the multidimensional tensor represents the data of pixels in a color image, and the sorting means sorts the multidimensional tensor by color. 19. An information processing method comprising one or more computers sorting a multidimensional tensor, assigning learnable parameters to the sorted multidimensional tensor, and generating and outputting a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor. 20. A recording medium that stores a program causing a computer to execute a sorting step of sorting a multidimensional tensor, and a parameter assignment step of assigning learnable parameters to the sorted multidimensional tensor, and generating and outputting a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor.

[0182] Some or all of the appendices 2 to 18, which are dependent on the information processing device described in appendice 1 above, may also be dependent on the information processing method in appendice 19 and the program and recording medium in appendice 20, in the same dependent relationship as between appendice 1 and appendices 2 to 18. Furthermore, without departing from the embodiments described above, some or all of the configurations described as appendices can be realized in various hardware, software, various recording means for recording software, or systems.

[0183] 10 Information Processing Unit 11 Preprocessing Unit 111 Sort Unit 112 Parameter Assignment Unit 113 Color Conversion Unit 114 Reduction Unit 12 Reference Histogram Extraction Unit 13 Selection Unit 14 Recognition Unit 15 Parameter Update Unit 151 Loss Calculation Unit 152 Gradient Calculation Unit 153 Update Unit 1A Processor 2A Memory 3A Input / Output I / F 4A Peripheral Circuits 5A Bus

Claims

1. An information processing apparatus having a preprocessing means comprising: sorting means for sorting a multidimensional tensor; parameter assignment means for assigning learnable parameters to the sorted multidimensional tensor, and generating and outputting a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor.

2. The information processing apparatus according to claim 1, wherein the sorting means converts the multidimensional tensor into a one-dimensional vector, sorts the one-dimensional vector to generate a sorted one-dimensional vector, and the parameter assignment means assigns the learnable parameters to the sorted one-dimensional vector.

3. The information processing apparatus according to claim 2, wherein the sorted one-dimensional vector is composed of N elements, the learnable parameter is composed of fewer than N elements, the parameter assignment means converts the learnable parameter into data with N elements, and assigns the learnable parameter, after being converted into data with N elements, to the sorted one-dimensional vector.

4. The information processing apparatus according to claim 2, wherein the sorted one-dimensional vector and the learnable parameters are configured by arranging the same number of elements in a sequence.

5. The information processing apparatus according to any one of claims 2 to 4, wherein the sorting means converts the multidimensional tensor into a one-dimensional vector having multiple elements arranged in a row, and sorts the multiple elements of the one-dimensional vector based on the magnitude of the values ​​of each element to generate the sorted one-dimensional vector.

6. The information processing apparatus according to any one of claims 2 to 5, wherein the parameter assignment means updates the value of the nth element in the sorted one-dimensional vector using the value of the nth element in the learnable parameter to generate an updated one-dimensional vector; returns a plurality of elements of the updated one-dimensional vector to their original order before sorting the one-dimensional vector to generate an inversely sorted one-dimensional vector; and generates the transformed multidimensional tensor based on the values ​​of each of the plurality of elements of the inversely sorted one-dimensional vector.

7. The information processing apparatus according to claim 6, wherein the parameter assignment means generates the updated one-dimensional vector by replacing the value of the nth element in the sorted one-dimensional vector with the value of the nth element in the learnable parameter.

8. The information processing apparatus according to claim 6, wherein the parameter assignment means generates the updated one-dimensional vector by replacing the value of the nth element in the sorted one-dimensional vector with the product of the value of the nth element in the sorted one-dimensional vector and the value of the nth element in the learnable parameter.

9. The information processing apparatus according to any one of claims 1 to 8, wherein the multidimensional tensor represents the data of pixels in a color image, the preprocessing means further comprises a color conversion means that performs a process to convert the color space of the color image to the multidimensional tensor, and the sorting means sorts the multidimensional tensor after the color space conversion process has been performed.

10. The information processing apparatus according to claim 9, wherein the parameter assignment means generates the converted multidimensional tensor and then performs a process to restore the color space of the color image to its original state for the converted multidimensional tensor.

11. The information processing apparatus according to any one of claims 1 to 10, wherein the multidimensional tensor represents the data of pixels in an image, the preprocessing means further comprises a reduction means that performs downsampling to reduce the number of pixels in the multidimensional tensor, the sorting means sorts the multidimensional tensor after downsampling, and the parameter assignment means generates the converted multidimensional tensor and then performs upsampling to increase the number of pixels in the converted multidimensional tensor.

12. The information processing apparatus according to any one of claims 1 to 11, further comprising a reference histogram extraction means for converting a reference multidimensional tensor into a reference one-dimensional vector, sorting the reference one-dimensional vector to generate a sorted reference one-dimensional vector, wherein the parameter assignment means uses the sorted reference one-dimensional vector as the learnable parameter.

13. The information processing apparatus according to any one of claims 1 to 11, wherein the learnable parameter is composed of a plurality of elements arranged in a sequence, and the parameter assignment means determines the value of each element of the learnable parameter using a random number generator.

14. An information processing apparatus according to any one of claims 1 to 13, further comprising a plurality of parameter assignment means, and a selection means for selecting a parameter assignment means from the plurality of parameter assignment means for processing the multidimensional tensor based on the data of the multidimensional tensor.

15. The information processing apparatus according to claim 14, wherein the selection means is configured using a trained network learned from images and language.

16. The information processing apparatus according to any one of claims 1 to 15, further comprising recognition means for receiving the input of the transformed multidimensional tensor and performing a process for recognizing the transformed multidimensional tensor, wherein the preprocessing means is configured to be differentiable, and further comprising parameter update means for updating the learnable parameters based on a loss function that calculates the difference between the output of the recognition means and the correct data.

17. The information processing apparatus according to claim 16, wherein the learning rate of the parameter assignment means during learning to update the learnable parameters is higher than the learning rate of the recognition means.

18. The information processing apparatus according to any one of claims 1 to 17, wherein the multidimensional tensor represents the data of pixels in a color image, and the sorting means sorts the multidimensional tensor for each color.

19. An information processing method comprising one or more computers sorting a multidimensional tensor, assigning learnable parameters to the sorted multidimensional tensor, and generating and outputting a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor.

20. A recording medium that stores a program that causes a computer to execute a sorting step of sorting a multidimensional tensor, and a parameter assignment step of assigning learnable parameters to the sorted multidimensional tensor, and generating and outputting a transformed multidimensional tensor by transforming the histogram of the multidimensional tensor.

Images