Processing system, learning system, processing method, learning method, and program

Abstract

This processing system comprises a conversion unit that converts m-number of items of data to be processed, which are obtained by dividing input data of an image or audio, into a discrete token row that can be input to a first machine learning model and in which n-number of discrete tokens are arranged. The conversion unit outputs the discrete token row so as to include n - k-number of the discrete tokens, wherein k-number of the discrete tokens having a low level of importance have been excluded from the n-number of discrete tokens (m, n, and k are all natural numbers, n is arbitrary relative to m, and n > k).

Description

Processing system, learning system, processing method, learning method, and program Cross-reference to related applications

[0001] This application is based on Japanese Patent Application No. 2025-6819 filed on January 17, 2025, claims the benefit of its priority, and all the contents of that patent application are incorporated herein by reference.

[0002] The present invention relates to a processing system, a learning system, a processing method, a learning method, and a program.

[0003] Conventionally, input data such as images and sounds has been converted into discrete tokens, which are one-dimensional information suitable for input into a machine learning model such as a VLM (Vision and Language Model) or an LLM (Large Language Model), or a multimodal AI similar to these, and the discrete tokens are input into the machine learning model to obtain a processing result, or input into a decoder to obtain restored data. Research has been underway on this. Since the amount of data of discrete tokens is compressed compared to the input data, there is an advantage that the processing can be speeded up or simplified by using this as an intermediate representation. This advantage is very beneficial in scenarios where high-speed processing is required, such as autonomous driving, or in scenarios where data with a large amount of data, such as videos, is stored in a database.

[0004] In relation to this, Non-Patent Document 1 describes that in the technique of converting an image into a discrete token sequence, by pre-learning to drop (exclude) a random number of discrete tokens from the discrete token sequence at the end of the token encoding of each frame, subsequent tokens can be omitted as needed during processing to reduce the load of information processing.

[0005] Furthermore, Patent Document 1 describes a device that divides a moving image into line blocks and captures them as block data, and dynamically sets the compression ratio of each block data, wherein the compression ratio of block data that starts capturing later is set lower than that of block data that starts capturing earlier. However, this technique does not convert the input data into discrete tokens that can be suitably input into a machine learning model.

[0006] Japanese Patent Publication No. 2011-211617

[0007] "ELASTICTOK: ADAPTIVE TOKENIZATION FOR IMAGE AND VIDEO," Wilson Yan et al., published October 10, 2024. "Release of Japanese Vision Language Model heron-blip-v1," Turing Inc., published March 6, 2024. https: / / zenn.dev / turing_motors / articles / 00df893a5e17b6

[0008] In the technology described in Non-Patent Document 1, the number of input data to be processed is the same as the number of feature information pieces to be discretized. Therefore, it was sometimes difficult to appropriately adjust the amount of information in the discrete token sequence or to adjust the processing load by arbitrarily changing the compression ratio. Furthermore, in the technology described in Non-Patent Document 1, if a machine learning model is used for the conversion to discrete tokens, there is a concern that the learning efficiency may decrease during the training phase. Thus, conventional technologies sometimes failed to achieve suitable data processing.

[0009] This invention has been made in consideration of these circumstances, and one of its objectives is to provide a processing system, a learning system, a processing method, a learning method, and a program that can realize suitable data processing.

[0010] One aspect of the present invention is a processing system comprising a conversion unit that converts m data to be processed, obtained by dividing input data which is an image or sound, into a discrete token sequence that can be input to a first machine learning model, the discrete token sequence being an array of n discrete tokens, wherein the conversion unit outputs a discrete token sequence containing n-k discrete tokens, with k discrete tokens of low importance excluded from the n discrete tokens (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k).

[0011] Another aspect of the present invention is a learning system comprising: a first trial unit that inputs m child learning data obtained by dividing learning data, which is an image or sound, into a second machine learning model and outputs n feature information along with ordering, and converts each of the n-k feature information obtained by excluding k feature information in ascending order from the n feature information into discretized feature information and encodes it, and outputs a discrete token sequence in which n-k discrete tokens are arranged; a second trial unit that converts the n-k discrete tokens into discretized feature information and obtains reconstructed data output by inputting the discretized feature information into a third machine learning model (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k); and an adjustment unit that adjusts the second and third machine learning models so that the evaluation result of evaluating the reproducibility of the reconstructed data with respect to the learning data is good.

[0012] According to one aspect of the present invention, suitable data processing can be achieved.

[0013] This is a diagram illustrating the configuration of the processing system according to the first embodiment. This is a diagram (part 1) illustrating the processing of the processing system according to the first embodiment. This is a diagram (part 2) illustrating the processing of the processing system according to the first embodiment. This is a diagram illustrating the configuration of the learning system. This is a diagram (part 1) illustrating the processing of the learning system. This is a diagram (part 2) illustrating the processing of the learning system. This is a diagram illustrating an example of the processing system being applied to an autonomous vehicle. This is a diagram illustrating the processing of the processing system according to the second embodiment. This is a diagram illustrating an example of the processing system being applied to a data server. This is a diagram illustrating the processing of the learning stage when handling voice. This is a diagram illustrating the processing of the processing system according to the third embodiment.

[0014] [Outline] The following describes embodiments of the processing system, learning system, processing method, learning method, and program of the present invention with reference to the drawings. Each of the processing system and the learning system is implemented by, for example, one or more processors. These may be dedicated systems or programs installed and functioning on a cloud server. The processing system converts the data to be processed, obtained by dividing input data which is an image or sound, into a discrete token sequence that can be input into a first machine learning model such as a VLM or LLM, or a similar multimodal AI. The processing system reduces the processing load on subsequent processing by excluding some discrete tokens that are judged to be of low importance from the discrete token sequence. Furthermore, in the processing system, the number of discrete tokens n included in the discrete token sequence is mutually arbitrary with respect to the number of data to be processed m. "Mutually arbitrary" means that m may be arbitrarily determined for a fixed number n, or m may be arbitrarily determined for a fixed number m, or both n and m may be arbitrarily determined. This allows the processing system to achieve more flexible data processing. The learning system trains the second machine learning model included in the above-mentioned transformation unit in order to enable the processor to function as a processing system. These will be explained in detail below.

[0015] <First Embodiment> [Processing System] The processing system 100 of the first embodiment will be described below. Figure 1 is a configuration diagram of the processing system according to the first embodiment. The processing system 100 includes, for example, an acquisition unit 110, a processing unit 120, a conversion unit 130, and a storage unit 170. The conversion unit 130 includes an encoder unit 132 and a discretization / encoding unit 134. The components other than the storage unit 170 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The program may be stored in advance on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored on a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into the drive device.

[0016] The storage unit 170 can be RAM (Random Access Memory), HDD, flash memory, etc. The storage unit 170 stores the second machine learning model 172, the codebook 174, etc. The storage unit 170 is not attached to the processing system 100, but may be a NAS (Network Attached Storage) that the processing system 100 can access via a network.

[0017] Figure 2 is a diagram (part 1) illustrating the processing of the processing system according to the first embodiment. The acquisition unit 110 acquires input data 300, which is either an image or audio. The acquisition unit 110 may acquire the input data 300 from an external device via communication, from a storage medium mounted on a drive device provided by the processing system 100, or by reading the input data from the non-volatile area of ​​the storage unit 170 and storing it in a main processing area such as RAM. In the example in Figure 2, the input data 300 is an image.

[0018] The processing unit 120 processes the input data 300 to generate the data to be processed 310. For example, the processing unit 120 may divide the input data 300, which is a single still image, into m images of arbitrary sizes, or extract m frame images at different times from the input data 300, which is a moving image. In addition, the processing unit 120 may perform various other processing necessary for image processing, such as excluding the outer frame area of ​​the input data 300. Here, m may be a fixed value, set arbitrarily, or changed dynamically.

[0019] The conversion unit 130 converts the data to be processed 310 (here, each of m images obtained by dividing a single still image into m parts) into a discrete token sequence that can be input to the first machine learning model (described later), which is a discrete token sequence in which multiple discrete tokens are arranged. At this time, for each data to be processed 310, the conversion unit 130 outputs a discrete token sequence containing n-k discrete tokens, which are obtained by excluding k discrete tokens of low importance from the n discrete tokens obtained from the data to be processed. m, n, and k are all natural numbers, and n is mutually arbitrary with respect to m, and n > k holds. The system may be configured such that n ≠ m.

[0020] Let me explain in more detail. The encoder unit 132 of the conversion unit 130 inputs m data points 310 to be processed into the second machine learning model 172 and converts them into n feature information points 320#1 to 320#n. Feature information can be, for example, Haar-Like feature information, HOG feature information, SIFT feature information, or feature information obtained by machine learning models such as CNN, VGG, and ResNet. For example, feature information has the form of a vector and can be rephrased as a feature vector.

[0021] The second machine learning model 172 is a model trained to output n feature information 320#1 to 320#n in an ordered manner. The second machine learning model 172 is trained to output the n feature information 320#1 to 320#n in an ordered manner, with higher importance being prioritized. In the n feature information 320#1 to 320#n, a smaller number after "#" indicates a higher order (higher importance).

[0022] The discretization and coding unit 134 removes the k feature information 320#n-k+1 to 320#n from the n feature information 320#1 to 320#n in descending order, resulting in n-k feature information 320#1 to 320#n-k. Each of these n-k feature information 320#1 to 320#n-k is converted into n-k discrete tokens 330#1 to 330#n-k, and these are output as a sequence of discrete tokens. A discrete token is a code or symbol that is represented by discrete values ​​or a finite number of symbols (i.e., not a continuous value assuming significant figures). In the figure, the numbers 14, 21, and 13 are examples of discrete tokens. The discretization and coding unit 134 converts the feature information into discrete tokens using, for example, a codebook 174. The codebook 174 is information in which the corresponding discrete token is described for each of the multiple feature information (in the same form as those output by the encoder unit 132, for example, vectors of the same number of dimensions). For example, the discretization and coding unit 134 extracts the feature information with the smallest distance index, such as cosine similarity, (i.e., the closest content) for each of the n-k feature information items 320#1 to 320#n-k from the codebook 174, and determines each of the discrete tokens corresponding to the extracted feature information as n-k discrete tokens 330#1 to 330#n-k. In other words, the discretization and coding unit 134 discretizes the continuous feature information by matching it to one of the finite number of feature information items registered in the codebook 174, and then codes the discretized feature information by converting it into labels (codes) such as 14, 21, and 13. The discretization and coding unit 134 outputs this as a discrete token sequence. Alternatively, the discretization and coding unit 134 may convert each of the n feature information items 320#1 to 320#n into n discrete tokens, and then exclude the k discrete tokens with the lowest order. The discretization and coding unit 134 is not limited to using the codebook 174; for example, it may perform discretization using a method such as a Lookup-Free Quantizer that discretizes each element of the vector into binary values ​​of 1 or -1.

[0023] Here, the second machine learning model 172 is trained together with the third machine learning model so that it produces a good evaluation result when the reproducibility of the reconstructed data to the training data is evaluated by inputting the m child training data obtained by dividing the training data, which is an image or sound, into the second machine learning model 172, and thereby converting the sequence of discrete tokens, which contains n-k discrete tokens that encode n-k discrete feature information, into discrete feature information, and inputting the discrete feature information into the third machine learning model. The process of converting discrete tokens into discrete feature information is performed, for example, by reverse lookup in the codebook 174 (obtaining feature information corresponding to discrete tokens). Through this training, the feature information that is the basis of discrete tokens of low importance is output from the second machine learning model 172 in a lower order.

[0024] The processing system 100 may also input n latent tokens 315 to the encoder unit 132 along with the data to be processed 310. Figure 3 is a diagram (part 2) illustrating the processing of the processing system according to the first embodiment. In this case, the encoder unit 132 outputs feature information in which m data to be processed 310 are projected onto an n-dimensional latent space corresponding to n latent tokens 315. The latent tokens 315 and the mask tokens, which will be described later, are learned together when the second machine learning model or the third machine learning model is trained. Whether or not to input the latent tokens 315 to the encoder unit 132 depends on the configuration (architecture) of the encoder unit 132, and either option is acceptable. If the latent tokens 315 are not input to the encoder unit 132, the encoder unit 132 includes a function to project m data to be processed onto an n-dimensional latent space. The same applies to the second and third embodiments.

[0025] According to the processing system 100 of the first embodiment described above, for each piece of data to be processed, a discrete token sequence containing n-k discrete tokens is output, which is obtained by excluding k discrete tokens of low importance from the n discrete tokens obtained from the data to be processed. This reduces the processing load for subsequent processing. Specific examples of subsequent processing will be described later in [Application Example 1], [Application Example 2], etc. Furthermore, by making n and m mutually arbitrary (there is no need to make n match m), the amount of information in the discrete token sequence can be adjusted as needed, and the compression ratio can be arbitrarily changed to adjust the processing load. As a result, suitable data processing can be achieved.

[0026] The number of latent tokens n corresponds to the expressive power when discrete tokenization is performed. Therefore, if the number of discrete tokens output is the same, increasing the number of latent tokens allows for greater expressive power (the ability to retain high-resolution image information). In contrast, in the method of Non-Patent Literature 1, the number of input patch sizes (the number of data to be processed in this application), the number of latent tokens, and the number of feature information outputs are the same. Therefore, it is not possible to adjust the expressive power as needed, but in this embodiment, n is made arbitrary relative to m, so it is possible to arbitrarily adjust the relative sizes of these during training. Therefore, if you want to increase the expressive power, you can build the system in that way. Conversely, if you want to perform inference calculations faster, it is advantageous to reduce the number of latent tokens. Again, in this embodiment, the system can be arbitrarily built to achieve this as well. Thus, according to this embodiment, the degree of freedom in system design is increased, and the system can be given desired characteristics.

[0027] The processing system 100 may be configured, for example, as a cloud server and include a plurality of conversion units 130 corresponding to at least two of still images, videos, and audio, and may be configured to selectively operate the conversion unit 130 according to the type of input data specified by the user. In this case, the conversion unit 130 for still images is one in which the second machine learning model 172 has been trained using training data that is still images, the conversion unit 130 for videos is one in which the second machine learning model 172 has been trained using training data that is video, and the conversion unit 130 for audio is one in which the second machine learning model 172 has been trained using training data that is audio.

[0028] [Learning System] The learning system 200 for learning the processing system 100 according to the first embodiment will be described below. Figure 4 is a diagram of the configuration of the learning system. The learning system 200 includes, for example, a first trial unit 210, a second trial unit 220, an adjustment unit 230, a weighting unit 240, and a storage unit 270. The components other than the storage unit 270 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, GPU, or SOC, or by the cooperation of software and hardware. The program may be stored in advance in a storage device such as an HDD or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is mounted on a drive device.

[0029] The memory unit 270 can be RAM, HDD, flash memory, etc. The memory unit 170 stores the second machine learning model 272 in the learning phase, the third machine learning model 274 in the learning phase, the codebook 174, etc. The codebook 174 is the same as the one used by the discretization and coding unit 134 of the processing system 100. Similar to the processing system 100, a method that does not use the codebook 174 may be used. The memory unit 270 is not attached to the learning system 200, and may be a NAS that the learning system 200 can access via a network.

[0030] Figure 5 is a diagram (part 1) illustrating the processing of the learning system. The first trial unit 210 inputs m child learning data 410 and n latent tokens 415, obtained by dividing the learning data 400, which is an image or sound, into the second machine learning model 272 in the learning stage, outputs n feature information along with ordering, and outputs a discrete token sequence of n-k discrete tokens 430#1 to 430#n-k, obtained by discretizing the n feature information by removing k feature information in ascending order from the n feature information. The content of this processing is the same as that of the encoder unit 132 and the discretization / encoding unit 134 of the processing system 100. Similarly, it is optional whether or not to input n latent tokens 415 into the second machine learning model 272 in the learning stage.

[0031] The second trial unit 220 converts n-k discrete tokens 430#1 to 430#n-k into discretized feature information, and outputs reconstructed data 420 by inputting the discretized feature information together with m mask tokens 417 into the third machine learning model 274 in the learning stage (see description of the second machine learning model 172). The second machine learning model 272 and the third machine learning model 274 have an encoder / decoder relationship. The m mask tokens 417 act to return the discretized feature information in the n-k dimensional latent space back to the original m dimensional space. Whether or not to input the m mask tokens 417 into the third machine learning model 274 is arbitrary, just as whether or not to input n latent tokens 415 into the second machine learning model 272 in the learning stage in the processing system 100.

[0032] The adjustment unit 230 compares the reconstructed data 420 with the training data 400 and adjusts the parameters of the second machine learning model 272 and the third machine learning model 274 so that the evaluation result of the reproducibility of the reconstructed data 420 with respect to the training data 400 is good. The evaluation result of reproducibility is, for example, the value obtained by calculating the difference in gradation for each pixel and summing or averaging it, and the smaller this value, the better the evaluation result. The evaluation result is not limited to this, and reproducibility may be evaluated by any method. For example, if the image evaluation index SSIM is used as the evaluation result, the larger the value, the better the evaluation result.

[0033] In the evaluation of reproducibility by the adjustment unit 230, the weights set by the weighting unit 240 may be reflected. The weighting unit 240 extracts a first region in the image, which is the training data 400, in which a predetermined object is represented, and assigns different weights to the first region and a second region that is not the first region. For example, if the processing system 100 is provided to an autonomous vehicle (hereinafter simply referred to as "vehicle") and determines the vehicle's behavioral guidelines based on an image taken of the vehicle's exterior, the weighting unit 240 assigns the largest weight to pedestrians, bicycles, oncoming vehicles, traffic lights, etc., followed by large weights to preceding vehicles and vehicles driving alongside, and small weights to background objects such as the road surface and buildings. Various methods for object extraction in images are already known and will not be described in detail here, but for example, template matching and methods using machine learning models may be used.

[0034] Here, the method for evaluating the reproducibility of the reconstructed data 420 to the training data 400 is not limited to simply directly comparing the training data 400 and the reconstructed data 420, but the following methods may be adopted. Figure 6 is a diagram (part 2) illustrating the processing of the learning system. In this method, the adjustment unit 230 calculates GAN (Generative Adversarial Network) Loss as the first evaluation result and Perceptual Loss as the second evaluation result. The adjustment unit 230 may, for example, calculate an integrated evaluation result by obtaining a weighted sum of the first evaluation result and the second evaluation result, or it may perform parameter adjustments based on the first evaluation result and parameter adjustments based on the second evaluation result separately.

[0035] The adjustment unit 230 inputs the training data 400 and the reconstructed data 420 to the discriminator network 232. The discriminator network 232 functions as a GAN together with the first trial unit 210 and the second trial unit 220, which are used as the generator network. In other words, the discriminator network 232 determines from the image alone whether the training data 400 or the reconstructed data 420 is the original data (i.e., the training data 400), and outputs the determination result. GANLoss returns a good evaluation result when the discriminator network 232 outputs a correct determination result, and also returns a good evaluation result when the second machine learning model 272 and the third machine learning model 274 in the training stage cause the discriminator network 232 to make a wrong determination. In this way, the adjustment unit 230 repeatedly improves (adjusts parameters) both the generator network and the discriminator network 232 competitively, adjusting the reconstructed data 420 to approach the training data 400.

[0036] Meanwhile, the adjustment unit 230 inputs the training data 400 and the reconstructed data 420 to the image recognition model 234, respectively. The image recognition model 234 is, for example, a pre-trained VGG network that performs hierarchical processing to reduce dimensionality compared to at least the input layer and has the function of discriminating objects based on the feature information of the intermediate layers (for example, discriminating vehicles, pedestrians, buildings, etc. based on the feature information of the third layer from the input layer). The adjustment unit 230 calculates a second evaluation result based on the result of comparing one or more intermediate layers of the same hierarchical level. The "result of comparing intermediate layers" refers to values ​​such as L1 loss and L2 loss.

[0037] GAN Loss enhances the "realism of the generated image," while Perceptual Loss works to improve "visual quality and detail reproduction." For example, in super-resolution and style transfer tasks, GAN Loss can be used to pursue realism while Perceptual Loss corrects visually unnatural parts, resulting in realistic and high-quality image generation.

[0038] When the above process is repeatedly performed on multiple training data 400, the final second machine learning model 272 at the training stage is used as the second machine learning model 172 in the processing system 100.

[0039] According to the learning system 200 described above, a second machine learning model 172, which functions as part of the processing system 100, can be suitably trained. The basic operation of the learning system is the same even when constructing the processing system in the form of Figure 3, or the processing systems of the second and third embodiments.

[0040] Furthermore, according to the learning system 200, in addition to making n mutually arbitrary with respect to m, the number of k can also be arbitrarily determined, which allows for efficient learning even when the number of latent tokens is small. For example, in the method of Non-Patent Literature 1, when the input patch size and the number of latent tokens are large compared to the number of discrete tokens to be output, many learning iterations are required to consolidate the information into the discrete tokens to be output. For example, if the former is 4096 and the latter is 8, the information needs to be consolidated into 0.2% of the discrete tokens. In contrast, in the method of this embodiment, by setting the number of latent tokens n to, for example, 256 and the number of discrete tokens to be output n-k to 8, it is sufficient to consolidate the information into 3.1% of the discrete tokens. Therefore, the learning efficiency, especially in the initial stages of learning, is improved compared to the method of Non-Patent Literature 1, and the processing system can be constructed more quickly.

[0041] [Application Example 1] The following describes an application example 1 of the processing system. Any of the first to third embodiments may be used in application example 1, but the first embodiment will be used here. Figure 7 shows an example in which the processing system is applied to an autonomous vehicle. The autonomous vehicle 10 is equipped with, for example, a camera 20, a detection device 22, a sensor 24, a first control device 26, a communication device 28, a second control device 30, a controlled object 32, and a storage device 34, and performs autonomous driving control using the processing results of the processing system 100 at least partially.

[0042] Camera 20 is mounted anywhere on the vehicle 10 and captures images of the scenery in front of, to the side of, and behind the vehicle 10. Detection device 22 includes radar, LIDAR (Light Detection and Ranging), ultrasonic sensors, etc., and detects objects present around the vehicle 10. Sensor 24 includes speed sensors, acceleration sensors, gyro sensors, steering angle sensors, or accelerator opening sensors, brake sensors, steering torque pressure sensors, etc., which are used when manual driving is performed.

[0043] The first control device 26 includes the processing system 100. The first control device 26 operates, for example, by a processor such as a CPU executing a program stored in a storage device. Note that part or all of the functions of the first control device 26 may be realized by hardware such as an LSI, an ASIC, an FPGA, a GPU, or a SoC, or may be realized by cooperation between software and hardware. The first control device 26 executes processing for performing integrated recognition and decision-making in autonomous driving. The processing system 100 that functions as a part of the first control device 26 inputs the image captured by the camera 20 as input data 300, and transmits the discrete token sequence generated by the processing system 100 as described above to the VLM server 40 using the communication device 28, and inputs it to the first machine learning model 42 of the VLM server 40. The communication device 28 accesses the network NW by wireless communication and communicates with the VLM server 40. The network NW includes a wireless base station, a WAN, a LAN, the Internet, and the like. The first machine learning model 42 includes, for example, an LLM (Large Language Model), as shown in Non-Patent Document 2, and is learned to recognize the environment in which the vehicle 10 is placed and output necessary action guidelines when a discrete token sequence is input. The action guidelines are information including, for example, instructions regarding the medium- and long-term actions of the vehicle 10 and the reasons for reaching the judgment to execute the instructions. For example, it is information close to natural language such as "Please stop because the signal color is red.", "There is a sharp curve ahead, so please slow down gently.", "Please turn on the blinker because you will turn left ahead.", "There is an intersection ahead, please go right."

[0044] The first machine learning model 42 may be mounted on the vehicle 10 instead of being placed outside the vehicle 10. In this case, for example, the processing system 100 holds the first machine learning model 42, other programs, data, etc. for functioning as a VLM or an LLM, or a multimodal AI similar thereto in the storage unit 170, and mounts the functions of a VLM or an LLM, or a multimodal AI similar thereto as the function of the encoder unit 132 or as the function of other functional units.

[0045] The second control device 30 controls the controlled object 32 with more specific details while referring to the map information 36 stored in the storage device 34. The controlled object 32 includes, for example, drive force output devices such as engines, motors, and transmissions, brakes, steering devices, and turn signals. The second control device 30, for example, in response to a guideline such as "There is a sharp curve ahead, so please slow down gradually," decelerates from XX [m] before the curve to XX [m / s]. ２ The control method, which involves decelerating at [ ], is determined, and a control instruction to achieve this is output to the controlled object 32. The controlled object 32 may include an ECU (Electronic Control Unit) that controls each device individually.

[0046] In this way, by applying the processing system 100 to an autonomous vehicle, the amount of information exchanged with the first machine learning model 42 is reduced while retaining important information and generating a discrete token sequence, thereby suppressing a decrease in the accuracy of processing in the first machine learning model 42, i.e., subsequent processing.

[0047] In the processing system 100 applied to an autonomous vehicle, the acquisition unit 110 acquires external information (external information related to input data) such as speed and driving environment, and based on the external information, the value of k may be dynamically changed. For example, when the speed of the vehicle 10 is low, since the amount of change in the image of the camera 20 is small, it is considered that even if the value of k is increased, the influence on the driving control of the vehicle 10 is small. Therefore, the conversion unit 130 may increase the value of k as the speed of the vehicle 10 decreases. Specifically, the conversion unit 130 may make the value of k when the speed of the vehicle 10 is less than the threshold greater than the value of k when the speed of the vehicle 10 is greater than or equal to the threshold. Further, the conversion unit 130 may change the value of k based on the attribute of the location where the vehicle 10 is traveling by analyzing the image captured by the camera 20 or based on a positioning device (not shown) and the map information 36. For example, when the vehicle 10 is traveling in an urban area, the conversion unit 130 may decrease the value of k because the amount of external environment information is large, and when the vehicle 10 is traveling on an exclusive vehicle road, the conversion unit 130 may increase the value of k because the amount of external environment information is small. Further, the conversion unit 130 may decrease the value of k when the vehicle 10 is traveling at night, or in rainy or snowy weather because it is necessary to acquire a large amount of external information, and may increase the value of k when the vehicle 10 is traveling during the day by analyzing the image captured by the camera 20 or based on the output value of an illuminance sensor (not shown). By doing so, the processing system 100 can reduce the amount of unnecessary data processing and perform faster processing. Also, the processing system 100 can maintain high processing accuracy by increasing the amount of data processing when necessary.

[0048] According to the first embodiment described above, it is possible to suppress a decrease in accuracy while reducing the processing load of subsequent processing.

[0049] <Second Embodiment> The second embodiment will be described below. Since the basic operations of the components are the same as those in the first embodiment, the same names and reference numerals as those in the first embodiment are used. FIG. 8 is a diagram for explaining the processing of the processing system according to the second embodiment.

[0050] In the second embodiment, the processing unit 120 generates processing target data 310 by extracting multiple still images and timestamps from the input data 300, which is a moving image. In this case, the processing target data 310 is a sequence of multiple timestamps and still images. In this case, the interval between timestamps, i.e., the time interval between still images, does not have to be constant (for example, still images with timestamps t=1, 3, 8, ... may be extracted). A timestamp is information that represents the order and time of still images, such as the elapsed time when the start time in the video is set to zero. In this case, the latent tokens 315 input to the encoder unit 132 are also arranged in the time direction.

[0051] [Application Example 2] The following describes an application example 2 of the processing system. In Application Example 1, any of the first to third embodiments may be used, but here the second embodiment will be applied. The processing system is applied to a data server. The data server stores, for example, video footage distributed via a network NW in the form of a discrete token sequence. The video footage is restored based on the discrete tokens in another terminal device or server. The processing system is implemented by the data server and another terminal device or server having a restoration unit 140. However, the processing system may also be implemented by a first device mounted on a first vehicle having a function equivalent to a data server, and a second device mounted on a second vehicle having a restoration unit 140. In this case, the network NW may include an inter-vehicle communication network. Figure 9 shows an example in which the processing system is applied to a data server. The restoration unit 140 receives discrete tokens 330#1 to 330#n-k and m mask tokens 317 as input. The restoration unit 140 converts the discrete tokens 330#1 to 330#n-k into discrete feature information using the codebook 174, inputs the discrete feature information together with m mask tokens 417 into the third machine learning model 176, and obtains the output of the third machine learning model 176 to restore a video image (restored data) that is close to the original video image input data 300.

[0052] In Application Example 2, the value of k may be changed so that the image evaluation index, such as SSIM, calculated by comparing the original video input data 300 with the video output by the restoration unit 140, is equal to or greater than a predetermined value. The image evaluation index is an example of "an evaluation result that assesses the reproducibility of the restored data with respect to the input data or the data to be processed." For example, the conversion unit 130 decreases the value of k if the image evaluation index is less than a predetermined value, and maintains or increases the value of k if it is equal to or greater than a predetermined value. The conversion unit 130 may use a dedicated machine learning model when deciding how much to change the value of k.

[0053] [When input data is audio] As mentioned above, the input data may be audio. When the input data is audio, the processing unit 120 performs processing such as Mel spectrogram conversion and frequency filtering. Figure 10 is a diagram illustrating the processing during the learning stage when handling audio. The same reference numerals are used for the same functional units as in Figure 5. The first trial unit 210 inputs the audio divided into audio sections, the child learning data 410 including timestamps, and the latent tokens 415 into the second machine learning model 272 in the learning stage, outputs n feature information along with ordering, and outputs a discrete token sequence of n-k discrete tokens 430#1 to 430#n-k obtained by excluding k feature information in descending order from the n feature information.

[0054] The second trial unit 220 performs the reverse process of discretization on n-k discrete tokens 430#1 to 430#n-k, and outputs reconstructed data 420 which is reconstructed by inputting the data processed using the mask token 417 into the third machine learning model 274 in the training stage.

[0055] The adjustment unit 230 compares the reconstructed data 420 with the training data 400 and adjusts the parameters of the second machine learning model 272 and the third machine learning model 274 so that the evaluation result of the reproducibility of the reconstructed data 420 with respect to the training data 400 is good. The evaluation result of reproducibility is, for example, the sum or average of the differences when comparing the intensity for each frequency, and the smaller this value, the better the evaluation result. The reproducibility may be evaluated using any method, not limited to this.

[0056] In the evaluation of reproducibility by the adjustment unit 230, the weights set by the weighting unit 240 may be reflected. The weighting unit 240 sets weights for frequencies based on the noise intensity of the speech, which is the learning data 400, or on pre-provided objective information. The objective information is information that represents the use and purpose of the processing system 100. For example, if the noise intensity is high, the weighting unit 240 sets a small weight for the frequency band corresponding to the noise so that many tokens in that frequency band are reduced. Also, if the objective information represents "speech recognition," the weighting unit 240 sets a small weight for the frequency band corresponding to silent sections or ambient sounds so that the speaker's utterances are preferentially tokenized. In this way, the weighting unit 240 may perform tokenization that emphasizes specific information (emotional characteristics, voice quality, content, etc.) depending on the purpose of use of the speech data.

[0057] Furthermore, in the process of changing the value of k in Application Example 2, the value of k may be dynamically changed according to the audio scene and application. For example, if the target information mentioned above is "speech recognition," the conversion unit 130 may change the value of k based on context prediction. Also, if the target information is "audio streaming," the conversion unit 130 may change the value of k according to the bandwidth of the audio being streamed.

[0058] According to the second embodiment described above, similar to the first embodiment, it is possible to reduce the processing load of subsequent processing while suppressing a decrease in accuracy.

[0059] <Third Embodiment> The third embodiment will now be described. The basic operation of the components is the same as in the second embodiment, so the same names and reference numerals as in the second embodiment will be used. Figure 11 is a diagram illustrating the processing of the processing system according to the third embodiment. In the third embodiment, first a reference frame image is input to the encoder unit 132. The reference frame image is the first frame image in a moving image, or the first few frames of a video. The target frame is an image that follows the reference frame image (a frame image that is later in time), and may be a difference image with respect to the reference frame image.

[0060] According to the third embodiment described above, similar to the first embodiment, it is possible to reduce the processing load of subsequent processing while suppressing a decrease in accuracy.

[0061] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A processing system comprising: a conversion unit that converts m data to be processed, obtained by dividing input data which is an image or sound, into a discrete token sequence that can be input to a first machine learning model, the discrete token sequence being an array of n discrete tokens, wherein the conversion unit outputs the discrete token sequence containing n-k discrete tokens, with k discrete tokens of low importance excluded from the n discrete tokens (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k).

2. The processing system according to claim 1, wherein the conversion unit has an encoder unit that inputs m data to be processed into a second machine learning model and converts them into n feature information, the second machine learning model is a model that has been trained to output the n feature information in an ordered manner, and the conversion unit further has a discretization and encoding unit that converts each of the n-k feature information obtained by excluding k feature information in order of decreasing order from the n feature information into discretized feature information and encodes it, and outputs n-k discrete tokens.

3. The processing system according to claim 2, wherein the second machine learning model is trained together with the third machine learning model to obtain a good evaluation result when the reproducibility of the reconstructed data to the training data is evaluated by inputting the m child training data obtained by dividing the training data, which is an image or sound, into the second machine learning model, and the output of the discrete token sequence, which includes n-k discrete tokens that encode n-k discrete feature information, into discrete feature information, and inputting the discrete feature information into the third machine learning model.

4. The processing system according to claim 1, wherein the conversion unit changes the value of k based on external information related to the input data.

5. The processing system according to claim 2, further comprising a restoration unit that converts the discrete token sequence output by the conversion unit into discrete feature information, and outputs restored data by inputting the discrete feature information into a third machine learning model.

6. The processing system according to claim 5, wherein the conversion unit changes the value of k based on an evaluation result that evaluates the reproducibility of the restored data output by the restoration unit with respect to the input data or the data to be processed.

7. The processing system according to claim 1, comprising a plurality of conversion units corresponding to at least two of still images, videos, and audio, wherein the conversion units are selectively operated according to the type of input data.

8. The processing system according to claim 1, further comprising: an acquisition unit that acquires input data; and a processing unit that divides the input data to generate m pieces of processing target data, wherein the input data is a moving image, the processing unit extracts m frame images from the moving image to generate the processing target data, and the time intervals between the m frame images are not constant.

9. The processing system according to claim 1, wherein the input data is a video image of the external scenery of a moving object, the first machine learning model outputs action guideline information for autonomous movement of the moving object when the discrete token sequence is input, and the conversion unit outputs the discrete token sequence as data to be input to the first machine learning model.

10. A processing system comprising: an encoder unit that inputs m data to be processed obtained by dividing input data, which is an image or sound, into a second machine learning model and converts it into n ordered feature information; a discretization and encoding unit that converts each of the n feature information into n discretized feature information and encodes them to convert them into n discrete tokens, and outputs a discrete token sequence containing n-k discrete tokens obtained by excluding k discrete tokens from the n discrete tokens in descending order of the corresponding feature information (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k); the second machine learning model is trained together with the third machine learning model so that the evaluation result is good when evaluating the reproducibility of the restored data to the learning data, which is output by inputting the n-k discrete feature information encoded into the discrete token sequence, which is output when m child learning data obtained by dividing learning data, which is an image or sound, into the second machine learning model, and inputting the discrete feature information into the third machine learning model.

11. The processing system according to claim 3 or 10, wherein the input data, the reconstructed data, and the training data are images, and the evaluation result for evaluating the reproducibility is derived based on both a first evaluation result obtained by directly comparing the reconstructed data and the training data, and a second evaluation result obtained by comparing the intermediate layers when the reconstructed data and the training data are input to an image recognition model that performs dimensionality reduction.

12. The processing system according to claim 11, wherein the first evaluation result is GAN (Generative Adversarial Network) Loss, and the second evaluation result is Perceptual Loss.

13. The processing system according to claim 1 or 10, wherein n ≠ m.

14. A learning system comprising: a first trial unit that inputs m child learning data obtained by dividing learning data which is an image or sound into a second machine learning model, outputs n feature information along with ordering, removes k feature information from the n feature information in ascending order, converts each of the n-k feature information to discretized feature information and encodes it, and outputs a discrete token sequence which is an arrangement of n-k discrete tokens; a second trial unit that converts the n-k discrete tokens to discretized feature information and obtains reconstructed data which is output by inputting the discretized feature information to a third machine learning model; and an adjustment unit that adjusts the second and third machine learning models so that the evaluation result of evaluating the reproducibility of the reconstructed data with respect to the learning data is good.

15. The learning system according to claim 14, wherein the learning data and the reconstructed data are images, and the evaluation result for evaluating the reproducibility is derived based on both a first evaluation result obtained by directly comparing the learning data and the reconstructed data, and a second evaluation result obtained by comparing the intermediate layers when the learning data and the reconstructed data are input to an image recognition model that performs dimensionality reduction.

16. The learning system according to claim 15, wherein the first evaluation result is GAN (Generative Adversarial Network) Loss, and the second evaluation result is Perceptual Loss.

17. The learning system according to claim 14, wherein the learning data is an image, and further comprises a weighting unit that extracts a first region in which a predetermined object is represented in the image and assigns different weights to the first region and a second region that is not the first region, and the adjustment unit evaluates the reproducibility by reflecting the weights.

18. The learning system according to claim 14, wherein the learning data is audio, and further comprises a weighting unit that assigns different weights to frequency bands according to pre-provided target information, and the adjustment unit evaluates the reproducibility by reflecting the weights.

19. The learning system according to claim 14, wherein n ≠ m.

20. A processing method performed by a computer, comprising the process of converting m data to be processed, obtained by dividing input data which is an image or sound, into a discrete token sequence that can be input to a first machine learning model, the discrete token sequence being an array of n discrete tokens, wherein the conversion process includes outputting the discrete token sequence containing n-k discrete tokens, with k discrete tokens of low importance excluded from the n discrete tokens (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k).

21. A learning method performed by a computer, comprising: a process of inputting m child learning data obtained by dividing learning data which is an image or sound into a second machine learning model and outputting n feature information along with ordering; a process of converting each of the n-k feature information obtained by excluding k feature information in ascending order from the n feature information into discretized feature information and encoding it, and outputting a discrete token sequence which is an array of n-k discrete tokens (where n and k are natural numbers, n > k); a process of converting the n-k discrete tokens into discretized feature information and obtaining reconstructed data which is output by inputting the discretized feature information into a third machine learning model; and a process of adjusting the second machine learning model and the third machine learning model so that the evaluation result of evaluating the reproducibility of the reconstructed data with respect to the learning data is good.

22. A program that causes a computer to perform a process of converting m data to be processed, obtained by dividing input data which is an image or sound, into a discrete token sequence that can be input to a first machine learning model, the discrete token sequence being an array of n discrete tokens, wherein the conversion process includes outputting the discrete token sequence containing n-k discrete tokens, wherein k discrete tokens of low importance have been excluded from the n discrete tokens (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k).

23. A program for causing a computer to perform the following: inputting m child training data obtained by dividing training data which is an image or sound into a second machine learning model and outputting n feature information along with ordering; converting each of the n-k feature information obtained by removing k feature information in ascending order from the n feature information into discretized feature information, encoding it, and outputting a discrete token sequence which is an array of n-k discrete tokens (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k); converting the n-k discrete tokens into discretized feature information and obtaining reconstructed data which is output by inputting the discretized feature information into a third machine learning model; and adjusting the second and third machine learning models so that the evaluation result of evaluating the reproducibility of the reconstructed data with respect to the training data is good.

Images