Information processing method and information processing device

Abstract

This information processing method is executed by a computer and includes inputting, to a large-scale language model, information including predetermined data and data of an instruction sentence for instructing output of information related to the predetermined data, and acquiring the information related to the predetermined data output from the large-scale language model on the basis of the input information. The predetermined data is data in which token reduction has been performed using information relating to an object of interest corresponding to the instruction sentence.

Description

Information Processing Method and Information Processing Apparatus

[0001] The present invention relates to an information processing method and an information processing apparatus.

[0002] Conventionally, large multimodal language models (LLMs; Large Language Models) capable of integrally handling a plurality of modalities such as text and images are known. In multimodal LLMs, image data such as still images and moving images has a larger input data volume than text data, and thus reduction of computational complexity is desired for expanded utilization.

[0003] Non-Patent Document 1 proposes a technique for token reduction by extracting representative tokens in an image and merging the remaining tokens on the basis of similarity.

[0004] Yuzhang Shang et al., "LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models", arXiv preprint arXiv:2403.15388, 2024

[0005] However, data after token reduction using the technique described in Non-Patent Document 1 above may have poor interpretability by the LLM, and there is a problem that it is difficult to identify whether the influence of accuracy degradation when token reduction is performed originates from the token reduction or the LLM. This is not limited to multimodal LLMs, and the same applies to LLMs other than multimodal LLMs.

[0006] The present invention has been made in view of the above, and an object thereof is to provide an information processing method and an information processing apparatus capable of suppressing accuracy degradation due to token reduction.

[0007] To solve the above-mentioned problems and achieve the objective, the present invention provides an information processing method for a computer, in which information including predetermined data and instruction data that instructs the output of information related to the predetermined data is input to a large-scale language model, and information regarding the predetermined data output from the large-scale language model based on the input information is obtained. The predetermined data is data on which token reduction has been performed using information of interest corresponding to the instruction.

[0008] According to the present invention, it is possible to suppress the decrease in accuracy caused by token reduction.

[0009] Figure 1 is a diagram illustrating an example of processing performed by the information processing device according to the embodiment. Figure 2 is a diagram illustrating an example of a template, an instruction statement for extracting a target of interest, a list of targets of interest, and a response statement in the information processing device according to the embodiment. Figure 3 is a diagram illustrating another example of processing performed by the information processing device according to the embodiment. Figure 4 is a diagram illustrating yet another example of processing performed by the information processing device according to the embodiment. Figure 5 is a diagram illustrating yet another example of processing performed by the information processing device according to the embodiment. Figure 6 is a schematic diagram illustrating the general configuration of the information processing device according to the embodiment. Figure 7 is a diagram illustrating an example of the target of interest list editing screen displayed on the display unit by the display processing unit in the processing unit of the information processing device according to the embodiment. Figure 8 is a schematic diagram illustrating the general configuration of the target of interest identification unit in the processing unit of the information processing device according to the embodiment. Figure 9 is a schematic diagram illustrating the general configuration of the token reduction processing unit in the processing unit of the information processing device according to the embodiment. Figure 10 is a flowchart illustrating an example of information processing by the processing unit of the information processing device according to the embodiment. Figure 11 is a flowchart illustrating an example of target of interest identification processing by the processing unit of the information processing device according to the embodiment. Figure 12 is a flowchart illustrating an example of response generation processing by the processing unit of the information processing device according to the embodiment. Figure 13 is a flowchart showing an example of token reduction processing by the processing unit of the information processing device according to the embodiment. Figure 14 is a diagram showing an example of a computer that executes an information processing program.

[0010] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0011] <Processing of the Information Processing Device> Figure 1 is a diagram illustrating an example of processing performed by the information processing device according to the embodiment. Figure 2 is a diagram illustrating an example of a template, an instruction statement for extracting a target of interest, a list of targets of interest, and a response statement in the information processing device according to the embodiment.

[0012] As shown in Figure 1, the information processing device 1 executes a process that includes a process for identifying the object of interest (step S10) and a response process with token reduction (step S20). In the following, we will mainly describe an example in which a multimodal large-scale language model (LLM) is used as the large-scale language model (LLM).

[0013] First, the process of identifying the object of interest in step S10 will be explained. The information processing device 1 receives information including task instruction data and image data. The task instruction data includes text data of an instruction that instructs the multimodal LLM to output information related to the image data.

[0014] The task instruction is used as part of the prompts entered into the multimodal LLM. In the following, the multimodal LLM may be referred to as MLLM. The image data is moving image data and may be referred to as input image data. Multiple still image data frames are extracted from the input image data. The still image data for each frame is an example of the target data.

[0015] First, the process of identifying the object of interest will be explained. The information processing device 1 acquires the data of the task instruction statement and the still image data of the keyframe extracted from the input image data (step S11). The keyframe is, for example, the first frame, but it may also be a frame specified by an operator (not shown). In the following, the still image data of the keyframe may be referred to as keyframe data.

[0016] The information processing device 1 performs an instruction creation process (step S12) to create instruction data for extracting a target of interest, which is suitable for the MLLM to extract the target of interest, using the task instruction data acquired in step S11. In the instruction creation process of step S12, the information processing device 1 creates instruction data for extracting a target of interest based on the task instruction data and predetermined template data. The target of interest is an object that should be noted in order to have the MLLM execute the task indicated in the task instruction, and is an object included in the image indicated by the input image data. The target of interest is not limited to an object, and may be, for example, a color, a character, a picture, etc.

[0017] For example, the information processing device 1 can create data for extracting objects of interest by embedding task instructions into a predetermined template. For example, if the task instructions are shown in Figure 2(a) and the template is shown in Figure 2(b), the instructions for extracting objects of interest are shown in Figure 2(c).

[0018] The information processing device 1 performs a tokenization process to convert the data of the instruction statement for extracting the target of interest, created in step S12, into multiple tokens (step S13). The information processing device 1 also performs a tokenization process to convert the keyframe data acquired in step S11 into multiple tokens (step S14).

[0019] The information processing device 1 performs a projection process (step S15) to project each of the multiple tokens obtained by the tokenization process in step S14 onto the language space. In step S15, the multiple tokens obtained by the tokenization process in step S14 are converted into multiple tokens in a format that can be handled by MLLM.

[0020] The information processing device 1 inputs the multiple tokens obtained in the tokenization process of step S13 and the multiple tokens obtained in the projection process of step S15 into the MLLM and obtains the information of the object of interest output from the MLLM (step S16). The MLLM may be included in the information processing device 1, or it may be used via an API (Application Programming Interface).

[0021] When the instruction statement for extracting the object of interest is shown in Figure 2(c), the information processing device 1 obtains information from the MLLM about the list of objects of interest, which includes "cats" and "stools" as information about the objects of interest, for example, as shown in Figure 2(d).

[0022] In this way, the information processing device 1 can perform a target identification process to identify the target of interest based on the task instruction statement and the input image data. As a result, the information processing device 1 can identify the target of interest with high accuracy.

[0023] Next, the response processing with token reduction in step S20 will be described. The information processing device 1 extracts still image data of multiple frames from the input image data and performs preprocessing on the extracted still image data (step S21). Preprocessing includes, for example, resizing and normalization. The information processing device 1 performs the processing in steps S22 to S25 for each still image data of the preprocessed still image data. In the following, the still image data of each frame extracted from the input image data may be referred to as frame data, and the preprocessed still image data may be referred to as preprocessed frame data.

[0024] The information processing device 1 performs a target coordinate detection process to detect the coordinates of the target in the still image shown in the preprocessed frame data, based on the preprocessed frame data and the information in the target list which includes information on the target identified in step S10 (step S22). The information processing device 1 also performs a tokenization process to generate tokens containing the image features of each region (patch) of the still image shown in the preprocessed frame data (step S23).

[0025] Next, the information processing device 1 performs a token reduction process (step S24) to select tokens from among the multiple tokens generated in step S23 that include at least a portion of the coordinates of the object of interest detected in the object of interest coordinate detection step of step S10. In step S24, tokens other than those corresponding to the coordinates of the object of interest are not selected, thereby allowing the information processing device 1 to reduce the number of tokens.

[0026] In step S24, if there are multiple tokens of the same object of interest, the information processing device 1 can merge these multiple tokens into a single token. This allows the information processing device 1 to reduce the number of tokens.

[0027] Furthermore, if the image is unclear or there is no object of interest in the image, the information processing device 1 can choose to continue processing while retaining the original number of tokens, without performing token reduction to guarantee accuracy. This approach ensures that subsequent processing is reliably executed even in cases where there are no tokens to retain as objects of interest.

[0028] Next, the information processing device 1 performs a projection process to project the tokens selected in the token reduction process of step S24 onto the language space (step S25). In step S25, the tokens selected in the token reduction process of step S24 are converted into tokens in a format that can be handled by MLLM. The tokens obtained in the projection process of step S25 are an example of predetermined data, which is data on which token reduction has been performed using the information of interest. The information processing device 1 also performs a tokenization process to convert the data of the task instruction into multiple tokens (step S26).

[0029] The information processing device 1 inputs the token obtained in the projection process of step S25 and the multiple tokens obtained in the tokenization process of step S26 into the MLLM and obtains the text data of the response statement output from the MLLM (step S27). The MLLM used in step S27 may be included in the information processing device 1, or it may be used via an API. Furthermore, the MLLM used in step S27 is different from the MLLM used in step S16, but is not limited to this example.

[0030] The information processing device 1 can perform the processing in step S27 for each frame data, and it can also input the tokens obtained from the processing in steps S21 to S25 for each of multiple frame data into the MLLM and perform the processing in step S27.

[0031] When the list of objects of interest is shown in Figure 2(d), the data of the response statement output from MLLM is, for example, the text data "two" shown in Figure 2(e), but is not limited to this example.

[0032] In this way, the information processing device 1 inputs data into the LLM (Likely Token Reduction Module) that has undergone token reduction using information about the object of interest corresponding to the task instruction. Therefore, the data that has undergone token reduction in the information processing device 1 accurately contains the tokens necessary to perform the task indicated in the task instruction. Consequently, the information processing device 1 can suppress the decrease in accuracy due to token reduction compared to techniques that extract representative tokens from an image and integrate the remaining tokens based on similarity.

[0033] In the example described above, the information processing device 1 extracts information of interest based on a single keyframe data extracted from the input image data, but it is not limited to this example. For example, the information processing device 1 can also extract keyframe data for each scene of a moving image shown in the input image data and extract information of interest for each scene. Figure 3 is a diagram illustrating another example of processing performed by the information processing device 1 according to the embodiment.

[0034] In the example shown in Figure 3, the information processing device 1 detects a scene change in the moving image shown by the input image data in step S30, and performs a frame extraction process to extract keyframe data for each scene based on the scene change detection result.

[0035] In step S10, the information processing device 1 identifies the object of interest for each scene based on the keyframe data for each scene obtained in the frame extraction process in step S30, and outputs information for the object of interest list, which is a list of the identified objects of interest, for each scene. In step S20, the information processing device 1 performs token reduction by selecting tokens containing the object of interest using the information from the object of interest list, which is different for each scene, and inputs the token-reduced data into the MLLM. This makes it possible to appropriately suppress the decrease in accuracy due to token reduction even when the input image data is long-duration video data.

[0036] The information processing device 1 uses, for example, the information in the list of objects of interest generated in the object of interest identification process of step S10 as is, but is not limited to this example. For example, the information processing device 1 may perform the response processing with token reduction in step S20 after the information in the list of objects of interest generated in the object of interest identification process of step S10 has been manually modified. Figure 4 is a diagram illustrating yet another example of processing performed by the information processing device 1 according to the embodiment.

[0037] In the example shown in Figure 4, the information processing device 1 displays the information of the list of objects of interest generated in the object of interest identification process in step S10 on an unshown display unit, and accepts user modifications to the list of objects of interest displayed on the display unit. Based on the list of objects of interest modified by the user, the information processing device 1 performs the response process with token reduction in step S20. As a result, the information processing device 1 can accurately include the tokens necessary to execute the task indicated in the task instruction statement, and can appropriately suppress the decrease in accuracy due to token reduction.

[0038] Also, in the above-described example, the information of the attention target list is generated in the attention target specifying process of step S10, but the present invention is not limited to such an example. For example, the information of the attention target list may be manually created. FIG. 5 is a diagram for explaining another example of the process performed by the information processing apparatus 1 according to the embodiment.

[0039] In the example shown in FIG. 5, the information processing apparatus 1 accepts a list of attention targets input by the user (step S10A). The information processing apparatus 1 performs the response process with token reduction process of step S20 based on the list of attention targets input by the user. Also with this configuration, the information processing apparatus 1 can accurately include tokens necessary for executing the task indicated by the task instruction statement, and can appropriately suppress a decrease in accuracy due to token reduction.

[0040] Hereinafter, the configuration and the like of the information processing apparatus 1 that performs such processes will be described in detail.

[0041] <Configuration of Information Processing Apparatus 1> FIG. 6 is a schematic diagram illustrating an exemplary schematic configuration of the information processing apparatus 1 of the embodiment. As shown in FIG. 2, the information processing apparatus 1 of the embodiment includes a communication unit 10, an input unit 11, a display unit 12, a storage unit 13, and a processing unit 14.

[0042] The communication unit 10 controls communication between an external apparatus and the processing unit 14 via an electric communication line such as a LAN or the Internet. The communication unit 10 is implemented by, for example, a NIC (Network Interface Card), a communication module, or the like.

[0043] The input unit 11 is an input device such as a mouse and a keyboard, and is operated by the user of the information processing apparatus 1.

[0044] The display unit 12 includes a liquid crystal display or an organic EL (Electro Luminescence) display, and displays various types of information. For example, the display unit 12 displays information of the attention target list and an image (for example, a moving image) obtained from input image data, and the like.

[0045] The storage unit 13 pre-stores an information processing program for operating the information processing apparatus 1, data used during execution of the information processing program, and the like, or temporarily stores such data each time processing is performed.

[0046] The storage unit 13 includes a template storage unit 20, a target-of-interest information storage unit 21, and a setting information storage unit 22. The template storage unit 20 stores information on a template (see, for example, (b) of FIG. 2) for creating an instruction for extracting a target of interest from a task instruction sentence.

[0047] The target-of-interest information storage unit 21 stores information on a target-of-interest list including information on one or more targets of interest. The information on the target-of-interest list stored in the target-of-interest information storage unit 21 is, for example, information on a target-of-interest list created or modified by the processing unit 14 or manually.

[0048] The setting information storage unit 22 stores setting information. The setting information includes, for example, information indicating whether to extract key frame data for each scene, information indicating whether to manually extract the target-of-interest list, information indicating whether to merge tokens, information indicating a token reduction condition, and the like.

[0049] The processing unit 14 is implemented using a processor such as a CPU or a GPU, and executes the information processing program stored in the storage unit 13. As a result, as illustrated in FIG. 6, the processing unit 14 functions as an acquisition unit 30, a reception unit 31, a display processing unit 32, a target-of-interest identification unit 33, a token reduction processing unit 34, a projection unit 35, a text tokenization unit 36, a generation unit 37, and an output unit 38 to execute information processing. Note that these functional units may be implemented entirely or partially in different hardware. In addition, the processing unit 14 may include other functional units.

[0050] The acquisition unit 30 acquires various types of information via the communication unit 10 or the input unit 11. For example, the acquisition unit acquires data of a task instruction sentence and input image data via the communication unit 10 or the like.

[0051] The reception unit 31 receives various requests via the communication unit 10 or the input unit 11. For example, the reception unit 31 receives setting requests. Setting requests include, for example, information indicating whether or not keyframe data should be extracted for each scene, information indicating whether or not the list of items of interest should be manually extracted, information indicating whether or not tokens should be merged, and information indicating token reduction conditions. For example, the reception unit 31 stores the information included in the received setting request in the setting information storage unit 22 as at least part of the setting information.

[0052] Furthermore, the reception unit 31 accepts modifications or settings to the list of objects of interest. For example, the reception unit 31 accepts settings to the list of objects of interest via the communication unit 10 or the input unit 11, and stores the information of the accepted list of objects of interest in the object of interest information storage unit 21. The reception unit 31 also accepts modifications to the list of objects of interest displayed on the display unit 12, and stores the information of the modified list of objects of interest in the object of interest information storage unit 21.

[0053] The display processing unit 32 displays various information on the display unit 12. For example, the display processing unit 32 displays information from the list of objects of interest generated by the object of interest identification unit 33 on the display unit 12 in a modifiable manner. The display processing unit 32 also displays a response statement to a task instruction statement on the display unit 12 based on the response statement data generated by the generation unit 37.

[0054] Furthermore, the display processing unit 32 displays the image shown in the input image data, the task instruction text, and the input frame for the list of items of interest on the display unit 12. This allows the user to input the list of items of interest using the input unit 11 while referring to the image shown in the input image data and the task instruction text.

[0055] Figure 7 shows an example of a focus list editing screen displayed on the display unit 12 by the display processing unit 32 in the processing unit 14 of the information processing device 1 according to the embodiment. The focus list editing screen 80 shown in Figure 7 includes focus list related information 81, a focus add button 82, and a focus delete button 83.

[0056] The list of items of interest related information 81 includes a combination of item of interest, scene number, and extraction type for each item of interest. The scene number is the number of the scene detected by the item of interest identification unit 33. The extraction type is "automatic" when extracted by the item of interest identification unit 33, and "manual" when extracted by the user.

[0057] The "Add Focus" button 82 is used to add a focus item to the focus item list. When the user presses the "Add Focus" button 82 by operating the input unit 11, the display processing unit 32 adds a new row to the focus item list related information 81. The user can then add a focus item or other information to the row added to the focus item list related information 81 by operating the input unit 11.

[0058] The "Delete Focused Item" button 83 is used to remove a focused item from the focused item list. When a user operates the input unit 11 to select a row in the focused item list related information 81 and then presses the "Delete Focused Item" button 83, the display processing unit 32 deletes the row selected by the user from the focused item list related information 81.

[0059] Returning to Figure 6, let's continue the explanation of the processing unit 14. The object of interest identification unit 33 of the processing unit 14 identifies the object of interest using MLLM from information including the data of the task instruction statement and the keyframe data extracted from the input image data. The object of interest identification unit 33 may have its own MLLM, or it may use an MLLM provided by an external device via an API.

[0060] Figure 8 is a schematic diagram illustrating the general configuration of the object of interest identification unit 33 in the processing unit 14 of the information processing device 1 according to the embodiment. As shown in Figure 8, the object of interest identification unit 33 includes a scene change detection unit 40, a keyframe extraction unit 41, an instruction text creation unit 42, a text tokenization unit 43, an image tokenization unit 44, a projection unit 45, and an identification processing unit 46.

[0061] The scene change detection unit 40 detects a scene change in the video shown in the input image data if the setting information includes information indicating whether keyframe data for each scene has been extracted. Scene change detection is performed using known techniques such as frame difference detection, histogram comparison detection, or machine learning model detection, but is not limited to these examples.

[0062] The keyframe extraction unit 41 extracts keyframe data, which is still image data of a keyframe. A keyframe is, for example, the first frame, but it may also be a frame specified by an operator (not shown). Furthermore, if a scene change is detected by the scene change detection unit 40, the keyframe extraction unit 41 extracts keyframe data for each scene of the moving image indicated by the input image data.

[0063] The instruction creation unit 42 creates instruction data for extracting a target of interest that is suitable for causing the MLLM to extract the target of interest, based on, for example, the task instruction data acquired by the acquisition unit 30 and the template data stored in the template storage unit 20.

[0064] For example, the instruction creation unit 42 creates instruction data for extracting a target of interest by embedding task instruction texts into a template. The template is, for example, the information shown in Figure 2(b), and the instruction text for extracting a target of interest is, for example, the information shown in Figure 2(c), but the system is not limited to these examples.

[0065] The text tokenization unit 43 performs a tokenization process that converts the data of the instruction statement for extracting the target of interest, created by the instruction statement creation unit 42, into multiple tokens. For example, the text tokenization unit 43 uses a tokenizer for MLLM to convert the data of the instruction statement for extracting the target of interest into multiple tokens.

[0066] The image tokenization unit 44 performs tokenization processing to convert the image data of the keyframe data acquired by the keyframe extraction unit 41 into multiple tokens. For example, the image tokenization unit 44 divides the image shown in the keyframe data into multiple regions and converts the image of each of the multiple regions obtained by the division into a token containing image features (feature vectors). The image tokenization unit 44 performs tokenization processing of image data using, for example, an image encoder (e.g., CLIP (Contrastive Language-Image Pre-Training) vision encoder), but is not limited to this example.

[0067] The projection unit 45 performs a projection process that projects each of the multiple tokens obtained by the tokenization process of the image tokenization unit 44 onto the language space, converting the multiple tokens obtained by the tokenization process of the image tokenization unit 44 into multiple tokens in a format that can be handled by MLLM. For example, the image tokenization unit 44 may use a model such as a multimodal projector including a linear layer and a GELU (Gaussian Error Linear Unit) function to convert the multiple tokens obtained by the tokenization process of the image tokenization unit 44 into multiple tokens in a format that can be handled by MLLM, but is not limited to this example.

[0068] The specific processing unit 46 inputs a plurality of tokens obtained by the tokenization process of the text tokenization unit 43 and a plurality of tokens obtained by the projection process of the projection unit 45 into the MLLM, and acquires information about the object of interest output from the MLLM.

[0069] Furthermore, the focus identification unit 33 can, for example, extract nouns from the task instruction text by morphological analysis and identify such nouns as information of the focus. In this case, keyframe data does not necessarily have to be used to identify the focus.

[0070] Returning to Figure 6, let's continue the explanation of the processing unit 14. The token reduction processing unit 34 of the processing unit 14 performs a token reduction process that generates data on which token reduction has been performed using the information of the object of interest corresponding to the task instruction statement.

[0071] Figure 9 is a schematic diagram illustrating the general configuration of the token reduction processing unit 34 in the processing unit 14 of the information processing device 1 according to the embodiment. As shown in Figure 9, the token reduction processing unit 34 includes an extraction unit 50, a preprocessing unit 51, a target coordinate detection unit 52, an image tokenization unit 53, and a token reduction unit 54.

[0072] The extraction unit 50 extracts still image data from multiple frames from the input image data if the input image data is moving image data. The still image data from each frame is an example of the target data.

[0073] The preprocessing unit 51 performs preprocessing on the still image data of each frame extracted by the extraction unit 50. The preprocessing includes resizing and normalization. The preprocessing unit 51 resizes the still image data to a predetermined image size and normalizes the resized still image data. The normalization of the still image data is performed, for example, using the mean value and standard deviation for each RGB (Red, Green, Blue) channel, but is not limited to this example.

[0074] The object of interest coordinate detection unit 52 detects the coordinates of the object of interest included in the image shown by the image data, based on the preprocessed frame data, which is still image data that has been preprocessed by the preprocessing unit 51, and the information in the object of interest list, which includes information on the object of interest identified by the object of interest identification unit 33. The object of interest coordinate detection unit 52 extracts the coordinates of the object of interest using, for example, an OVOD (Open-Vocabulary Object Detection) model or a semantic segmentation model.

[0075] The image tokenization unit 53 performs a tokenization process to generate tokens containing the image features of each region (patch) of a still image shown in the preprocessed frame data. The image tokenization unit 53 performs the tokenization process using, for example, an image encoder (e.g., CLIP vision encoder), but is not limited to this example.

[0076] The token reduction unit 54 includes a selection processing unit 60 and a merge processing unit 61. The token reduction unit 54 performs a token reduction process to select tokens from among a plurality of tokens generated by the image tokenization unit 53 that include at least a portion of the coordinates of the object of interest detected by the object of interest coordinate detection unit 52. The tokens selected in the token reduction process are an example of predetermined data.

[0077] The token reduction unit 54 has, for example, token position information that associates the position index of a token with its coordinates, and obtains the position index of the token containing the object of interest from this token position information. The token reduction unit 54 then selects the token corresponding to the coordinates of the object of interest by selecting the token corresponding to the position index of the token containing the object of interest. Tokens other than the token corresponding to the coordinates of the object of interest are not selected, thereby reducing the number of tokens.

[0078] The merge processing unit 61 performs a merge operation if the configuration information includes information indicating that tokens should be merged. The merge operation is the process of merging multiple tokens of the same interest into a single token. This allows the merge processing unit 61 to reduce the number of tokens.

[0079] Furthermore, the token reduction unit 54 may choose not to perform token reduction based on, for example, the configuration information. For example, the configuration information may include information indicating token reduction conditions, such as the image not being unclear or the presence of a subject of interest in the image. In this case, the token reduction unit 54 will not perform token reduction if the image is unclear or if the subject of interest does not exist in the image. For example, the token reduction unit 54 will not perform token reduction if the subject of interest is not extracted by the subject of interest identification unit 33.

[0080] Whether or not an image is unclear is determined, for example, by the focus identification unit 33. In the focus identification unit 33, the instruction statement creation unit 42 includes a sentence instructing the identification processing unit 46 to determine whether or not the image is unclear in the instruction statement for extracting the focus object, and causes the identification processing unit 46 to output information indicating whether or not the image is unclear. The token reduction unit 54 does not perform token reduction if the focus identification unit 33 determines that the image is unclear.

[0081] The determination of whether an image is blurry or not is not limited to the examples described above. For example, whether an image is blurry or not may be determined by detecting the amount of change in the edges of the image and comparing the variance of such change with a threshold, or by using a deep learning model that determines whether an image is blurry or not. Such determination may be performed, for example, by the token reduction unit 54 or by the object of interest identification unit 33.

[0082] Returning to Figure 6, let's continue the explanation of the processing unit 14. The projection unit 35 of the processing unit 14 performs a projection process that projects the tokens output from the token reduction processing unit 34 onto the language space, converting the tokens output from the token reduction processing unit 34 into tokens that can be handled by MLLM. The projection unit 35 performs token conversion using a model such as a multimodal projector that includes a linear layer and a GELU function, but is not limited to such an example.

[0083] The text tokenization unit 36 ​​performs a tokenization process that converts the data of the task instruction statement into multiple tokens. For example, the text tokenization unit 36 ​​uses a tokenizer for MLLM to convert the data of the task instruction statement into multiple tokens.

[0084] The generation unit 37 performs an input process to input information into the MLLM that includes tokens obtained from the projection process in the projection unit 35 and multiple tokens obtained from the tokenization process in the text tokenization unit 36. The generation unit 37 then performs an acquisition process to obtain response sentence data output from the MLLM based on the input information. The tokens obtained from the projection process in the projection unit 35 are an example of predetermined data, which is data on which token reduction has been performed using information of the object of interest corresponding to the instruction sentence. The response sentence data is an example of information related to predetermined data.

[0085] The MLLM may be included in the generation unit 37, or it may be used via the API. Furthermore, the MLLM used in the generation unit 37 is different from the MLLM used in the target of interest identification unit 33, but is not limited to this example.

[0086] The generation unit 37 performs input processing to input each frame data into the MLLM. The generation unit 37 can also perform input processing on a unit of multiple frame data. For example, the generation unit 37 can perform input processing to input the tokens for each of the multiple frame data obtained through the token reduction processing unit 34 and the projection unit 35 into the MLLM. Input processing on a unit of multiple frame data may also be performed on a scene-by-scene basis.

[0087] The output unit 38 outputs the response sentence data obtained by the generation unit 37. For example, the output unit 38 outputs the response sentence data obtained by the generation unit 37 to an external device via the communication unit 10. The display processing unit 32 can also display the response sentence on the display unit 12 based on the response sentence data obtained by the generation unit 37.

[0088] <Processing by the Processing Unit 14 of the Information Processing Device 1> Next, the procedure for information processing by the Processing Unit 14 of the Information Processing Device 1 according to the embodiment will be described. Figure 10 is a flowchart showing an example of information processing by the Processing Unit 14 of the Information Processing Device 1 according to the embodiment.

[0089] As shown in Figure 10, the processing unit 14 of the information processing device 1 determines whether or not the timing for identifying the object of interest has arrived (step S50). The timing for identifying the object of interest is, for example, the timing when the data of the task instruction statement and the input image data are acquired, or the timing when an instruction is entered by the user, but is not limited to these examples.

[0090] If the processing unit 14 determines that it is time to identify the object of interest (step S50: Yes), it performs the object of interest identification process (step S51). The object of interest identification process in step S51 is the process shown in steps S60 to S66 in Figure 11, which will be described in detail later.

[0091] If the processing unit 14 has completed the processing in step S51, or if it has determined that the timing for identifying the object of interest has not yet arrived (step S50: No), it determines whether or not the timing for generating a response has arrived (step S52). The timing for generating a response is, for example, the timing when the processing for identifying the object of interest has completed, or the timing when an instruction is input by the user, but is not limited to these examples.

[0092] If the processing unit 14 determines that it is time to generate a response (step S52: Yes), it performs the response generation process (step S53). The response generation process in step S53 is the process shown in steps S70 to S77 in Figure 12, which will be described in detail later.

[0093] The processing unit 14 outputs the result of the response generation process in step S53 (step S54). The result of the response generation process in step S53 is, for example, the data of the response statement described above.

[0094] If the processing in step S54 is completed, or if the processing unit 14 determines that it is not time for response generation (step S52: No), it determines whether it is time to terminate operation (step S55). The processing unit 14 determines that it is time to terminate operation, for example, when the power to the information processing device 1 is turned off.

[0095] If the processing unit 14 determines that it is not yet time to terminate the operation (step S55: No), it proceeds to step S50. If it determines that it is time to terminate the operation (step S55: Yes), it terminates the process shown in Figure 10.

[0096] Figure 11 is a flowchart showing an example of the object identification process performed by the processing unit 14 of the information processing device 1 according to the embodiment. As shown in Figure 11, the processing unit 14 extracts keyframe data from the input image data (step S60). In step S60, the processing unit 14 can extract keyframe data for each scene.

[0097] Next, the processing unit 14 creates data for the instruction statement for extracting the object of interest based on the task instruction statement (step S61), and tokenizes the created data for the instruction statement for extracting the object of interest (step S62). The processing unit 14 also tokenizes the keyframe data (step S63), and converts the tokens obtained from the tokenization in step S63 into tokens in the language space (step S64).

[0098] Next, the processing unit 14 inputs the token corresponding to the instruction data for extracting the object of interest obtained in step S62 and the token converted in step S64 into the MLLM (step S65). Then, the processing unit 14 obtains the information of the object of interest from the MLLM (step S66) and terminates the process shown in Figure 11. Note that the processing in step S62 and the processing in steps S63 and S64 may be performed in parallel, or the processing in steps S63 and S64 may precede the processing in step S62.

[0099] Figure 12 is a flowchart showing an example of the response generation process by the processing unit 14 of the information processing device 1 according to the embodiment. As shown in Figure 12, the processing unit 14 extracts multiple frame data from the input image data (step S70). The processing unit 14 also tokenizes the task instruction statement (step S71).

[0100] Next, the processing unit 14 selects the frame data with the earliest playback position from among the unselected frame data (step S72). Then, the processing unit 14 performs a token reduction process (step S73). The token reduction process in step S73 is the process shown in steps S80 to S85 in Figure 13, which will be described in detail later.

[0101] Next, the processing unit 14 projects the tokens obtained in the token reduction process of step S73 onto the language space (step S74). The processing unit 14 inputs the tokens obtained from the projection onto the language space and the tokens corresponding to the task instruction into the MLLM (step S75). Then, the processing unit 14 obtains information about the image indicated by the frame data from the MLLM, which corresponds to the task instruction (step S76).

[0102] In the example shown in Figure 12, the processing in step S75 is performed on a frame data basis, but the example is not limited to this, and the processing in step S75 can also be performed on multiple frame data units.

[0103] Next, the processing unit 14 determines whether or not there is unselected frame data (step S77). If the processing unit 14 determines that there is unselected frame data (step S77: Yes), it proceeds to step S72. If it determines that there is no unselected frame data (step S77: No), it terminates the process shown in Figure 12.

[0104] Figure 13 is a flowchart showing an example of token reduction processing by the processing unit 14 of the information processing device 1 according to the embodiment. As shown in Figure 13, the processing unit 14 performs preprocessing on the frame data (step S80).

[0105] Next, the processing unit 14 extracts and tokenizes image features from each region of the image shown in the preprocessed frame data (step S81). The processing unit 14 also detects the coordinates of the object of interest in the image from the preprocessed frame data and the list of objects of interest (step S82). Steps S81 and S82 are performed in parallel, but step S82 may be performed first.

[0106] Next, the processing unit 14 identifies the position index of the token corresponding to the object of interest based on the coordinates of the object of interest detected in step S82 (step S83). Based on the position index identified in step S83, the processing unit 14 selects the token corresponding to the object of interest from the multiple tokens obtained in step S81, and excludes the other tokens (step S84). As a result, tokens other than the token corresponding to the object of interest are removed from the multiple tokens obtained in step S81.

[0107] Next, if there are multiple tokens of the same object of interest, the processing unit 14 performs a process to merge the multiple tokens of the same object of interest into a single token for each object of interest (step S85), and then terminates the process shown in Figure 13.

[0108] <Variation> In the example described above, the data input to the information processing device 1 was described as moving image data, but the data input to the information processing device 1 may also be still image data.

[0109] Furthermore, while the above example uses a template to create an instruction for the object of interest, the system is not limited to this example. For example, the instruction creation unit 42 can input information including a task instruction and a creation instruction that instructs the creation of an instruction for the object of interest based on the task instruction into the LLM, and have the LLM output an instruction for the object of interest.

[0110] Furthermore, although the above example described an example where the input data is image data, the system is not limited to such examples. For example, the input data may be state variable distribution data (or time-series data of state variable distribution data) that shows the distribution of state variables, such as temperature distribution data showing temperature distribution or rainfall distribution data showing rainfall distribution. State variable distribution data is, for example, data in the form of black and white image data, for example.

[0111] In this case, the focus identification unit 33 can select a region that is in the state indicated by the task instruction (for example, a region where the temperature is above a threshold) as the focus. The token reduction processing unit 34 can then perform token reduction by tokenizing the state quantity distribution data, selecting tokens containing the focus from among the multiple tasks obtained through tokenization, and not selecting tokens that do not contain the focus.

[0112] Furthermore, while the above example uses MLLM as an example of LLM, the system is not limited to this example, and LLMs that do not support multimodal processing can also be used. For example, suppose the input data is structured data such as a CSV file. In this case, the focus identification unit 33 inputs the structured data and the instruction statement for extracting the focus into the LLM and obtains information about the focus from the LLM. The token reduction processing unit 34 can delete tokens by identifying the rows in the structured data that contain the focus, selecting the tokens corresponding to the identified rows, and not selecting tokens corresponding to rows other than the identified row. If the rows are separated by newline marks, the token reduction processing unit 34 can identify the rows that contain the focus by counting the number of newline marks. The token reduction processing unit 34 can also identify the rows that contain the focus by associating an index for each row with a token.

[0113] <Effects of the Embodiment> The information processing method according to the embodiment is an information processing method executed by a computer, which includes inputting information including predetermined data and task instruction data that instructs the output of information related to the predetermined data into a large-scale language model, and obtaining information about the predetermined data output from the large-scale language model based on the input information. The predetermined data is data on which token reduction has been performed using information of interest corresponding to the task instruction. In the token-reduced data, tokens that are not necessary to perform the task indicated in the task instruction have been accurately reduced. Therefore, the information processing method according to the embodiment can suppress the decrease in accuracy due to token reduction compared to a technique that extracts representative tokens in an image and integrates the remaining tokens based on similarity.

[0114] Furthermore, the information processing method according to the embodiment includes a process that identifies a target of interest from information including target data, which is the data subject to the task instruction, and instruction data, using a large-scale language model, and generates predetermined data by deleting some of the multiple tokens corresponding to the target data based on the target data, which is the data subject to the task instruction, and the information of the identified target of interest. As a result, the information processing method according to the embodiment can generate token-reduced data in a way that suppresses the decrease in accuracy due to token reduction.

[0115] Furthermore, the target data is image data, the large-scale language model is a multimodal large-scale language model, and the process of deleting some tokens includes generating tokens that represent image features in each region of the image, detecting the coordinates of the object of interest included in the image based on the image data and information about the object of interest, and selecting the token corresponding to the coordinates of the object of interest from among the generated multiple tokens. As a result, in the information processing method according to the embodiment, when extracting information corresponding to a task instruction from an image, the decrease in accuracy due to token reduction can be suppressed.

[0116] <System Configuration, etc.> Furthermore, the components of each part shown in the diagram are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown in the diagram, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. In addition, all or any part of the processing functions performed by each device can be realized by a CPU and the program executed on that CPU, or by hardware using wired logic.

[0117] Furthermore, among the processes described in the embodiments above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0118] <Program> It is also possible to create a program in which the processing performed by the information processing device 1 according to the above embodiment is written in a language that can be executed by a computer. In one embodiment, the information processing device 1 can be implemented by installing an information processing program that performs the above information processing as package software or online software on a desired computer. For example, by having the information processing device execute the above information processing program, the information processing device can be made to function as the information processing device 1. The information processing device referred to here is a general-purpose computer, but is not limited to this example, and may be a general-purpose server (for example, an on-premise server or a cloud server), or any other computer.

[0119] Figure 14 shows an example of a computer that executes an information processing program. Computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070. These components are connected by a bus 1080.

[0120] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. A removable storage medium, such as a magnetic disk or optical disk, is inserted into the disk drive 1041. A serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052. A video adapter 1060 is connected to, for example, a display 1061.

[0121] Here, the hard disk drive 1031 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. The information described in the above embodiment is stored, for example, in the hard disk drive 1031 or the memory 1010.

[0122] Furthermore, the information processing program is stored in the hard disk drive 1031 as a program module 1093 containing instructions to be executed by the computer 1000, for example. Specifically, the program module 1093 containing instructions for each process executed by the information processing device 1 described in the above embodiment is stored in the hard disk drive 1031.

[0123] Furthermore, the data used for information processing by the information processing program is stored as program data 1094, for example, in the hard disk drive 1031. The CPU 1020 then reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as needed and executes the procedures described above.

[0124] Furthermore, the program module 1093 and program data 1094 related to the information processing program are not limited to being stored in the hard disk drive 1031; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 related to the information processing program may be stored in another computer connected via a network such as a LAN or WAN and read by the CPU 1020 via a network interface 1070.

[0125] The functions of the elements disclosed herein may be implemented using circuits or processing circuitry that include general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), conventional circuits, and / or combinations thereof that are programmed using one or more programs stored in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered processing circuitry or circuitry because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions.

[0126] There is a memory for storing a computer program that includes computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to perform the methods disclosed herein. This computer program can be implemented in commonly known forms, such as computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs and DVDs, and / or memory in FPGAs and ASICs.

[0127] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention.

[0128] 1 Information Processing Device 10 Communication Unit 11 Input Unit 12 Display Unit 13 Storage Unit 14 Processing Unit 20 Template Storage Unit 21 Target Information Storage Unit 22 Setting Information Storage Unit 30 Acquisition Unit 31 Reception Unit 32 Display Processing Unit 33 Target Identification Unit 34 Token Reduction Processing Unit 35, 45 Projection Unit 36, 43 Text Tokenization Unit 37 Generation Unit 38 Output Unit 40 Scene Change Detection Unit 41 Keyframe Extraction Unit 42 Instruction Text Creation Unit 44, 53 Image Tokenization Unit 46 Identification Processing Unit 50 Extraction Unit 51 Preprocessing Unit 52 Target Coordinate Detection Unit 54 Token Reduction Unit 60 Selection Processing Unit 61 Merge Processing Unit

Claims

1. An information processing method performed by a computer, characterized in that information including predetermined data and instruction data that instructs the output of information related to the predetermined data is input to a large-scale language model, information regarding the predetermined data output from the large-scale language model based on the input information is obtained, and the predetermined data is data on which token reduction has been performed using information of interest corresponding to the instruction.

2. The information processing method according to claim 1, characterized in that, using a large-scale language model, the information includes target data which is the data subject to the instruction statement and the data of the instruction statement, the information includes identifying the object of interest, and based on the target data and the information of the identified object of interest, the information includes deleting some of the tokens among a plurality of tokens corresponding to the target data to generate the predetermined data.

3. The information processing method according to claim 2, wherein the target data is image data, the large-scale language model is a multimodal large-scale language model, and the deletion of some tokens includes generating tokens that represent image features in each region of the image, detecting the coordinates of the object of interest included in the image based on the image data and the information of the object of interest, and selecting a token from the generated plurality of tokens that corresponds to the coordinates of the object of interest.

4. An information processing device comprising a generation unit that inputs information including predetermined data and instruction data that instructs the output of information related to the predetermined data into a large-scale language model, and acquires information regarding the predetermined data output from the large-scale language model based on the input information, wherein the predetermined data is data on which token reduction has been performed using information of interest corresponding to the instruction.

Images