Information processing method, information processing system, and computer program

Abstract

Provided is an information processing method that generates information by using a trained model.　The information processing method includes a data acquisition step for acquiring input data, a biometric information acquisition step for acquiring biometric information of a user, a preprocessing step for preprocessing the input data on the basis of the biometric information, and a generation step for generating, on the basis of the preprocessed input data, information to be provided to the user. In the preprocessing step, a region of interest is identified from the input data on the basis of the biometric information, and the input data is preprocessed so as to include more information from the region of interest or to reduce the amount of information from areas other than the region of interest.

Description

Information Processing Method, Information Processing System, and Computer Program

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing method and an information processing system that utilize a learned model, and a computer program.

[0002] Large language models (LLMs) form the basis of natural language processing and support many advancements in human language understanding and generation. For example, there have been proposed video generation systems that use generative AI models such as large language models to generate scenarios, keyword and text information, and logo information necessary for sound generation, or to determine the font, size, and position of text display (see Patent Document 1).

[0003] Japanese Patent No. 7578209

[0004] An object of the present disclosure is to provide an information processing method, an information processing system, and a computer program that generate information using a learned model.

[0005] The present disclosure has been made in consideration of the above problems, and a first aspect thereof is an information processing method having: a data acquisition step of acquiring input data; a biological information acquisition step of acquiring biological information of a user; a preprocessing step of preprocessing the input data based on the biological information; and a generation step of generating information to be provided to the user based on the preprocessed input data.

[0006] In the data acquisition step, a first-person perspective video of the user is acquired. In the biological information acquisition step, gaze information of the user is acquired. In the preprocessing step, based on the gaze information, a target area is specified in the panoramic video of the first-person perspective video, and the first-person perspective video is preprocessed so as to include more information of the target area or reduce the amount of information outside the target area.

[0007] More specifically, in the preprocessing step, a gaze video consisting of a focus area centered on the user's gaze is extracted from a full-view video of a first-person perspective video, and in the generation step, information to be provided to the user is generated based on the gaze video. Then, in the generation step, the gaze video is encoded into a token sequence using an image encoder, and information to be provided to the user is generated from the token sequence using a multimodal large-scale language model.

[0008] In the aforementioned preprocessing step, preprocessing is performed to reduce the number of tokens input to the multimodal large-scale language model while maintaining the understanding level of the multimodal large-scale language model.

[0009] Furthermore, the second aspect of this disclosure is an information processing system comprising: a data acquisition unit for acquiring input data; a biometric information acquisition unit for acquiring the user's biometric information; a preprocessing unit for preprocessing the input data based on the biometric information; and a generation unit for generating information to be provided to the user based on the preprocessed input data.

[0010] However, the term "system" as used here refers to a logical collection of multiple devices (or functional modules that perform specific functions), without regard to whether each device or functional module resides within a single enclosure. In other words, both a single device consisting of multiple components or functional modules, and a collection of multiple devices, qualify as a "system."

[0011] Furthermore, a third aspect of this disclosure is a computer program written in a computer-readable format to cause the computer to function as a data acquisition unit that acquires input data, a biometric information acquisition unit that acquires the user's biometric information, a preprocessing unit that preprocesses the input data based on the biometric information, and a generation unit that generates information to be provided to the user based on the preprocessed input data.

[0012] The computer program relating to the third aspect of this disclosure defines a computer program written in a computer-readable format to perform a predetermined process on a computer. The computer program can be provided to a computer capable of executing various program codes via a storage medium, communication medium, such as an optical disk, magnetic disk, semiconductor memory, or a network, in a computer-readable format. By installing the computer program relating to the third aspect of this disclosure onto a computer via any of these media, collaborative effects can be achieved on the computer, similar to the effects of the information processing system relating to the second aspect of this disclosure.

[0013] Figure 1 is a diagram showing the functional configuration of the information processing system 100 according to this disclosure. Figure 2 is a flowchart showing the processing procedure executed by the information processing system 100. Figure 3 is a diagram showing the data flow when the information processing system 100 processes a user's first-person perspective video. Figure 4 is a diagram showing an example of operation in which the information processing system 100 is used to generate a video description. Figure 5 is a diagram showing an example of operation in which the information processing system 100 is used to generate a video description. Figure 6 is a diagram showing an example of operation in which the information processing system 100 is used to generate a video description. Figure 7 is a diagram showing an example of operation in which the information processing system 100 is used to generate a video description. Figure 8 is a diagram showing an example of operation of the "Stop and Ask" approach of the information processing system 100. Figure 9 is a diagram showing an example of operation of the "Stop and Ask" approach of the information processing system 100. Figure 10 is a diagram illustrating a panoramic video and a viewpoint video and a center video extracted from this panoramic video, respectively. Figure 11 is a diagram showing an example of prompts used to generate a description of an evaluation video. Figure 12 is an example of prompts used when numerically evaluating the information processing system 100 using LLM. Figure 13 is a diagram showing the results of numerical evaluation of the accuracy of video descriptions generated by the multimodal large-scale language model from each video type (when BLEU is used). Figure 14 is a diagram showing the results of numerical evaluation of the accuracy of video descriptions generated by the multimodal large-scale language model from each video type (when ROUGE is used). Figure 15 is a diagram showing the results of numerical evaluation of the accuracy of video descriptions generated by the multimodal large-scale language model from each video type (when SBERT is used). Figure 16 is a diagram showing the results of numerical evaluation of the accuracy of video descriptions generated by the multimodal large-scale language model from each video type (when LLM is used). Figure 17 is a diagram showing the results of human evaluation of video descriptions generated by the multimodal large-scale language model from each video type. Figure 18 is a diagram showing the relationship between the length of the video and the length of the video description generated from the video by the multimodal large-scale language model. Figure 19 is a diagram illustrating a first modification of this disclosure. Figure 20 is a diagram illustrating a second modification of the present disclosure.Figure 21 is a diagram illustrating a third modification of the present disclosure. Figure 22 is a diagram showing an example of the hardware configuration of an information processing device. Figure 23 is a diagram showing a user wearing smart glasses on their head and controllers on both hands. Figure 24 is a diagram showing a user wearing smart glasses on their head. Figure 25 is a diagram showing an example of the configuration of an information processing system 100 to which smart glasses are applied. Figure 26 is a diagram showing an example of the operation of a multimodal large-scale language model.

[0014] Hereinafter, embodiments of the present disclosure will be described in the following order with reference to the drawings.

[0015] A. Overview B. Functional Configuration C. Related Technologies D. Examples D-1. First Example D-2. Second Example E. Evaluation E-1. Dataset E-2. Evaluation Results Based on Numerical Indicators E-3. Human Evaluation Results E-4. Summary of Evaluation Results F. Modifications F-1. First Modification F-2. Second Modification F-3. Third Modification G. Information Processing Device Configuration G-1. First Configuration Example G-2. Second Configuration Example H. Conclusion

[0016] A. Overview Large-scale language models are evolving into multimodal large-scale language models that can process not only text but also various modalities such as images, audio, and video. Multimodal large-scale language models integrate multisensory perception that depends on visual, auditory, and other sensory modes, and reflect the complex information processing methods of humans, thus representing a major leap towards Artificial General Intelligence (AGI).

[0017] Multimodal large-scale language models, by processing time-series information such as images, videos, and audio, offer promising potential for understanding human activity through video and audio, enabling numerous human-computer interaction applications, including supporting human activity, real-world agents, and skill transfer to robots and other individuals. From a human-computer interaction (HCI) perspective, multimodal large-scale language models also hold great potential. If such models can perceive the world in a similar way to humans, a wide range of applications become possible. These applications include technologies such as recording, understanding, and transmitting skilled human actions to others, evaluating skill development, recognizing real-world behavior to provide personalized assistance, and extending sensory perception of the environment to assist people with disabilities.

[0018] Figure 26 schematically illustrates an example of the operation of a multimodal large-scale language model. The multimodal large-scale language model 2600 has encoders on the input side that encode the input data for each modal into a latent representation. Each encoder (text encoder 2601, audio encoder 2602, image encoder 2603, etc.) encodes the input data of text, audio, and video modals into token sequences 2611 to 2613, which consist of a set of feature vectors (tokens) on a latent space shared by all modals. The multimodal large-scale language model 2600 then takes the input data of each modal in the form of a token sequence, integrates it on the shared latent space, performs processing (inference, etc.), and outputs generated data 2621. The generated data 2621 is, for example, text information that describes the input data (text, audio, or video). Note that the multimodal large-scale language model 2600 does not need to process data from multiple modals simultaneously; it only needs to be equipped with encoders corresponding to the input modals. In this embodiment, since only video is processed with a multimodal large-scale language model, only an image encoder is used.

[0019] However, the computational load of multimodal large-scale language models remains a significant challenge. Generally, the computational load of a multimodal large-scale language model increases proportionally to the square of the number of input tokens, leading to increased processing time and memory demands. This problem is particularly severe when processing image or video data.

[0020] For example, Vision Transformers (ViT), a major image encoder used in multimodal large-scale language models, processes video data by dividing the image into a two-dimensional grid frame by frame and assigning a token to each grid. When high-resolution, long-duration videos are input, the number of tokens generated by ViT increases, leading to an excessive computational load on the multimodal large-scale language model. While it is possible to reduce the number of tokens and thus memory usage by lowering the video resolution or downsampling the frame rate, this comes at the cost of a decrease in the multimodal large-scale language model's understanding (how accurately it understands the input data and how well it generates appropriate output). For example, tasks such as reading text on signs in front of a user require sufficient image resolution for OCR (Optical Character Recognition) processing.

[0021] On the other hand, human vision does not process all visual information equally. Humans selectively focus on specific areas through eye movements, but peripheral vision information is perceived at low resolution or only responds to movement. This visual attention mechanism allows humans to achieve both accuracy in understanding the world and efficiency in processing visual input.

[0022] This disclosure leverages the visual attention mechanisms naturally present in humans to reduce the processing load of dynamic visual information in multimodal large-scale language models. For example, this disclosure refers to a use case in which a user's first-person perspective video is input into a multimodal large-scale language model, and the multimodal large-scale language model recognizes the content of a task from that first-person perspective video and provides the user with task guidance and answers to questions. Specifically, in this disclosure, when processing a user's first-person perspective video captured with a first-person perspective camera equipped with eye-tracking capabilities into a multimodal large-scale language model, preprocessing is performed to extract only the eye-tracking video that focuses on the user's gaze focus region from the full-resolution panoramic video.

[0023] According to this disclosure, by preprocessing by extracting gaze-tracking video from first-person perspective video, the number of pixels input to the ViT (image encoder) can be reduced to about one-tenth of that of the original full-resolution panoramic video, thereby significantly reducing the number of tokens input to the multimodal large-scale language model and solving the problem of computational load on the multimodal large-scale language model during video processing. Furthermore, according to this disclosure, by selectively processing the region that the user is fixated on within the panoramic video, it is possible to achieve an understanding of the multimodal large-scale language model equivalent to or better than that achieved when processing a full-resolution panoramic image, while significantly reducing the data input to the multimodal large-scale language model.

[0024] Therefore, according to this disclosure, an efficient solution can be provided for interpreting and utilizing human skills using a multimodal large-scale language model. By applying a first-person perspective as input data, the multimodal large-scale language model can accurately understand work procedures in a variety of real-world activities, such as cooking, repair, first aid, and sports.

[0025] The effects described herein are illustrative only, and the effects brought about by this disclosure are not limited to those described herein. Furthermore, this disclosure may produce additional effects beyond those described above. Further objectives, features, and advantages of this disclosure will become apparent from the more detailed descriptions based on the embodiments and accompanying drawings described later.

[0026] B. Functional Configuration Diagram 1 schematically shows the functional configuration of the information processing system 100 according to this disclosure. The information processing system 100 may be, for example, smart glasses or other wearable devices. If it is a wearable device, it is easy to estimate the task being performed based on information directly acquired through the user while the task is being performed, and to generate and provide data to the user that supports the performance of that task. Of course, the information processing system 100 may be configured as a combination of multiple physically independent devices.

[0027] The information processing system 100 shown in Figure 1 comprises a data acquisition unit 101, a biological information acquisition unit 102, a preprocessing unit 103, and a generation unit 104. Each unit will be described below.

[0028] The data acquisition unit 101 acquires data that will serve as the source for input data to the multimodal large-scale language model, which acts as the generation unit 104. For example, if the input to the multimodal large-scale language model is a first-person perspective video of the user, the data acquisition unit 101 may be a first-person perspective camera worn on the user's head, or it may be equipped with an interface to receive video footage from the first-person perspective camera. The data acquisition unit 101 may acquire live first-person perspective video footage being captured by the first-person perspective camera, or it may acquire recorded and saved first-person perspective video footage. Furthermore, if the multimodal large-scale language model, which acts as the generation unit 104, processes modal data other than images and videos, such as text and audio, the data acquisition unit 101 may similarly acquire modal data other than videos.

[0029] The biometric information acquisition unit 102 acquires the user's biometric information. The acquired biometric information is used in the subsequent preprocessing unit 103 to identify areas of interest from the data acquired by the data acquisition unit 101. For example, if the data acquisition unit 101 acquires a first-person perspective video of the user, and the MLLM, acting as the generation unit 104, processes the first-person perspective video, the biometric information acquisition unit 102 acquires the user's gaze information as the user's biometric information. Specifically, the biometric information acquisition unit 102 uses an infrared camera to track eye movements and calculates the direction of the gaze by detecting the reflection of the pupil and iris. Of course, the biometric information acquisition unit 102 may acquire the user's gaze information by other means.

[0030] The preprocessing unit 103 performs preprocessing to reduce the number of tokens input to the multimodal large-scale language model used by the generation unit 104, while maintaining the model's level of understanding.

[0031] To maintain the understanding of the multimodal large-scale language model used in the generation unit 104, the preprocessing unit 103 identifies important and noteworthy areas of interest from the data acquired by the data acquisition unit 101, based on the user's biometric information acquired by the biometric information acquisition unit 102. For example, when a user's first-person perspective video is to be processed by the multimodal large-scale language model, the preprocessing unit 103 identifies a predetermined range centered on the user's gaze within the original first-person perspective panoramic video as the area of ​​interest (also referred to as the "gaze video" in this specification). In this specification, the area of ​​interest may refer not only to a spatial area, such as a part of an image frame, but also to a specific temporal area of ​​time-series data.

[0032] The preprocessor 103 may first estimate the task the user will perform based on the biometric information acquired by the biometric information acquisition unit 102, and then identify the area of ​​interest corresponding to that task from the data acquired by the data acquisition unit 101. The preprocessor 103 may also use movements of specific body parts, such as the user's hands, in addition to the user's gaze information to estimate the task the user is performing. The preprocessor 103 may also use a combination of multiple types of biometric information to estimate the task. To obtain biometric information other than gaze information, such as the position information of the user's hands, the biometric information acquisition unit 102 may further include a camera that captures the movements of the user's hands and body, or an IMU (Internal Measurement Unit) attached to the user's body. Furthermore, the preprocessor 103 may use an AI model trained to estimate a task from one or more pieces of biometric information to estimate the task from the biometric information acquired by the biometric information acquisition unit 102, or it may use an AI model trained to estimate the area of ​​interest from a first-person perspective video based on the task to identify the area of ​​interest.

[0033] The preprocessing unit 103 then preprocesses the original data acquired by the data acquisition unit 101 to preserve (or include more of) information about the areas of interest (spatial and temporal), or to reduce the amount of information outside of the areas of interest, thereby constructing the input data for the generation unit 104. In short, the preprocessing performed by the preprocessing unit 103 is a process that significantly reduces the amount of data from the original data while maintaining the understanding of the multimodal large-scale language model by preserving information about the areas of interest that are important in the original data.

[0034] For example, if the data acquired by the data acquisition unit 101 is a first-person perspective video of the user, the preprocessing unit 103 cuts out the eye-line video identified based on the user's gaze information from the original panoramic video and constructs the input data for the generation unit 104. Alternatively, the preprocessing unit 103 samples the area near the eye line from the original panoramic video at high resolution, while downsampling a certain range of surrounding areas away from the eye line to a low resolution, and then cuts off the area outside the surrounding area to construct the input data for the generation unit 104.

[0035] The generation unit 104 receives the data that has been preprocessed by the preprocessing unit 103 and generates text information to be provided to the user. This text information includes guidance to support the user in performing tasks and answers to questions from the user. The generation unit 104 may also generate this guidance and answers to questions, and other information to be provided to the user, as data in modalities other than text, such as audio, images, or video.

[0036] The generation unit 104 generates information to be provided to the user using a generation model. That is, the generation model recognizes the tasks to be performed by the user from the input data and generates text that supports the user in performing those tasks. Specifically, the generation model referred to here is a multimodal large-scale language model. In this case, as shown in Figure 26, the generation unit 104 encodes the input data for each modality into a token sequence in the latent space, and then the multimodal large-scale language model integrates and processes the token sequences of each modality. When processing a first-person perspective video of the user, the preprocessing unit 103 extracts only the gaze video from the first-person perspective panoramic video, thereby reducing the number of pixels input to the image encoder (ViT) to about one-tenth of the original full-resolution panoramic video. This significantly reduces the number of tokens input to the multimodal large-scale language model, thus solving the problem of the computational load of the multimodal large-scale language model during video processing.

[0037] The multimodal large-scale language model used in the generation unit 104 is, for example, Gemini 1.5 Pro or Qwen2-VL, but this disclosure is not limited to these. Furthermore, the generation model used by the generation unit 104 is not necessarily limited to a multimodal large-scale language model, and may be, for example, a Video2Text model or an Image2Text model.

[0038] Figure 2 shows the processing procedure executed by the information processing system 100 in flowchart format.

[0039] First, the data acquisition unit 101 acquires data that will serve as the basis for the input data to the multimodal large-scale language model, which will function as the generation unit 104 (step S201). The input data is, for example, a first-person perspective video of a user performing a task. The data acquisition unit 101 acquires live first-person perspective video footage captured by a first-person perspective camera worn on the user's head, or reads a pre-recorded first-person perspective video file from a recording medium at the storage location.

[0040] Furthermore, the biometric information acquisition unit 102 acquires the user's biometric information (step S202). The acquired biometric information is used in the next step S203 to identify a region of interest from the data acquired by the data acquisition unit 101. For example, when the data acquisition unit 101 acquires a video from the user's first-person perspective, the biometric information acquisition unit 102 acquires the user's gaze information (as described above).

[0041] Next, the preprocessing unit 103 constructs input data in a format suitable for processing in the generation unit 104. To this end, the preprocessing unit 103 identifies important and noteworthy areas from the data acquired by the data acquisition unit 101 based on the user's biometric information acquired by the biometric information acquisition unit 102 (step S203). In other words, the preprocessing unit 103 performs preprocessing to reduce the number of tokens input to the multimodal large-scale language model used in the generation unit 104, while maintaining the level of understanding of the model.

[0042] Then, the preprocessing unit 103 constructs the input data for the generation unit 104 so as to reduce the amount of data acquired by the data acquisition unit 101 while preserving information about the area of ​​interest (step S204). For example, if the MLLM as the generation unit 104 is to process a video from the user's first-person perspective, the preprocessing unit 103 cuts out a video of the user's line of sight from the original panoramic video from the first-person perspective, consisting of a predetermined range centered on the user's line of sight, and constructs the input data for the generation unit 104 (as described above).

[0043] The generation unit 104 inputs the data after preprocessing by the preprocessing unit 103 and generates text information to be provided to the user (step S205). The text information mentioned here is guidance for assisting the user in task execution or an answer to a question from the user.

[0044] FIG. 3 schematically shows the data flow when the information processing system 100 processes the first-person perspective video of the user.

[0045] First, the data acquisition unit 101 acquires a first-person perspective video of the user performing a task from a first-person perspective camera or the like. However, the data acquisition unit 101 may read out the pre-recorded and saved first-person perspective video from a recording medium which is the storage location. The first-person perspective video 301 acquired by the data acquisition unit 101 is a panoramic video with full resolution (for example, 1440×1440 pixels). This panoramic video 301 includes a region of interest 301' within a predetermined range centered on the user's line of sight.

[0046] Based on the user's line-of-sight information acquired by the biometric information acquisition unit 102, the preprocessing unit 103 identifies the region of interest 301' from within the panoramic video 301 and constructs a line-of-sight video 302 obtained by cutting out that portion as input data to the generation unit 104. Considering that the number of pixels of the line-of-sight video 302 is only approximately one-tenth of that of the original panoramic video 301, it can be said that the line-of-sight video 302 is an effective information reduction as input to the generation unit 104.

[0047] The generation unit 104 includes an image encoder 311 such as a ViT and a multimodal large language model 312. When the gaze video 302 is input, the image encoder 311 divides the input image into a two-dimensional grid, converts each grid into a feature vector (latent representation) in the latent space, further assigns tokens by vector quantization, and encodes the gaze video 302 into a token sequence 303. Since the number of pixels is reduced to approximately one-tenth from the panoramic video 301 to the gaze video 302 by the preprocessing unit 103, the number of tokens in the token sequence 303 is also significantly reduced compared to the number of tokens when encoding the original panoramic video 301 as it is. In FIG. 3, among the token sequence 303, the tokens reduced due to the reduction in the number of pixels are indicated by dotted lines.

[0048] When the multimodal large language model 312 receives the token sequence 303 as input, it estimates the task that the user is performing and generates text information 304 such as guidance to assist the user in task execution and answers to questions from the user, and presents it to the user. The computational load of the multimodal large language model 312 increases in proportion to the square of the number of tokens, and both the processing time and memory requirements increase. Therefore, by the preprocessing unit 103 narrowing down the panoramic video 301 to the gaze video 302 with approximately one-tenth the number of pixels, the number of tokens input to the multimodal large language model 312 is reduced, and the computational load of the multimodal large language model 312 can be significantly suppressed. Also, by using the gaze video with the information of important attention areas remaining while reducing the number of pixels to approximately one-tenth, it can be expected that the multimodal large language model 312 can generate text information of quality equal to or better than when the panoramic video is input. That is, it can be said that the preprocessing unit 103 is performing preprocessing to reduce the number of tokens input to the multimodal large language model while maintaining the understanding degree of the multimodal large language model used in the generation unit 104.

[0049] In the above example, the information processing system 100 created an eye-tracking video by integrating a first-person perspective video of the user with the user's eye-tracking data. However, it is not always necessary to use a first-person perspective video to create an eye-tracking video. For example, when using a third-person perspective video taken with a camera positioned to observe the entire scene including the target user, the user's gaze can be estimated using algorithms such as Attention Target Detection (ATN). Alternatively, the third-person perspective video may be converted to a video from another perspective, such as first-person, using techniques such as projection transformation, before the eye-tracking video is extracted.

[0050] C. Regarding related technologies, the field of multimodal large-scale language models is an area of ​​research on large-scale language models that has been rapidly advancing in recent years. Gemini 1.5Pro and Qwen2-VL are examples of multimodal large-scale language models that can process video content in addition to still images and audio. Gemini 1.5Pro can process video input exceeding one hour, and Qwen2-VL can operate as a locally executed MLLM system. Gemini 1.5Pro and Qwen2-VL can be introduced into the generation unit 104 of the information processing system 100 according to this disclosure and used to generate text information related to the user's task.

[0051] Multimodal large-scale language models such as Gemini 1.5Pro and Qwen2-VL provide a Vision Transformer (ViT) or its variations as an encoder specifically for video processing. ViT divides an image into a two-dimensional grid, transforms each grid into a feature vector (latent representation) in latent space, assigns tokens through vector quantization, and encodes it into a token sequence. Multimodal large-scale language models can handle the token sequence encoded from an image by integrating it with tokens from other modalities. Multimodal large-scale language models have a trade-off: as the number of pixels in the input image increases, the number of tokens also increases, leading to an increased computational load. Lowering the image resolution or downsampling the video frame rate may impair the accuracy of the multimodal large-scale language model's visual understanding.

[0052] Similarities exist between human visual patterns and attention mechanisms in large-scale language models, and similarities have been confirmed between human eye movements when interpreting text and attention mechanisms within large-scale language models. The information processing system 100 relating to this disclosure primarily utilizes user gaze information to improve the processing efficiency of first-person perspective videos using a multimodal large-scale language model.

[0053] The Multimodal Human-Like Attention Network (MULAN) is a novel algorithm used in the field of Visual Question Answering (VQA) that aims to answer questions with higher accuracy by integrating human visual attention to both images and text. There are also technologies that enable efficient viewing of first-person perspective videos. The information processing system 100 disclosed herein differs from these in that it effectively utilizes the user's gaze information as an aid when a multimodal large-scale language model processes long-duration videos.

[0054] The Ego-Exo4D project aims to build a large-scale dataset of first-person perspective data. A glasses-type device called "Aria" has been developed that can capture first-person perspective, gaze, and gesture recognition. Multiple research institutions are collaborating to collect data across various task domains. The main objective of this project is to establish and provide a data infrastructure for studying human behavior. However, this project does not include specific attempts to effectively utilize user gaze information as an aid in processing long-duration videos in multimodal large-scale language models, such as the information processing system 100 described herein.

[0055] D. Examples The information processing system 100 according to this disclosure uses a first-person perspective video of the user as input and utilizes a multimodal large-scale language model to understand the task being performed by the user and easily generate video-based guidance and answers to questions. Furthermore, the information processing system 100 according to this disclosure uses the user's gaze information to extract only the area around the gaze from the first-person perspective video and uses this gaze video as input to the generation unit 104 (multimodal large-scale language model). The gaze video has approximately one-tenth the number of pixels of the original full-view video, which can significantly reduce the computational load on the multimodal large-scale language model.

[0056] Section D describes an embodiment in which the input data of the information processing system 100 according to this disclosure is a first-person perspective video. The generation unit 104 shall use a multimodal large-scale language model such as Gemini 1.5pro.

[0057] D-1. Figures 4 to 7 of the first embodiment show an example of operation in which a video description is generated using the information processing system 100 according to the present disclosure. In the operation example presented here, a first-person perspective video of a user cooking an omelet, which has been pre-recorded and saved, is input to the information processing system 100, and the information processing system 100 is assumed to perform a system operation in which appropriate descriptive text is added to each scene of this video.

[0058] In each video scene shown in Figures 4 to 7, the rectangular area enclosed by a thick dashed line represents the region centered on the user's viewpoint. The preprocessing unit 103 extracts this region as a gaze-view video and inputs it to the generation unit 104. The generation unit 104 then estimates the task being performed (in this case, cooking an omelet) based on the input gaze-view video, and generates text information about the task (such as guidance to support the task's execution) and provides it to the user. The method of recording and providing the explanatory text and timestamp generated by the information processing system 100 for the first-person video is arbitrary.

[0059] Figures 4 to 7 show the playback video for each scene and the text information generated by the generation unit 104 from the gaze video for each scene. The text information generated by the generation unit 104 includes both a task description (guidance on cooking omelets in each scene) and the corresponding timestamp in the video. Examples of the task description and timestamp generated by the generation unit 104 for each scene shown in Figures 4 to 7 are shown below.

[0060] Figure 4: Make the sauce. In a small bowl, combine ketchup, Worcestershire sauce, salt, and pepper and mix well. [Timestamp: 00:01.42] Figure 5: Make the omelet. In a small bowl, combine eggs and milk and mix well. [Timestamp: 00:04.43] Figure 6: Cook the omelet. Add a small amount of oil to a clean frying pan and heat over medium heat. Pour in the egg mixture and cook, rotating the pan to ensure the egg is evenly distributed. [Timestamp: 00:06.13] Figure 7: Fold the omelet. Fold the remaining half of the omelet over the toppings. [Timestamp: 00:07.02]

[0061] When this video is played after being processed by the information processing system 100 as described above, the task description generated for each scene is displayed to the right of the video. For example, the video playback device may be controlled to retrieve the corresponding description and display it to the right of the video each time a timestamp is reached during video playback. Such a video with descriptions can be used as a tutorial video.

[0062] D-2. Second Embodiment As another example of the operation of the information processing system 100 according to this disclosure, we will introduce the "Stop and Ask" approach.

[0063] In this approach, the LLM is first pre-trained on the sequence of steps to perform the task using a first-person perspective video of the instructor. The instructor's first-person perspective video is pre-recorded as an instruction video. Next, the task learner is equipped with a headset capable of recording first-person perspective video and attempts to perform the task following the same steps as the instructor. As the learner progresses through the task, they pause and ask the information processing system 100 questions about the next step in the task. For example, if the task is cooking, the learner pauses in the middle of the cooking process and asks the information processing system 100 the following question:

[0064] "What should I do next?" "Which cooking utensil should I use?"

[0065] The information processing system 100 identifies the next necessary steps based on the learner's first-person perspective video recorded up to that point, generates text information (such as guidance for the next process of the task) that serves as an answer to the learner's question, and provides it to the learner. The information processing system 100 also refers to a previously recorded first-person perspective video of the instructor and provides the timestamp of the playback position corresponding to the answer to the learner in this first-person perspective video, along with the answer to the learner.

[0066] Figures 8 and 9 show an example of the operation of the "Stop and Ask" approach using the information processing system 100 related to this disclosure. In this example, cooking is assumed as the task.

[0067] Figure 8 shows a first-person perspective video of the learner. The learner pauses during cooking (while looking at a package of ginger) and asks the information processing system 100, "What should I do next?" Based on the first-person perspective video of the learner recorded up to that point, the information processing system 100 identifies the next necessary step and provides guidance on the next process of cooking. The information processing system 100 also refers to a pre-recorded instructional video of the instructor (described above) and displays the playback timestamp corresponding to the guidance in the instructional video. Figure 9 shows the video of the playback timestamp in the instructional video. For example, the following Q&A exchange takes place between the learner and the information processing system 100. The information processing system 100 may display the answer to the learner on a display, output it as audio guidance, or provide both display and audio guidance.

[0068] Q: What should I do next? A: You're asking how to prepare ginger. The next step is to wash and peel the ginger. As shown in the video, scrubbing it with aluminum foil is an effective method. The most important part of the video, washing and peeling the ginger, can be found between 2:21 and 3:58, or approximately 10 seconds from 2:34 to 2:44.

[0069] From the embodiment of the information processing system 100 described above, it is clear that combining a wearable device with an AI model is beneficial. Integrating a multimodal large-scale language model into a wearable device using first-person perspective video as contextual input opens up the possibility of using the AI ​​model as an agent in real-world applications.

[0070] For example, in a use case where a user wears smart glasses (or a see-through head-mounted display) to perform a task, the information processing system 100 can use a first-person view camera mounted on the smart glasses to acquire a first-person view video of the user while they are performing the task. The information processing system 100 uses a generation unit 104 (i.e., a multimodal large-scale language model) to understand the context of the task the user is performing from the gaze video extracted from the first-person view video. Then, when the information processing system 100 receives a question from the user via a microphone mounted on the smart glasses, it can use the multimodal large-scale language model to generate an answer to the user (text information such as task guidance) and display it on the smart glasses.

[0071] E. Evaluation To confirm the effectiveness of the information processing system 100 relating to this disclosure, the video description text generated by the information processing system 100 was evaluated using both numerical metrics and human-based crowdsourced evaluation.

[0072] E-1. Dataset The Ego-Exo4D dataset was used to evaluate the information processing system 100 related to this disclosure. The Ego-Exo4D dataset includes gaze information captured using "Aria" glasses developed by Meta, first-person view images, and third-person view video acquired from an external camera. This dataset includes various scenarios such as cooking, repair, and sports, and is a valuable resource for research.

[0073] In the evaluation of the information processing system 100 related to this disclosure, only first-person perspective videos and gaze information were used. To ensure task diversity, first-person perspective videos from six different task categories—"bicycle (bicycle repair)," "sushi (sushi preparation)," "omelet (omelet preparation)," "soccer (soccer activity)," "PCR (polymerase chain reaction testing equipment preparation and adjustment)," and "CPR (cardiopulmonary resuscitation training)"—were selected from the Ego-Exo4D dataset and used for evaluation. Each of these first-person perspective videos for each task category is assumed to have information indicating the user's gaze information attached. Table 1 shows the number of videos prepared for each task category, as well as the length of the videos (in seconds) and the standard deviation (std). The total number of tasks is 137.

[0074]

[0075] Then, from the first-person perspective videos of each task category selected from the Ego-Exo4D dataset, we prepared three types of videos: panoramic videos, line-of-sight videos, and center videos.

[0076] Panoramic video: The panoramic video is video data that uses the 1440 x 1440 pixel first-person wide-field panoramic video provided by Ego-Exo4D as is. However, the video frame rate has been reduced to 1 fps. The panoramic video does not contain line-of-sight information. In the information processing system 100, the preprocessing unit 103 outputs the first-person view video for each task by simply downsampling it to 1 fps without performing any cropping, so that the subsequent generation unit 104 generates a video description from the first-person view panoramic video.

[0077] Eye-tracking video: An eye-tracking video is video data obtained by cropping a rectangular area set to 448 x 448 pixels around the line of sight from a first-person view panoramic video. The frame rate of the eye-tracking video is maintained at 1 fps, the same as the panoramic video. In the information processing system 100, the pre-processing unit 103 downsamples the first-person view video for each task to 1 fps and performs cropping based on the line of sight information, so that the subsequent generation unit 104 generates a video description from the eye-tracking video.

[0078] Center video: The center video is simply a frame center cut out from a first-person perspective panoramic video, consisting of the same 448 x 448 pixel rectangular area as the viewpoint video, and the frame rate is 1 fps, the same as the panoramic video. In the information processing system 100, the preprocessing unit 103 downsamples the first-person perspective video for each task to 1 fps and simply cuts out the area at the center of the frame, so that the subsequent generation unit 104 generates a video description from the center video.

[0079] The number of pixels in the eye-tracking video and the central video (448 x 448 = 200,704 pixels) corresponds to 10.3% of the panoramic video (1440 x 1440 = 2,073,600 pixels). The evaluation criteria described later compare the quality of the video descriptions generated by the information processing system 100 related to this disclosure from the eye-tracking video and the central video, respectively, with the video descriptions generated from the original panoramic video.

[0080] Figure 10 illustrates the different video types: a full-view video 1001 from the user's first-person perspective, and two videos extracted from it: a viewpoint video 1002 and a center video 1003. The viewpoint video 1002 is extracted from the full-view video 1001, focusing on the area centered on the user's line of sight, while the center video 1003 is simply extracted from the full-view video 1001, focusing on the central area of ​​the frame. The user's line of sight is not necessarily located at the center of the frame in the full-view video 1001; therefore, the viewpoint video 1002 and the center video 1003 are different videos. None of the above videos contain audio (or, the generation unit 104 of the information processing system 100 does not use audio information to generate the video description).

[0081] In this evaluation, Gemini 1.5pro, applied to the generation unit 104 in the information processing system 100, was selected as the evaluation target. Gemini 1.5 Pro is a multimodal large-scale language model capable of processing videos up to approximately one hour long (downsampled to a frame rate of 1 fps). Gemini 1.5 Pro generates video descriptions (text information describing the task) from videos of each video type in each of the above task categories, according to the prompts shown below (see Figure 11).

[0082] "Please clear the chat history and start a new session. Forget all previous information and use only the information in this video. Please create instructions in English for the tasks shown in this video. I would like to insert images of the process into the instructions to make them easier to understand, so please specify the timestamps of the video frames you will be inserting."

[0083] As will be described later, the evaluation methods are broadly divided into two types: numerical evaluation based on numerical indicators and evaluation by humans. The former, numerical evaluation, uses four types of numerical indicators. In each evaluation method, by aggregating the evaluation results for each video type, it is possible to verify the performance of the multimodal large-scale language model according to the video type. Furthermore, by aggregating the evaluation results by task category, it is possible to verify whether or not the performance of the multimodal large-scale language model depends on the task.

[0084] E-2. Evaluation Results Based on Numerical Indicators The purpose of the numerical evaluation is to evaluate the accuracy of the video descriptions generated by the generation unit 104 (Gemini 1.5 Pro) from each video type, such as gaze-tracking videos and center videos, by comparing them with the video descriptions generated by the generation unit 104 from panoramic videos. For this purpose, the following four numerical indicators were used.

[0085] BLEU: The BLEU score is one of the most widely used methods for evaluating machine translation, and is based on the principle that the closer the machine-generated text is to the output of a professional translator, the higher its accuracy. In this evaluation, the BLEU score was used to measure the accuracy of the video descriptions generated by the generation unit 104 from each video type, gaze video and center video, using the video description generated by the generation unit 104 from the panoramic video as a baseline.

[0086] ROUGE: ROUGE (Recall-Oriented Understanding for Gisting Evaluation) is also widely used in natural language processing to evaluate machine-generated summaries and translations by comparing them with human-created references. In this evaluation, ROUGE-L based on the Longest Common Subsequence (LCS) was used to measure the accuracy of the video descriptions generated by the generation unit 104 from the gaze video and the central video, respectively, compared to the video description generated by the generation unit 104 from the panoramic video. ROUGE-L evaluates sentence-level similarity by sequentially counting the words that co-occur between the target summary and the reference summary.

[0087] SBERT: Sentence-BERT (SBERT) is a finely tuned version of BERT for sentence comparison and assesses semantic similarity. SBERT utilizes Siamese and triplet network structures to derive meaningful sentence feature vectors that can be compared using cosine similarity. This approach is particularly well-suited for measuring similarity between descriptions of multiple sentences. In this evaluation, the similarity between the video description generated by the generator 104 from the panoramic video and the video descriptions generated by the generator 104 from each video type (gaze video and center video) was assessed. A higher similarity indicates that the video description generated by the generator 104 closely matches the video description generated from the panoramic video.

[0088] LLM: Evaluation was conducted using LLM itself. Specifically, using ChatGPT-4, the descriptions of the videos generated by the generation unit 104 from each video type, the gaze video and the center video, were scored on a scale from 0 to 100 compared to the description of the video generated by the panoramic video, based on the following prompts (Figure 12).

[0089] "You are given two texts, Text A and Text B. Please evaluate to what extent Text B covers the content explained in Text A. Regardless of differences in wording or phrasing, evaluate the similarity of the two texts based on whether Text B contains the important information, ideas, and explanations found in Text A. Please enter a score from 0 to 100. - 100 means that Text B completely covers all the important points and information in Text A. - 0 means that Text B does not cover any of the important points in Text A. - A score from 0 to 100 represents partial coverage. Display your score as "** Score: 50 **". After assigning a score, please explain the reason for that score in a few sentences."

[0090] As described above, the numerical evaluation uses first-person perspective videos selected from the Ego-Exo4D dataset for each task category: "bicycle," "sushi," "omelet," "soccer," "PCR," and "CPR." The Gemini 1.5pro applied to the generation unit 104 then evaluates the video descriptions generated from each first-person perspective video type using the aforementioned numerical indicators: BLEU, ROUGE, SBERT, and LLM.

[0091] Figures 13 to 16 show the evaluation results of the video descriptions generated by Gemini 1.5pro from each video type (eye-tracking video and central video) using the numerical indices BLEU, ROUGE, SBERT, and LLM, respectively. In Figures 13 to 16, the scores of each numerical indices for the video descriptions generated from each video type (eye-tracking video and central video) are shown in pairs for each task category: "Bicycle," "Sushi," "Omelet," "Soccer," "PCR," and "CPR."

[0092] Figures 13 to 16 show that in all task categories, the description score of the video generated by the generation unit 104 from the gaze video is higher than the description score of the video generated from the center video. This indicates that the gaze video, which is cropped based on gaze information, captures task-related information more accurately than the center video, which is simply cropped from the center of the frame.

[0093] BLEU and ROUGE are both n-gram-based statistical measures that respond to the same vocabulary and phrases. In contrast, SBERT uses cosine similarity within the sentence embedding space to evaluate sentence similarity at a broader level beyond specific vocabulary. The LLM condition relies on LLM, which is intended to provide a comprehensive evaluation based on overall sentence understanding. By using the four different numerical indices described above, it becomes possible to suitably compare the descriptions of the videos generated by the generation unit 104 from the gaze video and center video with the descriptions of the videos generated by the generation unit 104 from the panoramic video.

[0094] The above numerical indicators evaluate the descriptions of videos generated by the generation unit 104 (Gemini 1.5pro) from both the line-of-sight video and the center video, with the description of the video generated from the panoramic video by the generation unit 104 (Gemini 1.5pro) being given a perfect score (BLEU, ROUGE, SBERT: 1.0, LLM: 100 points). However, it is important to note that the description of the video generated by the generation unit 104 from the panoramic video does not necessarily represent the most accurate truth. This is because, considering the wide field of view of first-person perspective videos, the description of the video generated from the panoramic video may include objects in the peripheral field of view that are not directly related to the task.

[0095] E-3. Human Evaluation Results Next, a human evaluation experiment was conducted. In the human evaluation experiment, as described above, first-person perspective videos selected from the Ego-Exo4D dataset for each task category, "bicycle," "sushi," "omelet," "soccer," "PCR," and "CPR," were used. The participants in this experiment were then asked to evaluate the video descriptions generated by Gemini 1.5pro, applied to the generation unit 104, from each type of first-person perspective video.

[0096] This experiment was conducted using the Prolfic crowdsourcing platform, and 20 participants were recruited online to evaluate video descriptions. Since the video descriptions were in English, participants were recruited based on their English proficiency. The average age of the participants was 29.5 years (std = 8.4).

[0097] The experiment was conducted by having participants first watch a first-person perspective video for each task category as a panoramic video, and then evaluate the video descriptions generated by the generation unit 104 (Gemini 1.5 Pro) from each video type: panoramic video, gaze video, and center video. However, to prevent the expected effect, the order of the video descriptions was randomized. Each participant provided both a numerical score on a 10-point scale and free-form text feedback for the descriptions generated from each video type.

[0098] Figure 17 shows the aggregated evaluation scores from participants for the descriptions of videos generated by the generation unit 104 (Gemini 1.5 Pro) from each video type: panoramic video, eye-tracking video, and center video. Figure 17 shows the participants' evaluation results for the descriptions of videos generated from each video type, separated by task categories: "bicycle," "sushi," "omelet," and "soccer."

[0099] As can be seen from Figure 17, in all task categories, the video descriptions generated by the generation unit 104 from the gaze video received the highest average scores. Each participant viewed only the full-view video of the first-person perspective before the evaluation, and it can be inferred that they tended to rate descriptions related to the image in the area around their gaze higher and give lower ratings to descriptions of areas other than the area around their gaze that were not relevant to the task.

[0100] The relationship between video length and description length was also investigated. Table 2 shows the length (number of characters) of the descriptions generated by the generation unit 104 from first-person perspective videos for each task category: "bicycle," "sushi," "omelet," "soccer," "PCR," and "CPR," for each video type: panoramic, line-of-sight, and center video. Furthermore, Figure 18 shows the relationship between the video length and the length (number of characters in the description) of the video generated by the generation unit 104 for each video type (i.e., panoramic, line-of-sight, and center video). As shown in Figure 18, there was only a weak correlation between video length and the number of characters in the video description (correlation coefficient: 0.13). This indicates that panoramic videos have a wider capture range and contain many objects (many of which may not be directly related to the task) compared to line-of-sight and center videos, and that the quality of the video description may be reduced as the generation unit 104 (Gemini 1.5 Pro) attempts to describe these additional elements.

[0101]

[0102] E-4. Summary of Evaluation Results Summarizing the evaluation results based on the numerical indicators described above and the human evaluation results, the following can be confirmed.

[0103] In human evaluations, the video descriptions generated by the generation unit 104 (Gemini 1.5 Pro) from eye-tracking videos received higher ratings compared to the video descriptions generated from panoramic videos and center-focused videos. Considering that the number of pixels in eye-tracking videos is only one-tenth that of panoramic videos, this indicates that eye-tracking videos are an effective way to reduce the amount of information in the input data to the generation unit 104.

[0104] In each numerical evaluation (based on BLEU, ROUGE, Sentence-BERT, and LLM), the video description generated by the generation unit 104 from the gaze video achieved a higher score than the video description generated from the center video, indicating that cutting out from the panoramic video based on gaze is a more effective way to reduce information.

[0105] F. Modified Information The information processing system 100 relating to this disclosure can reduce the amount of data from first-person perspective videos by utilizing gaze information, thereby enabling efficient understanding by a multimodal large-scale language model.

[0106] This specification describes an embodiment in which a rectangular region centered on the line of sight is cropped from the original panoramic video. However, this disclosure is not limited to such cropping methods, and there are other variations and improvements that can effectively reduce the information of input data while maintaining the quality of the text information generated by the multimodal large-scale language model.

[0107] F-1. Modification 1 Figure 19 shows a first approach that provides a multimodal large-scale language model with both a gaze-centered video (Figure 19 left) and a downsampled panoramic video (Figure 19 right) to enable understanding of areas outside the line of sight at low resolution. This approach is achieved by the preprocessing unit 103 extracting gaze information from the panoramic video of the first-person viewpoint video and downsampling the panoramic video, and inputting both of these videos to the subsequent generation unit 104.

[0108] The first approach corresponds to the relationship between central and peripheral vision in human vision. Research on multimodal large-scale language models has explored methods such as multimodal rotational position embedding (M-RoPE), which provides a unified position encoding for multistream inputs with multiple time layers. M-RoPE allows for the effective integration of positional information between different data modalities. By applying M-RoPE, the multimodal large-scale language model applied to the generation unit 104 can simultaneously input both gaze-centered video and downsampled panoramic video, integrate and process them within the model.

[0109] F-2. Modification 2 Figure 20 shows a second approach in which a deformed grid is introduced when dividing the input image into a two-dimensional grid in the preceding image encoder (such as ViT) of the multimodal large-scale language model (see, for example, Figure 3). In the second approach, the input image becomes denser (higher resolution) near the point of interest (gaze position) and sparser (lower resolution) towards the periphery (in Figure 20, the areas divided by fine rectangles are high-resolution regions, and the areas divided by coarser rectangles around them are low-resolution regions). The second approach, like the first approach, is realized by the preprocessing unit 103 performing preprocessing that reflects the relationship between the central and peripheral vision of human vision.

[0110] F-3. Third Modification The information processing system 100 relating to this disclosure primarily uses gaze information when extracting gaze videos from first-person gaze videos. Figure 3 shows the operation in which the preprocessing unit 103 extracts gaze videos based on gaze information acquired by the biometric information acquisition unit 102. In addition to gaze information, the preprocessing unit 103 can also perform the operation of extracting gaze videos using biometric information acquired by the biometric information acquisition unit 102.

[0111] For example, an object being manipulated by the hand is likely to be related to a task. The user's hand position information can be prioritized for task recognition by the generation unit 104. Since eye gaze tends to predict the next action point, combining hand activity data with eye gaze input allows for a more accurate understanding of the action. Therefore, the preprocessing unit 103 may acquire the user's hand position information along with the user's eye gaze information, integrate the eye gaze information and the user's hand position information to identify a region of interest from the overall image, and crop the image to be processed by the generation unit 104 to include this region of interest. Based on the cropped image obtained by combining the eye gaze information and the user's hand position information, the generation unit 104 will be able to accurately recognize the user's task and generate more effective text information to support the task.

[0112] Figure 21 shows an example of a first-person video (whole view image) including the user's left and right hands. When the preprocessor 103 acquires this whole view video 2100 from the data acquisition unit 101, it can detect the regions 2101 and 2102 of the left and right hands by object recognition or the like. The preprocessor 103 can also acquire the viewpoint position 2103 in the whole view video 2100 from the biometric information acquisition unit 102. The preprocessor 103 may then integrate the user's hand position information 2101 and 2102 with the user's gaze information 2103 to identify a region of interest from the whole view image 2100 and crop the image to be processed by the generation unit 104 so that it includes this region of interest. Based on the cropped image obtained by combining the user's hand position information 2101 and 2102 with the user's gaze information 2103, the generation unit 104 will be able to accurately recognize the user's task and generate more effective text information to support the task.

[0113] The biometric information acquisition unit 102 may be equipped with various biosensors other than the gaze sensor, such as a pupil sensor, pulse sensor, and IMU, to supply various types of biometric information to the preprocessing unit 103. The preprocessing unit 103 may also detect various objects other than hands included in the panoramic video through object recognition processing. The information processing system 100 may further include an object recognition module. The preprocessing unit 103 may estimate the task the user is performing based on various biometric information of the user and information on detected objects, and further estimate the situation the user is in while performing the task (for example, whether the task is progressing well), identify a region of interest according to the situation, and extract an image from the panoramic video. In such a case, the preprocessing unit 103 is not necessarily limited to gaze information, but can derive a region of interest that is important for the user to continue performing the task. The generation unit 104 can generate a task description that serves as accurate guidance to the user from the image of the region of interest.

[0114] G. Configuration of the Information Processing Device G-1. First Configuration Example Figure 22 shows an example of the hardware configuration of the information processing device 2000 applicable to this disclosure. The information processing device 2000 is composed of, for example, a personal computer (PC), but some functions may be composed of information terminals such as tablets and smartphones. The information processing system 100 according to this disclosure can be constructed using the information processing device 2000. The information processing system 100 according to this disclosure can be constructed using one information processing device 2000, or it can be constructed by linking multiple information processing devices 2000. Furthermore, evaluation experiments of the information processing system 100 according to this disclosure can be conducted using the information processing device 2000.

[0115] This information processing device 2000 includes a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, a RAM (Random Access Memory) 2003, a host bus 2004, a bridge 2005, an expansion bus 2006, an interface unit 2007, an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013.

[0116] CPU 2001 controls the overall operation of the information processing unit 2000 according to various programs. When performing computationally intensive processes such as AI (Artificial Intelligence) model training on the information processing unit 2000, it is desirable that CPU 2001 be a multi-core CPU (e.g., Apple M1 Max), or that the information processing unit 2000 also be equipped with a multi-core processor such as a GPU or GPGPU (General-purpose computing on graphics processing units) (e.g., NVIDIA's "RTX A6000"). However, for convenience, these will be collectively referred to simply as CPU 2001 below.

[0117] ROM2002 non-volatilely stores programs (such as the basic input / output system) and arithmetic parameters used by CPU2001. RAM2003 is used to load programs to be executed by CPU2001 and to temporarily store parameters such as work data that change as needed during program execution. Programs loaded into RAM2003 and executed by CPU2001 include, for example, various application programs and operating systems (OS).

[0118] The CPU 2001, ROM 2002, and RAM 2003 are interconnected by a host bus 2004, which consists of a CPU bus and the like. The CPU 2001, through the collaborative operation of ROM 2002 and RAM 2003, can execute various application programs under the execution environment provided by the OS, thereby realizing a variety of functions and services. If the information processing device 2000 is a PC, the OS is, for example, Microsoft's Windows®, Unix®, or its successor OS. For example, application programs that perform processes such as cropping a viewpoint video from a panoramic video in the preprocessing unit 103 of the information processing system 100 according to this disclosure, or generating text information from a viewpoint video in the generation unit 104, are executed on the information processing device 2000.

[0119] The host bus 2004 is connected to the expansion bus 2006 via the bridge 2005. The expansion bus 2006 is, for example, a PCI (Peripheral Component Interconnect) bus or PCI Express, and the bridge 2005 is based on the PCI standard. However, the information processing device 2000 does not need to be configured in a way that isolates its circuit components by the host bus 2004, the bridge 2005, and the expansion bus 2006; it may be an implementation in which almost all circuit components are interconnected by a single bus (not shown).

[0120] The interface unit 2007 connects peripheral devices such as the input unit 2008, output unit 2009, storage unit 2010, drive 2011, and communication unit 2013 in accordance with the expansion bus 2006 standard. However, not all peripheral devices shown in Figure 22 are necessarily required, and the information processing device 2000 may include additional peripheral devices not shown. Furthermore, the peripheral devices may be built into the main body of the information processing device 2000, or some peripheral devices may be externally connected to the main body of the information processing device 2000.

[0121] The input unit 2008 consists of an input control circuit that generates an input signal based on user input and outputs it to the CPU 2001. If the information processing device 2000 is a PC, the input unit 2008 may include a keyboard, mouse, touch panel, camera, and microphone. The input unit 2008 may include a first-person view camera, or it may have an interface for inputting first-person view video captured by a first-person view camera. Furthermore, the input unit 2008 may include a biosensor such as an eye-tracking sensor, or it may have an interface for inputting biometric information from a biosensor.

[0122] The output unit 2009 includes, for example, display devices such as liquid crystal display (LCD) devices, organic EL (Electro-Luminescence) display devices, and LED (Light Emitting Diode) devices, as well as sound output devices such as speakers. The display device is used, for example, to display a playback video of a first-person perspective video acquired by the input unit 2008, or to display text information such as a description of a video generated from a first-person perspective video (or a viewpoint video extracted from a first-person perspective video).

[0123] The storage unit 2010 stores files such as programs (applications, OS, etc.) and various data executed by the CPU 2001. The storage unit 2010 is composed of a large-capacity storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), but may also include an external storage device. For example, the storage unit 2010 is used for recording first-person perspective videos and for recording text information such as descriptions of videos generated from first-person perspective videos (or viewpoint videos extracted from first-person perspective videos).

[0124] The removable storage medium 2012 is a storage medium configured in a cartridge format, such as a microSD card. The drive 2011 performs read and write operations on the loaded removable storage medium 2012. The drive 2011 outputs data read from the removable storage medium 2012 to the RAM 2003 or storage unit 2010, and writes data on the RAM 2003 or storage unit 2010 to the removable storage medium 2012.

[0125] The communication unit 2013 is a device that performs wireless communication such as Wi-Fi®, Bluetooth®, and cellular communication networks such as 4G and 5G. The communication unit 2013 may also be equipped with terminals such as USB (Universal Serial Bus) and HDMI® (High-Definition Multimedia Interface), and may further have the function of performing HDMI® communication with USB devices such as scanners and printers, and displays. Programs executed on the information processing device 2000 are installed externally, for example, through the communication unit 2013. Furthermore, text information such as descriptions of videos generated from first-person perspective videos (or viewpoint videos extracted from first-person perspective videos) can be output externally via the communication unit 2013.

[0126] G-2. Second Configuration Example Next, a system configuration example will be described when the information processing system 100 according to this disclosure is applied to smart glasses. Smart glasses are glasses-type wearable devices worn by the user and include components such as a camera, sensors, and a display, and can also incorporate voice assistants and AI functions. For example, the information processing system 100 can overlay text information, such as a video description generated from a first-person perspective video (or a viewpoint video extracted from a first-person perspective video), onto the user's field of view in the real world using smart glasses.

[0127] Figure 23 shows a user wearing smart glasses 2301 on their head, and controllers 2302 and 2303 on their left and right hands, respectively.

[0128] The smart glasses 2301 have a function to overlay and display information onto the user's field of view in the real world. The information displayed by the smart glasses 2301 includes virtual objects and information generated by the information processing system 100 (text information such as guidance to assist the user in performing a task).

[0129] Each controller 2302 and 2303 has the functions of detecting hand position, recognizing finger posture, and recognizing finger gestures. Therefore, the smart glasses 2301 can recognize the position of each hand, the posture of the fingers, and the gestures of the fingers, both left and right, through the controllers 2302 and 2303. The smart glasses 2301 also has the function of detecting the position and posture of the user's head. Therefore, the smart glasses 2301 can detect the relative position between the user's head and the controllers 2302 and 2303, in other words, the relative position of the user's left and right hands.

[0130] However, controllers 2302 and 2303 are not essential for realizing the information processing system 100 related to this disclosure. For example, the position and posture of the user's hand can be detected by object recognition of the first-person view video captured by the gaze camera (described later) mounted on the smart glasses 2301. Furthermore, if information on the position and posture of the hand is not used in the pre-processing of the first-person view video (eye-view video cropping), then controllers 2302 and 2303 and object recognition of the user's hand are completely unnecessary.

[0131] Figure 24 shows the smart glasses 2301 being worn on the user's head. The smart glasses 2301 are a glasses-type wearable device worn on the user's head. The smart glasses 2301 include a display unit 2401 for the right eye and a display unit 2402 for the left eye. The display units 2401 and 2402 are transparent or semi-transparent, i.e., see-through, and can superimpose digital information such as virtual objects at predetermined positions in real space, emphasize or attenuate specific real objects, or remove specific real objects to make them appear as if they do not exist. A first-person view camera 2403 for recording video from the user's first-person perspective is positioned approximately in the center of the smart glasses 2301. The display units 2401 and 2402 may display text information generated by the generation unit 104 (a description of the task captured by the first-person view camera 2403).

[0132] Figure 25 shows an example of the functional configuration of an information processing system 100, which includes smart glasses 2301, left and right controllers 2302 and 2303, and an information processing device 2000. The smart glasses 2301 include a display unit 2501, a speaker 2502, and a head sensor unit 2503. The controllers 2302 and 2303 include a hand position detection unit 2511, a finger posture recognition unit 2512, a finger gesture recognition unit 2513, and a haptic feedback unit 2514. However, controllers 2302 and 2303 are not essential components of the information processing system 100. The configuration of the information processing device 2000 is as shown in Figure 22, but for the sake of explanation, it is assumed here to include the functional components of a control unit 2521, a storage unit 2522, and a communication unit 2523.

[0133] Controllers 2302 and 2303 include a hand position detection unit 2511, a finger posture recognition unit 2512, a finger gesture recognition unit 2513, and a haptic feedback unit 2514. The hand position detection unit 2511 detects the position of the user's hand. The finger posture recognition unit 2512 recognizes the posture of the user's fingers. The finger gesture recognition unit 2513 recognizes finger gestures, for example, whether the fingertips of the thumb and other fingers (such as the index finger) are in contact or separated. The haptic feedback unit 2514 is configured, for example, by arranging electromagnetic or piezoelectric vibrators in an array, and provides haptic feedback to the back of the user's hand by presenting vibrations.

[0134] The head sensor unit 2503 is mounted on the smart glasses 2301 and includes a first-person view camera 2531, a gaze sensor 2532, a microphone 2533, a gyroscope 2534, an accelerometer 2535, and a compass sensor 2536. The head sensor unit 2503 can function as a biometric information acquisition unit 104 of the information processing system 100.

[0135] The first-person view camera 2531 is, for example, an RGB camera and is installed to capture first-person view video from the viewpoint of a user wearing smart glasses 2301. In addition to the first-person view camera 2531, the system may also include one of the following as a means of capturing the outside world: an IR camera consisting of an IR light-emitting unit and an IR light-receiving unit, or a ToF camera. The gaze sensor 2532 is, for example, an IR camera and monitors the gaze of the user wearing smart glasses 2301. The first-person view video captured by the first-person view camera 2531 and the user's gaze information captured by the gaze sensor 2532 are transferred to the information processing device 2000 and recorded, for example, in the storage unit 2522.

[0136] The microphone 2533 may be a single sound-receiving element or a microphone array consisting of multiple sound-receiving elements. The microphone 2533 picks up the voice spoken by the user wearing the smart glasses 2301 and the sounds around the user. The user can, for example, voice input questions about the task being performed via the microphone 2533. The audio signal picked up by the microphone 2533 is transferred to the information processing device 2000.

[0137] The gyro sensor 2534, the acceleration sensor 2535, and the orientation sensor 2536 may be configured as an IMU (Inertial Measurement Unit). The sensor signals of the gyro sensor 2534, the acceleration sensor 2535, and the orientation sensor 2536 are transferred to the information processing device 2000. Based on these sensor signals, the information processing device 2000 can detect the position and orientation of the head of the user wearing the smart glasses 2301, or estimate the actions and tasks being performed by the user.

[0138] The display unit 2501 corresponds to the left and right display units 2401 and 2401 shown in FIG. 24, and is composed of a transmissive display (such as a glasses lens), and performs a display operation based on a control signal from the information processing device 2000. The display unit 2501 displays text information such as information (virtual objects) and task descriptions, or emphasizes, attenuates, or deletes real objects, thereby expanding the real world as seen by the user.

[0139] The speaker 2502 is composed of a single sound emitting element or an array of multiple sound emitting elements, and is mounted on the smart glasses 2301. From the smart glasses 2301, for example, a voice related to the virtual object displayed on the display unit 2501 is output. Also, the speaker 2502 may output a voice reading of text information such as a task description generated by the information processing system 1000 from the first-person perspective video of the user, or may output other audio signals.

[0140] The communication unit 2523 corresponds to the communication unit 2013 in FIG. 22. The control unit 2521 corresponds to the CPU 2001, ROM 2002, and RAM 2003 in FIG. 22, and executes the process of extracting the line-of-sight video from the panoramic video in the preprocessing unit 103 of the information processing system 100 and the process of generating text information such as video descriptions from the line-of-sight video in the generation unit 104. Further, the storage unit 2522 corresponds to the storage unit 2010 in FIG. 22, and is used for recording the first-person viewpoint video captured by the first-person viewpoint camera 2531 on the smart glass 2301 side and recording text information such as descriptions of videos generated from the first-person viewpoint video (or the line-of-sight video clipped from the first-person viewpoint video).

[0141] H. Conclusion The information processing system 100 according to the present disclosure has a function of mainly generating text information from videos by using a multimodal large language model. In this specification, the information processing system 100 according to the present disclosure has been described mainly with an embodiment of generating a task description from the first-person viewpoint video of a user performing a task, together with its evaluation experiment.

[0142] In the evaluation experiment, it was carried out using six types of videos (a total of 135 videos) such as cooking, bicycle repair, healthcare, and sports. The information processing system 100 according to the present disclosure was evaluated for how appropriately the task is described by a line-of-sight image clipped based on the line of sight, even if the image clipped from the full-resolution panoramic image contains only one-tenth of the pixels of the original image. For the evaluation, BLEU, ROUGE, sentence-BERT metric, and LLM-based explanation evaluation were used. In all these cases, the line-of-sight video clipped based on the line of sight obtained a high evaluation score compared with the central video clipped from the center of the image. Also, in the user evaluation (20 participants), it was confirmed that the explanations generated from the line-of-sight video clipped based on the line of sight were evaluated higher than the explanations generated from the panoramic video or the central video clipped from the center.

[0143] From the results of the evaluation experiment, it was confirmed that when the information processing system 100 according to the present disclosure generates an explanation of a first-person perspective video using a multimodal large language model, it is an effective approach to perform cropping based on the line of sight. The information processing system 100 according to the present disclosure can reduce the number of pixels to be processed, thereby reducing the computational load and memory usage of the multimodal large language model. In addition, the information processing system 100 according to the present disclosure can efficiently process longer videos by the approach of cropping a first-person perspective video based on the line-of-sight information.

[0144] As described above, the present disclosure has been described in detail with reference to specific embodiments. However, the present disclosure should not be construed as being limited to the above-described embodiments, and it is obvious that those skilled in the art can make modifications and substitutions to the embodiments without departing from the gist of the present disclosure. In addition, the effects described in this specification are merely examples, and the effects brought by the present disclosure are not limited, and there may be additional effects not described in this specification.

[0145] In this specification, the information processing system 100 according to the present disclosure has been mainly described in the embodiment of generating an explanation of a task from a first-person perspective video of a user performing a task. However, the gist of the present disclosure is not limited to this. The processing of videos by the information processing system 100 according to the present disclosure can be applied to various uses such as navigation support for disabled or elderly people, rehabilitation support and behavior monitoring for patients, education, sports instruction, development of first-person perspective games, production of first-person perspective scenes in movie production, analysis and danger detection of videos captured by surveillance cameras, in addition to task support. Of course, the information processing system 100 according to the present disclosure can also be applied to data processing of various modalities other than videos, such as text and audio, and generation of explanations for data of each modality.

[0146] In short, the present disclosure has been described in the form of examples, and the description in this specification should not be construed restrictively. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0147] A series of processes described in this specification can be executed by hardware, software, or a combination of hardware and software. When executing the process by software, a program recording the process sequence related to the realization of the present disclosure is installed in the memory in a computer incorporated in dedicated hardware and executed. It is also possible to install the program in a general-purpose computer capable of executing various processes and execute the process related to the realization of the present disclosure.

[0148] The program can be stored in advance in a recording medium equipped in a computer, such as an HDD, SSD, or ROM as a recording medium. Alternatively, the program can be temporarily or permanently stored in a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto-optical) disk, DVD (Digital Versatile Disc), BD (Blu-Ray Disc (registered trademark)), magnetic disk, USB (Universal Serial Bus) memory. Using such a removable recording medium, a program related to the realization of the present disclosure can be provided as so-called package software.

[0149] Further, the program may be transferred to a computer wirelessly or wired via a network such as a WAN (Wide Area Network) represented by cellular, LAN (Local Area Network), or the Internet from a download site. In the computer, the program transferred in this way can be received and installed in a large-capacity storage device such as an HDD or SSD in the computer.

[0150] Note that the present disclosure can also be configured as follows.

[0151] (1) An information processing method having: a data acquisition step of acquiring input data; a biometric information acquisition step of acquiring biometric information of a user; a preprocessing step of preprocessing the input data based on the biometric information; and a generation step of generating information to be provided to the user based on the preprocessed input data.

[0152] (2) In the preprocessing step, based on the biometric information, a target area in the input data is specified, and the input data is preprocessed so as to include more information of the target area or reduce the amount of information outside the target area. The information processing method according to (1) above.

[0153] (3) In the generation step, using a learned model, information to be provided to the user is generated from the preprocessed input data. The information processing method according to any one of (1) or (2) above.

[0154] (4) In the data acquisition step, multimodal input data is acquired, and in the generation step, using a multimodal large language model, text information to be provided to the user is generated from the preprocessed multimodal input data. The information processing method according to any one of (1) to (3) above.

[0155] (5) In the data acquisition step, a first-person perspective video of the user is acquired, in the biometric information acquisition step, the user's gaze information is acquired, and in the preprocessing step, based on the gaze information, a target area in the panoramic video of the first-person perspective video is specified, and the first-person perspective video is preprocessed so as to include more information of the target area or reduce the amount of information outside the target area. The information processing method according to any one of (1) to (4) above.

[0156] (6) In the preprocessing step, a gaze video composed of a target area narrowed around the user's gaze is cut out from the panoramic video of the first-person perspective video, and in the generation step, information to be provided to the user is generated based on the gaze video. The information processing method according to (5) above.

[0157] (7) In the generation step, the gaze video is encoded into a token sequence using an image encoder, and information to be provided to the user is generated from the token sequence using a multimodal large language model. The information processing method according to (6) above.

[0158] (8) In the biometric information acquisition step, information on a specific part of the user's body is further acquired, and in the preprocessing step, a region of interest is specified in the panoramic video of the first-person perspective video based on a combination of the user's gaze information and information on the specific part. The information processing method according to any one of (5) to (7) above.

[0159] (9) The information on the specific part includes the position information of the user's hand. The information processing method according to (8) above.

[0160] (10) In the generation step, based on the first-person perspective video after preprocessing, the user's task is estimated and information for assisting the user in task execution is generated. The information processing method according to any one of (5) to (9) above.

[0161] (11) In the preprocessing step, preprocessing is performed to reduce the number of tokens input to the multimodal large language model while maintaining the understanding degree of the multimodal large language model. The information processing method according to (7) above.

[0162] An information processing system including: a data acquisition unit that acquires input data; a biometric information acquisition unit that acquires the user's biometric information; a preprocessing unit that preprocesses the input data based on the biometric information; and a generation unit that generates information to be provided to the user based on the input data after preprocessing.

[0163] A computer program described in a computer-readable format so as to cause a computer to function as a data acquisition unit that acquires input data, a biometric information acquisition unit that acquires the user's biometric information, a preprocessing unit that preprocesses the input data based on the biometric information, and a generation unit that generates information to be provided to the user based on the input data after preprocessing.

[0164] 100... Information processing system, 101... Data acquisition unit 102... Biometric information acquisition unit, 103... Preprocessing unit, 104... Generation unit 2000... Information processing device, 2001... CPU, 2002... ROM 2003... RAM, 2004... Host bus, 2005... Bridge 2006... Expansion bus, 2007... Interface unit 2008... Input unit, 2009... Output unit, 2010... Storage unit 2011... Drive, 2012... Removable recording medium 2013... Communication unit 2301... Smart glasses, 2302, 2303... Controllers 2401, 2402... Display units 2501... Display unit, 2502... Speaker, 2503... Head sensor unit 2511... Hand position detection unit, 2512... Finger posture recognition unit 2513... Finger gesture recognition unit, 2514... Tactile feedback unit 2521... Control unit, 2522... Memory unit, 2523... Communication unit 2531... First-person perspective camera, 2532... Gaze sensor 2533... Microphone, 2534... Gyro sensor 2535... Acceleration sensor, 2536... Azimuth sensor

Claims

1. An information processing method comprising: a data acquisition step of acquiring input data; a biometric information acquisition step of acquiring the user's biometric information; a preprocessing step of preprocessing the input data based on the biometric information; and a generation step of generating information to be provided to the user based on the preprocessed input data.

2. The information processing method according to claim 1, wherein the preprocessing step involves identifying a region of interest from the input data based on biological information, and preprocessing the input data so as to include more information from the region of interest or reduce the amount of information from areas other than the region of interest.

3. The information processing method according to claim 1, wherein the generation step involves using a trained model to generate information to be provided to the user from preprocessed input data.

4. The information processing method according to claim 1, wherein the data acquisition step involves acquiring multimodal input data, and the generation step involves generating text information to be provided to the user from the pre-processed multimodal input data using a multimodal large-scale language model.

5. The information processing method according to claim 1, wherein the data acquisition step involves acquiring a first-person perspective video of the user, the biometric information acquisition step involves acquiring the user's gaze information, and the preprocessing step involves identifying a region of interest from the full-view video of the first-person perspective video based on the gaze information, and preprocessing the first-person perspective video so as to include more information about the region of interest or reduce the amount of information outside the region of interest.

6. The information processing method according to claim 5, wherein in the preprocessing step, a gaze video consisting of a focus area centered on the user's gaze is extracted from a full-view video of a first-person perspective video, and in the generation step, information to be provided to the user is generated based on the gaze video.

7. The information processing method according to claim 6, wherein the generation step involves encoding the gaze video into a token sequence using an image encoder and generating information to be provided to the user from the token sequence using a multimodal large-scale language model.

8. The information processing method according to claim 5, wherein the biometric information acquisition step further acquires information on a specific part of the user's body, and the preprocessing step identifies a region of interest in a full-view video of a first-person perspective video based on a combination of the user's gaze information and the information on the specific part.

9. The information processing method according to claim 8, wherein the information of the specific body part includes the position information of the user's hand.

10. The information processing method according to claim 5, wherein the generation step involves estimating the user's task and generating information to support the user's task performance based on a pre-processed first-person perspective video.

11. The information processing method according to claim 7, wherein the preprocessing step involves performing preprocessing to reduce the number of tokens input to the multimodal large-scale language model while maintaining the level of understanding of the multimodal large-scale language model.

12. An information processing system comprising: a data acquisition unit for acquiring input data; a biometric information acquisition unit for acquiring the user's biometric information; a preprocessing unit for preprocessing input data based on the biometric information; and a generation unit for generating information to be provided to the user based on the preprocessed input data.

13. A computer program written in a computer-readable format to cause a computer to function as a data acquisition unit that acquires input data, a biometric information acquisition unit that acquires the user's biometric information, a preprocessing unit that preprocesses the input data based on the biometric information, and a generation unit that generates information to be provided to the user based on the preprocessed input data.

Images