Motion information analysis device, method, and program

Abstract

According to one embodiment of the present invention: first motion information representing a motion of a student and second motion information representing a motion of a teacher that serves as a model for the motion of the student are acquired; a first feature amount and a second feature amount representing features of the motions at a desired motion timing are extracted from the acquired first motion information and second motion information, respectively; and information representing the difference between the extracted first feature amount and second feature amount is derived.

Description

Motion Information Analysis Device, Method, and Program

[0001] One aspect of this invention relates to a motion information analysis device, method, and program used for analyzing, for example, human motions in sports, cooking, and daily life.

[0002] For example, in the fields of sports and cooking, improving a player's movements can lead to performance improvement. Therefore, in recent years, various training systems have been proposed that detect the difference between a player's movement and an ideal movement by a coach or the like, generate coaching information based on the detected difference, and present it to the player.

[0003] For example, Non-Patent Document 1 describes a technique for supporting a student to acquire an ideal movement by comparing the movement videos of the student and the teacher in a cooking process, detecting the difference between the two movements, and coaching based on the detected difference. At that time, as a method for detecting the difference in movements, text data for explaining the movement content is generated from the movement video of the student and the movement video of the teacher, and the same section of each of the above movement videos is extracted based on the degree of coincidence of the two generated text data, or the above two movement videos are input together with the above text data into a VLM (Vision & Language Model) to detect the difference between the movement of the teacher and the movement of the student.

[0004] Nagarajan et al, “Step Differences in Instructional Video”, CVPR, 2024. <arXiv:2404.16222v2 [cs.CV] 27 Jun 2024>

[0005] However, the method described in Non-Patent Document 1 generates text data describing the actions from two videos to be compared, and then uses the generated text data to detect differences in action. In other words, it performs alignment of the video using text data and then detects differences in action. For this reason, it can only detect differences in action on a unit basis of the actions of multiple individual processes that constitute the overall process (hereinafter referred to as global motion) (hereinafter referred to as local motion).

[0006] For example, in the cooking process, it is only possible to detect differences in actions at the level of independent, individual steps within the overall process, such as "cutting ingredients with a knife" or "sautéing ingredients in a frying pan." It is difficult to detect instantaneous differences in movement within a single step, such as "cutting ingredients with a knife."

[0007] This invention was made in view of the above circumstances and aims to provide a technology that enables the analysis of instantaneous motion during local motion.

[0008] To solve the above problems, one embodiment of the motion information analysis device or motion information analysis method according to the present invention acquires first motion information representing the student's movements and second motion information representing the teacher's movements which serve as a model for the student's movements, extracts first and second feature quantities representing the characteristics of the movements at desired timings from the acquired first and second motion information, and obtains information representing the difference between the extracted first and second feature quantities.

[0009] According to one aspect of this invention, it is possible to calculate difference information between the actions of a teacher and a student at a desired timing in the actions of the teacher and the student. Therefore, for example, by referring to the difference information, it is possible to generate support information to help the student correct their actions.

[0010] In other words, according to one aspect of this invention, it is possible to provide a technology that enables the analysis of instantaneous motion during local motion.

[0011] Figure 1 is a diagram used to provide an overview of the motion information analysis process according to one embodiment of the present invention. Figure 2 is a block diagram showing an example of the hardware configuration of a motion information analysis device according to one embodiment of the present invention. Figure 3 is a block diagram showing an example of the software configuration of a motion information analysis device according to one embodiment of the present invention. Figure 4 is a flowchart showing an example of the processing procedure and processing content of the motion information analysis process executed by the control unit of the motion information analysis device shown in Figure 3. Figure 5 is a flowchart for explaining a first embodiment of the video alignment process among the motion information analysis processes shown in Figure 4. Figure 6 is a flowchart for explaining a second embodiment of the video alignment process among the motion information analysis processes shown in Figure 4. Figure 7 is a diagram used to provide an overview of the motion information analysis process according to another embodiment of the present invention.

[0012] Embodiments of this invention will be described below with reference to the drawings.

[0013] [One Embodiment] (Overview) Figure 1 is a diagram illustrating the overview of the operation information analysis process according to one embodiment of the present invention.

[0014] One embodiment of this invention is a system that supports training to improve a person's gait, for example, by extracting important frames representing the movements at a desired moment for comparison from video footage VS1 and VS2, which capture the walking movements of a student and a teacher (instructor), respectively. Specifically, alignment processing AL using image features is performed on the student video VS1 and the teacher video VS2. Then, by inputting the aligned student video AV1 and the teacher video AV2 into, for example, a VLM (Vision & Language Model) GA, the difference between the student's movements and the teacher's movements is detected, and coaching information CD is generated based on this difference.

[0015] Therefore, in one embodiment of this invention, it becomes possible to extract desired momentary movements during walking by image alignment processing AL based on image features, thereby enabling detailed coaching on momentary movements during walking.

[0016] (Configuration Example) Figures 2 and 3 are block diagrams showing examples of the hardware and software configurations of the motion information analysis device MA according to one embodiment of the present invention.

[0017] The motion information analysis device MA is installed, for example, in a personal computer provided in a training system. The motion information analysis device MA includes a control unit 1 that uses a hardware processor such as a Central Processing Unit (CPU), and the control unit 1 is connected to a storage medium having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter referred to as I / F) unit 4 via a bus 5.

[0018] The input / output I / F section 4 is connected to the camera CM and the display device DP, for example, via a signal cable. A wireless interface employing a low-power wireless data communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark) may be used as the connection medium.

[0019] The camera CM is positioned, for example, to the side of a walker set up for training, and captures images of the walking movements of students and teachers as they walk on the walker from the side. The captured video data representing the walking movements of the students and teachers is then output to the motion information analysis device MA. The camera CM may also be positioned to capture images of students and teachers from the front, back, or above, or multiple cameras may be positioned to capture images from multiple directions.

[0020] The display device DP is used to present coaching information generated by the motion information analysis device MA to students, and a display such as a television monitor is used. Alternatively, the display device may be a personal computer, a smartphone owned by a student, or a projector that projects video information onto a wall.

[0021] The program storage unit 2 is configured, for example, by combining a non-volatile memory that can be written to and read at any time, such as an SSD (Solid State Drive), and a non-volatile memory such as ROM (Read Only Memory), and stores application programs necessary to execute various controls according to one embodiment, in addition to middleware such as an OS (Operating System). Hereafter, the OS and each application program will be collectively referred to as a program.

[0022] The data storage unit 3 combines, for example, a non-volatile memory such as an SSD that can be written to and read at any time, and a volatile memory such as RAM (Random Access Memory) as a storage medium. The storage area is provided with a teacher video storage unit 31, a student video storage unit 32, an aligned teacher video storage unit 33, an aligned student video storage unit 34, and a differential information storage unit 35, which serve as the main storage units according to one embodiment of this invention.

[0023] The teacher video storage unit 31 stores video data representing the teacher's walking movements captured by the camera CM. The student video storage unit 32 stores video data representing the student's walking movements captured by the camera CM.

[0024] The aligned teacher video storage unit 33 stores important image frames representing moments of walking motion that are to be compared with student videos, extracted from the video data representing the teacher's walking motion, as aligned teacher videos.

[0025] The aligned student video storage unit 34 stores important image frames representing moments of walking motion extracted from the video data representing the student's walking motion, which are to be compared with the teacher's video, as aligned student videos.

[0026] The differential information storage unit 35 stores information representing the difference between the aligned teacher video and the aligned student video.

[0027] The control unit 1 includes, as processing functions necessary to realize one embodiment of this invention, a video data acquisition processing unit 11, a video alignment processing unit 12, a critical frame comparison processing unit 13, and a coaching information generation processing unit 14.

[0028] Each of the above-mentioned processing units 11 to 14 is implemented by having the hardware processor of the control unit 1 execute an application program stored in the program storage unit 2. Note that some or all of the above-mentioned processing units 11 to 14 may be implemented using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

[0029] The video data acquisition processing unit 11 acquires video data representing the walking movements of the teacher and student, captured by the camera CM during the training support period, via the input / output I / F unit 4, and stores the acquired video data of the teacher and student in the teacher video storage unit 31 and student video storage unit 32, respectively.

[0030] The video alignment processing unit 12 performs alignment processing on the teacher's video data and the student's video data based on image features. For example, the video alignment processing unit 12 reads the teacher's video data and the student's video data from the teacher's video storage unit 31 and the student's video storage unit 32, respectively, and extracts important image frames from each of the read video data that represent moments during walking motion to be compared. Then, the extracted important frames of the teacher and student are stored as aligned video data in the aligned teacher's video storage unit 33 and the aligned student's video storage unit 34, respectively. A specific example of the process of extracting the important image frames will be explained in the operation example.

[0031] The important frame comparison processing unit 13 reads important image frames of the teacher and important image frames of the student from the aligned teacher image storage unit 33 and the aligned student image storage unit 34, respectively, and calculates difference information between the read important image frames. The calculated difference information is then stored in the difference information storage unit 35. An example of the process for acquiring the difference information will also be explained in the operation example.

[0032] The coaching information generation processing unit 14 generates coaching information to support training to correct the student's walking motion based on the differential information stored in the differential information storage unit 35, and outputs the generated coaching information from the input / output I / F unit 4 to the display device DP.

[0033] (Example of operation) Next, an example of the operation of the motion information analysis device MA configured as described above will be explained.

[0034] Figure 4 is a flowchart showing an example of the processing procedure and processing content of the motion information analysis process performed by the control unit 1 of the motion information analysis device MA.

[0035] (1) In step S1, the control unit 1 of the motion information analysis device MA monitors the input of a training support start request. When a teacher or student performs a training support start operation in this state, the control unit 1 of the motion information analysis device MA performs the teacher video and student video acquisition process under the control of the video data acquisition processing unit 11 as follows.

[0036] In other words, assuming that, for example, the teacher and then the student perform walking movements, the video data acquisition processing unit 11 first acquires video data representing the teacher's walking movements, captured from the side by the camera CM, via the input / output I / F unit 4, and then sequentially stores the acquired video data in the teacher video storage unit 31 on a frame-by-frame basis.

[0037] When the teacher finishes walking and the student then performs walking, the video data acquisition processing unit 11 acquires video data representing the student's walking motion, captured from the side by the camera CM, via the input / output I / F unit 4 in step S3, and sequentially stores the acquired video data frame by frame in the student video storage unit 32.

[0038] In the example above, video data was acquired in the order of teacher followed by student, but it is also possible to acquire the video data in the order of student followed by teacher.

[0039] (2) Video alignment processing When step S4 determines that the acquisition of the teacher and student video data has been completed, the control unit 1 of the motion information analysis device MA then performs video alignment processing on the teacher video and student video under the control of the video alignment processing unit 12 in step S5 as follows.

[0040] For example, the following two algorithms can be used as processing algorithms for video alignment.

[0041] (2-1) Method using SIFT (Scale-Invariant Feature Transform) features Figure 5 is a flowchart showing an example of the processing procedure and processing content of video alignment processing using a method for extracting SIFT features.

[0042] The video alignment processing unit 12 first reads the teacher's video data from the teacher video storage unit 31 frame by frame in step S51, and extracts the SIFT feature amounts from each of the read frames.

[0043] The SIFT feature amounts are extracted in two stages: a feature point detection process and a feature amount description process. First, the feature point detection process is a process of narrowing down points with a large amount of spatial change in the image and containing a large amount of information such as edges and textures.

[0044] Specifically, first, Gaussian filters are used to generate smoothed images with different scales from the original video. Then, after obtaining the difference images between the generated smoothed images, the extreme values of this difference image are searched, and points that are not suitable as feature points are deleted to narrow down the feature points. The points remaining after this narrowing down become the feature points.

[0045] On the other hand, the feature amount description process is a process of determining the gradient direction and creating a gradient direction histogram of the feature amounts based on the feature points obtained in the previous process.

[0046] Specifically, first, a gradient direction histogram is obtained based on the gradient direction and intensity of the luminance around the feature points. Then, in the obtained histogram, a gradient direction whose intensity exceeds, for example, 80% of the maximum value is adopted as the "direction of the feature point". When calculating the intensity, for example, weighting is performed using the scale of the Gaussian filter of the feature point so that feature amounts close to the feature point can be emphasized. Next, based on the determined direction of the feature point, the gradient direction histogram of the feature amounts is reconstructed. For example, the area for describing the feature amounts is rotated according to the direction of the feature point and divided into small cells, and a gradient direction histogram is created for each cell.

[0047] That is, the SIFT feature amounts are obtained by vectorizing the calculated gradient direction histogram, and since the feature amounts are described according to the direction of the feature points, it is possible to extract the position information of the object. Therefore, it becomes an image feature amount that is not affected even if the object rotates or the angle changes between images.

[0048] Next, in step S52, the video alignment processing unit 12 matches the image feature amounts obtained for each frame with the reference feature amounts of the frames to be extracted, which are prepared in advance. Then, in step S53, the image feature amount having the highest similarity with the reference feature amounts is extracted as an important image frame representing the desired moment during the walking motion. Examples of the image frames representing the desired moment during the walking motion include frames including "the moment when the right foot touches the ground", "the moment when the knee is at the highest position", and "the moment when the left heel touches the ground".

[0049] Finally, in step S54, the video alignment processing unit 12 stores the extracted important image frame in the aligned teacher video storage unit 33 as aligned teacher video data.

[0050] Next, the video alignment processing unit 12 reads the video data of the student frame by frame from the student video storage unit 32. Then, in steps S52 to S54, in the same manner as the processing for the teacher video described above, the SIFT feature amounts are extracted from each frame of the read video data of the student, the extracted image feature amounts are matched with the reference feature amounts of the frames to be extracted, and the image feature amount having the highest similarity is extracted as an important image frame representing the moment during the walking motion, and the extracted image frame is stored in the aligned student video storage unit 34 as the aligned student video.

[0051] (2-②) Method Using a Machine Learning Model Fig. 6 is a flowchart showing an example of the processing procedure and processing content of video alignment processing using a machine learning model. <00,00103>

[0052] First, in step S61, the video alignment processing unit 12 reads the video data of the teacher frame by frame from the teacher video storage unit , inputs each read frame into a convolutional neural network (CNN), and extracts the image feature amounts of the teacher video.

[0053] CNN is a method for extracting image features through convolution and downsampling operations. Specifically, the convolution unit first performs a convolution operation between a filter and the image to generate a "feature map." Then, the downsampling unit calculates the maximum value of the feature map for each block to reduce the image size. The filter weight coefficients are optimized through learning and determined based on the image data set so that important parts such as edges can be extracted.

[0054] CNN then performs downsampling on the feature map. CNN downsampling is a process that reduces the size of the feature map computationally, using either maximum pooling or average pooling.

[0055] Maximum pooling is a process that takes the maximum value within a block, thereby extracting the pixel information deemed most important. As a result, a feature vector with emphasized edge information is obtained. On the other hand, mean pooling is a process that calculates the average value within a block, averaging the feature components between edges and non-edges. Therefore, areas where edges are concentrated will have generally blurred outlines. Both downsampling methods can improve robustness to positional information by removing or averaging pixel information.

[0056] As described above, in CNN, the convolution unit extracts features while incorporating surrounding pixel information using a weighted filter, and the downsampling unit removes positional information. Therefore, this two-stage processing makes it possible to extract features that can be matched even from images with different orientations and sizes.

[0057] In step S62, the video alignment processing unit 12 inputs the image features of the training video extracted by the CNN as explanatory variables into a deep neural network (DNN) using deep learning. Then, in step S63, it obtains important image frames representing each moment during the training video's walking motion from the DNN, and in step S64, it stores these obtained important image frames in the aligned training video storage unit 33.

[0058] Similarly to the teacher video described above, in steps S61 to S64, the system extracts image features from the student video using a CNN, inputs the extracted image features as explanatory variables into a DNN, obtains important image frames representing each moment of the student's walking motion from the DNN, and stores them in the aligned student video storage unit 34.

[0059] (3) Comparison of important frames between teacher and student The control unit 1 of the MA motion information analysis device then, in step S6, under the control of the important frame comparison processing unit 13, performs a process to compare the aligned teacher video and the aligned student video and calculate the difference information.

[0060] In other words, the important frame comparison processing unit 13 first reads the aligned teacher video and aligned student video from the aligned teacher video storage unit 33 and the aligned student video storage unit 34, respectively, for the time position, i.e., the frame number, that corresponds to the aligned teacher video and aligned student video. That is, for both the teacher and the student, it reads the important image frames at each moment during walking motion that have been set in advance as the comparison target.

[0061] Next, the important frame comparison processing unit 13 inputs the read important image frames of the teacher and student to the VLM as a prompt and representing the processing instructions for each moment of the comparison target, and obtains information from the VLM representing the difference between the important image frames of the teacher and student. Then, it stores the obtained difference information in the difference information storage unit 35.

[0062] (4) Coaching Information Generation and Output The control unit 1 of the motion information analysis device MA reads the difference information from the difference information storage unit 35 in step S7 under the control of the coaching information generation processing unit 14. The coaching information generation processing unit 14 then generates coaching information to advise on foot movements to be paid attention to in order to reduce the difference, based on the difference information it has read. The coaching information may be, for example, text data or voice messages, or video information, or a combination of these.

[0063] The coaching information generation processing unit 14 outputs the coaching information acquired from the VLM to the display device DP via the input / output I / F unit 4. As a result, the coaching information is displayed on the display device DP, and students can refer to the displayed coaching information to perform training to improve their walking motion.

[0064] Furthermore, VLM may perform both the process of generating the difference information and the process of generating the coaching information described above.

[0065] (Effects) As described above, in one embodiment, first, video data representing the walking movements of the teacher and student is acquired from camera CM, and video alignment processing is performed to extract image frames of desired moments of walking movements from the acquired video data of the teacher and student. Next, the difference between the aligned teacher video and the student video is calculated, and based on the calculated difference information, coaching information is generated that includes advice on foot movements to pay attention to in order to eliminate the difference, and the generated coaching information is presented to the student.

[0066] Therefore, it becomes possible to calculate the difference in movement between the teacher and student at a desired moment in their walking motion, thereby enabling the teacher to advise the student on areas for improvement in their walking motion.

[0067] [Other Embodiments] (1) When extracting the characteristics of the movements, in addition to the movement characteristics of the student and the teacher, the timing of the extraction of these movement characteristics is detected, and the difference between each of the above movement characteristics and the time difference between the extraction timings of each of the above movement characteristics are determined. Then, the system may be configured to generate support information based on the difference between the above movement characteristics and the time difference between the extraction timings of the above movement characteristics.

[0068] For example, when extracting important frames for teachers and students, the system can detect the position of each extracted important frame in the original video, thereby detecting the walking tempo of each teacher and student, such as the time per step. By inputting the detected walking tempo information along with the important frames into the VLM, coaching information is generated that reflects not only the differences in movement at a desired moment in the walking motion, but also the differences in walking tempo. In this way, it becomes possible to provide more appropriate advice to students.

[0069] (2) In one embodiment, important frames representing the characteristics of movement at a desired moment in walking motion are extracted from video data visually representing the walking motion of the teacher and student, and coaching information is generated based on the differences in movement at the moment of walking motion using the extracted important frames.

[0070] However, this invention is not limited to that. For example, as shown in Figure 7, sensing data such as foot pressure at the moment of walking motion of the teacher and student, vital data such as heart rate of the teacher and student, and voice may be acquired, and motion features may be extracted from the acquired sensing data. Coaching information may then be generated by combining at least one of the extracted features with features extracted from video and inputting them into the VLM.

[0071] This approach is expected to generate higher-quality coaching information that reflects at least one of the various sensing data, such as foot pressure, vital data, and voice, in addition to the visual features of walking motion. Alternatively, motion acceleration may also be used as sensing data.

[0072] (3) In one embodiment, the function of the motion information analysis device MA was described using the example of a case where it is installed on a personal computer equipped with a training system, but it may also be installed on a smartphone, tablet, or wearable device used by a student or teacher, or on a server computer located on the Web or the cloud. Furthermore, the function of the motion information analysis device MA may be distributed across multiple personal computers or server computers.

[0073] (4) In one embodiment, the case of analyzing walking motion was described as an example, but it may also be used to analyze motions when performing other activities such as gymnastics, cooking, playing musical instruments, crafts, pottery, sculpture, carpentry, etc. Furthermore, the configuration, processing procedures, and processing content for realizing the processing functions of the motion information analysis device MA can be modified in various ways without departing from the spirit of this invention.

[0074] Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment.

[0075] In short, this invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments described above. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined.

[0076] MA... Motion Information Analysis Device CM... Camera DP... Display Device 1... Control Unit 2... Program Storage Unit 3... Data Storage Unit 4... Input / Output I / F Unit 5... Bus 11... Video Data Acquisition Processing Unit 12... Video Alignment Processing Unit 13... Important Frame Comparison Processing Unit 14... Coaching Information Generation Processing Unit 31... Teacher Video Storage Unit 32... Student Video Storage Unit 33... Aligned Teacher Video Storage Unit 34... Aligned Student Video Storage Unit 35... Difference Information Storage Unit

Claims

1. An action information analysis device comprising: a first processing unit that acquires first action information representing the actions of a student and second action information representing the actions of a teacher that serve as a model for the student's actions; a second processing unit that extracts first and second feature quantities representing the characteristics of actions at desired action timings from the first and second action information, respectively; and a third processing unit that obtains information representing the difference between the first and second feature quantities.

2. The motion information analysis device according to claim 1, further comprising a fourth processing unit that generates support information for assisting in correcting the student's actions based on the information representing the difference.

3. The operation information analysis device according to claim 1, wherein the second processing unit extracts operation feature quantities from the first operation information and the second operation information at predetermined unit time intervals, and extracts the first feature quantity and the second feature quantity by comparing each extracted operation feature quantity with a reference feature quantity prepared in advance corresponding to the desired operation timing.

4. The operation information analysis device according to claim 2, wherein the second processing unit detects the extraction timing of the first and second feature quantities in addition to the first and second feature quantities; the third processing unit obtains information representing the difference between the first and second feature quantities and information representing the time difference between the extraction timing of the first feature quantity and the extraction timing of the second feature quantity; and the fourth processing unit generates the support information based on the information representing the difference between the feature quantities and the information representing the time difference of the extraction timings.

5. The motion information analysis device according to claim 2, wherein the first processing unit acquires two or more sensing data as first motion information and second motion information, respectively, from among video footage of the student's and teacher's movements, pressure generated by the student's and teacher's movements, acceleration of the student's and teacher's movements, sound generated when the student and teacher perform the movements, and vital values ​​when the student and teacher perform the movements; the second processing unit extracts a first feature quantity and a second feature quantity from the two or more acquired sensing data; the third processing unit obtains information representing the difference between the first feature quantity and the second feature quantity for each of the two or more sensing data; and the fourth processing unit generates the support information by combining the information representing the difference obtained for each of the two or more sensing data.

6. An action information analysis method performed by an information processing device, comprising: a process of acquiring first action information representing a student's actions and second action information representing a teacher's actions that serve as a model for the student's actions; a process of extracting first and second feature quantities representing the characteristics of actions at desired timings from the first and second action information, respectively; and a process of obtaining information representing the difference between the first and second feature quantities.

7. A program that causes a processor in an operation information analysis device to execute a process performed by a processing unit in an operation information analysis device according to any one of claims 1 to 5.

Images