Information processing device, information processing method, and program

Abstract

An information processing device according to the present invention comprises a control unit. The control unit predicts the position of a moving body using a dynamic model that at least represents the relationship between the velocity vector of the moving body and interaction between the moving body and another moving body or an obstacle. The control unit estimates a first velocity vector of the moving body using a first model. The first model estimates the first velocity vector of the moving body on the basis of past observation data of the moving body. The control unit estimates a second velocity vector of the moving body using a second model. The second model has undergone machine learning using skeleton data of the moving body as training data. When the control unit determines that the moving body will make a sudden move, the control unit applies the second velocity vector to the dynamic model. When the control unit does not determine that the moving body will make a sudden move, the control unit applies the first velocity vector to the dynamic model.

Description

Information processing device, information processing method, and program Cross-reference to related applications

[0001] This application claims priority to Japanese Patent Application No. 2025-040298, filed in Japan on March 13, 2025, and the entire disclosure of the earlier application is incorporated herein by reference.

[0002] This disclosure relates to an information processing device, an information processing method, and a program.

[0003] Predicting the position of moving objects such as pedestrians is useful. For example, by sequentially predicting the position of a moving object, its trajectory can be predicted. If a pedestrian's trajectory is predicted, for example, a robot can move safely in cooperation with the pedestrian. Therefore, technologies for predicting the position of moving objects such as pedestrians have become known in recent years.

[0004] Patent Document 1 describes a moving object prediction device that predicts the future position of moving objects around a vehicle using the potential method.

[0005] Patent document 2 describes a tracking device that uses a Kalman filter to obtain an estimated value of the state of a target based on sensor observations.

[0006] Non-patent document 1 describes how to predict pedestrian trajectories using NSP (Neural Social Physics) with a deep neural network.

[0007] Non-patent document 2 describes learning the shape of the interaction potential acting on pedestrians in a Social Force Model (SFM).

[0008] Non-patent document 3 describes learning interactions between pedestrians to predict their trajectories.

[0009] Japanese Patent Publication No. 2018-124663 Japanese Patent Publication No. 2018-060326

[0010] Jiangbei Yue et al., “Human Trajectory Prediction via Neural Social Physics”, March 31, 2023 Sven Kreiss, “Deep Social Force”, September 24, 2021 Alexandre Alahi et al., “Social LSTM: Human Trajectory Prediction in Crowded Spaces”, “2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)”, June 27, 2016

[0011] An information processing device according to one embodiment of the present disclosure includes a control unit that predicts the position of a moving body by a dynamic model that represents at least the relationship between the interaction between the moving body and another moving body or obstacle and the velocity vector of the moving body, wherein the control unit estimates a first velocity vector of the moving body by a first model, the first model being a model that estimates the first velocity vector of the moving body based on past observation data of the moving body, estimates a second velocity vector of the moving body by a second model being a machine learning model that uses the skeletal data of the moving body as training data, applies the second velocity vector to the dynamic model when it is determined that the moving body is making a sudden movement, and applies the first velocity vector to the dynamic model when it is not determined that the moving body is making a sudden movement.

[0012] An information processing method according to one embodiment of the present disclosure includes: predicting the position of a moving body using a dynamic model that represents at least the relationship between the interaction between the moving body and another moving body or obstacle and the velocity vector of the moving body; estimating a first velocity vector of the moving body using a first model; estimating a second velocity vector of the moving body using a second model; applying the second velocity vector to the dynamic model if it is determined that the moving body is making a sudden movement; and applying the first velocity vector to the dynamic model if it is not determined that the moving body is making a sudden movement, wherein the first model is a model that estimates the first velocity vector of the moving body based on past observation data of the moving body, and the second model is a machine learning model that uses the skeletal data of the moving body as training data.

[0013] A program according to one embodiment of the present disclosure causes a computer to perform the following actions: predict the position of a moving body using a dynamic model that represents at least the relationship between the interaction between the moving body and other moving bodies or obstacles and the velocity vector of the moving body; estimate a first velocity vector of the moving body using a first model; estimate a second velocity vector of the moving body using a second model; apply the second velocity vector to the dynamic model if it is determined that the moving body is making a sudden movement; and apply the first velocity vector to the dynamic model if it is not determined that the moving body is making a sudden movement, wherein the first model is a model that estimates the first velocity vector of the moving body based on past observation data of the moving body, and the second model is a machine learning model that uses the skeletal data of the moving body as training data.

[0014] This figure shows an example of an information processing device and a mobile body according to one embodiment of the present disclosure. This is a block diagram of an example of the information processing device shown in Figure 1. This is a block diagram of an example of a second model. This figure shows an example of the trajectory of a mobile body. This figure shows an example of the trajectory of a mobile body. This is a diagram for explaining the first condition. This is a diagram for explaining the second condition. This is a flowchart showing an example of the operation of the information processing device shown in Figure 1. This is a diagram for explaining the dataset used in the experiment. This is a diagram for explaining the dataset used in the experiment. This is a figure showing the experimental results.

[0015] Incidentally, moving objects such as pedestrians can sometimes make sudden movements. For example, moving objects may suddenly start, suddenly stop, or suddenly turn. Even when moving objects make sudden movements, there is a need for technology that can accurately predict the position of a moving object. According to one embodiment of this disclosure, it is possible to provide technology that can accurately predict the position of a moving object even when moving objects make sudden movements. Hereinafter, embodiments of this disclosure will be described with reference to the drawings.

[0016] (Example of Information Processing Device Configuration) The information processing device 1 according to this embodiment predicts the position of a moving object 2 as shown in Figure 1. The information processing device 1 can predict the trajectory of the moving object 2 by sequentially predicting the position of the moving object 2. The moving object 2 is a moving object that can move autonomously. The moving object 2 may be, for example, a pedestrian, a robot, a delivery device, a vacuum cleaner, an animal, a bicycle, a vehicle such as a car or motorcycle, a drone, an autonomous vehicle, or a combination thereof. The robot may be, for example, a delivery robot that delivers luggage or goods. However, the moving object 2 may be any moving object as long as it can move autonomously. In Figure 1, moving objects 2A, 2B, 2C, and 2D are shown as moving objects 2. In Figure 1, the trajectory of the moving object 2 is shown by a dashed line. In Figure 1, the direction of travel of the moving object 2 is shown by an arrow. Moving objects 2A to 2C are pedestrians. Moving object 2D is a robot. Moving objects 2A and 2B are both walking. Figure 2 also shows an obstacle 3. The moving object 2 moves towards its destination while avoiding collisions with other moving objects 2 and obstacle 3.

[0017] The information processing device 1 may be any information processing device. In this embodiment, the information processing device 1 may be mounted on a robot 4 as shown in Figure 1. The information processing device 1 may be a dedicated device or a general-purpose device. The robot 4 may be, for example, a delivery robot that delivers luggage or goods. However, the information processing device 1 may be carried by a pedestrian. When the information processing device 1 is carried by a pedestrian, it may be a general-purpose terminal device such as a smartphone or a dedicated terminal device. The robot 4 can travel safely in cooperation with the mobile body 2 by the information processing device 1 predicting the trajectory of the mobile body 2. In addition to the robot 4 shown in Figure 1, the information processing device 1 may be mounted on a delivery device, a vacuum cleaner, a bicycle, a car or motorcycle or other vehicle, a drone or an autonomous vehicle, etc. The information processing device 1 of this disclosure does not have to be mounted on a robot 4, etc. as shown in Figure 1. In this case, the information processing device 1 of this disclosure may be connected to the robot 4, etc. as shown in Figure 1 by a wired, wireless, or combination thereof network. The information processing device 1 of this disclosure may send and receive information with the robot 4, etc. via the network. The information processing device 1 of this disclosure may predict the position of the mobile body 2 by sending and receiving information with a robot 4 or the like, or it may predict the trajectory of the mobile body 2 by sequentially predicting the position of the mobile body 2.

[0018] As shown in Figure 2, the information processing device 1 comprises a stereo camera 10, a storage unit 11, and a control unit 12.

[0019] The stereo camera 10 is positioned on the robot 4 so as to be able to capture images of the surrounding scenery. The stereo camera 10 generates two images by capturing images of the scenery around the robot 4. These two images are generated by capturing the same subject from different viewpoints. These two images are also called a "stereo image". Processing to correct geometric distortion of the stereo image may be performed on the stereo image. The stereo camera 10 generates the stereo image at a predetermined frame rate. The stereo camera 10 outputs the generated stereo image to the control unit 12.

[0020] In this embodiment, the control unit 12 analyzes a stereo image generated by the stereo camera 10, thereby acquiring position information of the moving object 2 and position information of the obstacle 3. However, the information processing apparatus 1 may acquire the position information of the moving object 2 and the position information of the obstacle 3 by any element other than the stereo camera 10. The information processing apparatus 1 may include, as any element other than the stereo camera 10 for acquiring the position information of the moving object 2 and the position information of the obstacle 3, millimeter wave radar, an infrared camera, LiDAR (Light Detection And Ranging), or a combination of at least two of these. When the information processing apparatus 1 acquires the position information of the moving object 2 and the position information of the obstacle 3 by an element other than the stereo camera 10, the control unit 12 may acquire the position information of the moving object 2 and the position information of the obstacle 3 from the element other than the stereo camera 10. The control unit 12 may use the position information of the moving object 2 and the position information of the obstacle 3 acquired from said element for predicting the position of the moving object 2.

[0021] The storage unit 11 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The storage unit 11 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 11 stores data used for the operation of the information processing apparatus 1 and data obtained by the operation of the information processing apparatus 1. A program executed by the control unit 12 may be stored in the storage unit 11.

[0022] The control unit 12 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 12 controls each part of the information processing device 1 and executes processes related to the operation of the information processing device 1.

[0023] The control unit 12 predicts the position of the moving body 2 using a dynamic model. This dynamic model represents at least the relationship between the interaction between the moving body 2 and other moving bodies 2 or obstacles and the velocity vector of the moving body 2. In this embodiment, this dynamic model is an SFM. In the SFM, the interaction between the moving body 2 and other moving bodies 2 or obstacles is represented by the repulsive force vector F, which will be described later. １ or repulsive force vector F ３ This is reflected in the control unit 12 predicts the trajectory of the moving body 2 by sequentially predicting the position of the moving body using SFM. In SFM, the movement of the moving body 2 is predicted based on the resultant force vector, which is the sum of three force vectors. In SFM, the equation of motion for the moving body 2 is given by equation (1). In equation (1), mass m is the weight of the moving body 2. The symbol with "→" above the symbol "v" is the velocity vector v of the moving body 2. In equation (1), the symbol "F １ The symbol with "→" above it represents the hypothetical repulsive force vector F that moving object 2 receives from another moving object 2. １ For example, if one moving object 2 tries to avoid another moving object 2, the repulsive force vector F １ This occurs. In equation (1), the symbol "F ２ The symbol with "→" above it represents the hypothetical gravitational force vector F acting on the moving object 2. ２ For example, when moving object 2 tries to move towards its destination, the gravitational vector F ２occurs. In formula (1), the symbol F ３ with an arrow "→" added above it is the virtual repulsive force vector F that the moving body 2 receives from an obstacle ３ . For example, when the moving body 2 attempts to avoid an obstacle, the repulsive force vector F ３ occurs.

[0024] The control unit 12 estimates the velocity vector v of the moving body 2 using a first model or a second model described below. The control unit 12 may calculate the repulsive force vector F of formula (1) １ , attractive force vector F ２ and repulsive force vector F ３ by a known method. The control unit 12 inputs the estimated velocity vector v into the SFM formula (1), and further inputs the calculated repulsive force vector F １ , attractive force vector F ２ and repulsive force vector F ３ into the SFM formula (1), thereby predicting the position of the moving body 2.

[0025] The first model estimates the velocity vector v of the mobile body 2 based on past observation data of the mobile body 2. The first model may estimate the velocity vector v and the velocity of the mobile body 2 based on past observation data of the mobile body 2, or it may estimate the velocity vector v or the velocity of the mobile body 2. The first model is a model that does not utilize machine learning. In this embodiment, the first model is a Kalman filter (KF). The first model, which is a Kalman filter, is also referred to as "first model (KF)". In this embodiment, the first model (KF) estimates the current velocity vector v of the mobile body 2 by estimating the current state of the mobile body 2 based on past observation data of the mobile body 2. The control unit 12 inputs position information of the mobile body 2 at multiple past points in time as past observation data of the mobile body 2 to the first model (KF), and the first model estimates the velocity vector v of the mobile body 2. The control unit 12 may estimate the velocity of the moving body 2 by inputting position information of the moving body 2 at multiple past points in time into the first model (KF). The control unit 12 may also estimate the velocity of the moving body 2 using the first model (KF) and obtain the velocity of the moving body 2 from the velocity vector v of the moving body 2 estimated by the first model. The position information of the moving body 2 input into the first model (KF) is, for example, the position information of the moving body 2 in three-dimensional space. The position information of the moving body 2 input into the first model (KF) may be the position information of any part of the moving body 2, such as the head. The part of the moving body 2 used for the position information of the moving body 2 may be set in advance. Hereinafter, the velocity vector v of the moving body 2 estimated by the first model will be referred to as the "first velocity vector v".

[0026] The second model is a model that utilizes machine learning. The second model is a model that has been trained using the skeletal data of the mobile body 2 as training data. In this embodiment, the second model is a model that utilizes ST-GCN (Spatio-Temporal Graph Convolutional Networks) as machine learning. ST-GCN estimates the future skeletal data of the mobile body 2 based on the skeletal data of the mobile body 2 at multiple past points in time. By estimating the future skeletal data of the mobile body 2 using ST-GCN, the second model can estimate the future velocity vector v of the mobile body 2. The second model may estimate the future velocity vector v and the future velocity of the mobile body 2 by estimating the future skeletal data of the mobile body 2 using ST-GCN, or it may estimate the future velocity vector v or the future velocity of the mobile body 2. The second model may be a machine learning model, i.e., a trained model, that outputs the future velocity vector v of the mobile body 2 when the skeletal data of the mobile body 2 is input. In this case, the control unit 12 may estimate the future velocity of the moving body 2 using the second model and obtain the future velocity of the moving body 2 from the future velocity vector v of the moving body 2 output from the second model. Hereinafter, the second model utilizing ST-GCN will also be referred to as "second model (ST-GCN)".

[0027] Figure 3 shows an example of a second model 20 utilizing ST-GCN. In this embodiment, the control unit 12 may use the second model 20 shown in Figure 3 as the second model (ST-GCN). In Figure 3, the control unit 12 inputs skeletal data 21 of the mobile body 2 at multiple past points in time to the second model 20. The skeletal data 21 of the mobile body 2 is two-dimensional data. The skeletal data 21 of the mobile body 2 includes, for example, the coordinates of multiple joints of the mobile body 2. The control unit 12 obtains the skeletal data 21 of the mobile body 2 at multiple past points in time by analyzing stereo images of multiple frames generated by the stereo camera 10. For example, the control unit 12 obtains the skeletal data 21 of the mobile body 2 at five past points in time by analyzing stereo images of five frames. For example, the skeletal data 21 of the mobile body 2 at multiple past points in time is data at 0.3-second intervals. The second model 20 includes a BN (Batch Normalization) 23, an ST-GCN, an AP (Average Pooling) layer 27, and a vector prediction unit 28.

[0028] The skeletal data 21 of the mobile body 2 at multiple past points in time, which is input to the second model 20, is input to BN23. BN23 performs batch normalization on the skeletal data 21. The data output from BN23 is input to the ST-GCN of the second model 20.

[0029] The ST-GCN of the second model 20 includes an SC (Spatial Convolution) layer 24, a TC (Temporal Convolution) layer 25, and a residual connector 26. In the ST-GCN of the second model 20 shown in Figure 3, the configuration including the SC layer 24, the TC layer 25, and the residual connector 26 is repeated nine times. First, the data input to the ST-GCN from BN23 is input to the SC layer 24. The SC layer 24 performs spatial feature convolution on the data input to the ST-GCN. The data output from the SC layer 24 is input to the TC layer 25. The TC layer 25 performs temporal feature convolution on the data input from the SC layer 24. The data output from the TC layer 25 is input to the residual connector. The residual connector 26 suppresses excessive smoothing of features by adding the feature quantities of the data input from BN23 to the ST-GCN with the feature quantities convolved by the TC layer 25. In the ST-GCN shown in Figure 3, these processes by the SC layer 24, TC layer 25, and residual connector 26 are repeated nine times.

[0030] The data output from the ST-GCN of the second model 20 is converted into a feature map through feature convolution. The feature map is input to the AP layer 27. The AP layer 27 performs average pooling on the feature map. The feature map output from the AP layer 27 is input to the vector prediction unit 28.

[0031] The vector prediction unit 28 receives a feature map from the AP layer 27. Based on the feature map, the vector prediction unit 28 predicts (estimates) the future velocity vector v of the moving object 2. For example, suppose the skeletal data 21 of the moving object 2 at multiple past points in time, which is input to the second model 20, is data at 0.3-second intervals. In this case, the vector prediction unit 28 predicts (estimates) the velocity vector v of the moving object 2 0.6 seconds from the present as its future velocity vector v.

[0032] Hereinafter, the velocity vector v of the moving object 2 estimated by the second model will be referred to as "second velocity vector v". In this embodiment, the second velocity vector v may be the velocity vector v estimated by the second model 20 shown in Figure 3.

[0033] Incidentally, the second velocity vector v estimated by the second model is the future velocity vector v of the moving body 2. Therefore, the information processing device 1 according to this embodiment can also predict the trajectory of the moving body 2 by sequentially estimating the future velocity vector v of the moving body 2, i.e., the second velocity vector v, using the second model. However, unlike dynamic models such as SFM, the second model does not consider the interaction between the moving body 2 and other moving bodies 2 or obstacles. Also, there is a time lag between the time an interaction occurs between the moving body 2 and another moving body 2 or obstacle and the movement of the moving body 2 due to that interaction is reflected in the movement of the skeletal data of the moving body 2. Therefore, if another moving body 2 or obstacle is present near the moving body 2, the second model cannot accurately predict the trajectory of the moving body 2. For example, in Figure 4, an obstacle 3 is present in the direction of travel of the moving body 2. In this case, the moving body 2 will try to avoid the obstacle 3. Therefore, a virtual repulsive force is generated that the moving body 2 receives from the obstacle 3. The second model does not consider this virtual repulsive force that the moving body 2 receives from the obstacle 3. Therefore, the trajectory of the moving object 2 estimated by the second model collides with obstacle 3. In Figure 4, the trajectory of the moving object 2 estimated by the second model is indicated by the label "Trajectory by second model (ST-GCN)". In contrast, SFM takes into account the interaction between the moving object 2 and other moving objects 2 or obstacles, as can be seen from the third term on the right-hand side of equation (1). Therefore, using SFM, the trajectory of the moving object 2 can be predicted more accurately than with the second model when there is an obstacle 3 in the direction of the moving object 2's movement, as shown in Figure 4. In Figure 4, the case where the first velocity vector v of the moving object 2 from the first model (KF) is substituted into equation (1) of SFM to predict the trajectory of the moving object 2 is indicated by the label "Trajectory by first model (KF) and SFM". Also in Figure 4, the correct trajectory of the moving object 2 is indicated by the label "Correct trajectory".

[0034] Next, the information processing device 1 according to this embodiment can also predict the trajectory of the moving body 2 by continuously substituting the first velocity vector v of the moving body 2 according to the first model (KF) into equation (1) of the SFM. Here, the moving body 2 may make sudden movements of its own volition, regardless of other moving bodies 2 or obstacles. For example, the moving body 2 may suddenly start, suddenly stop, or suddenly turn. When the moving body 2 makes such sudden movements, the estimation accuracy of the first velocity vector v of the moving body 2 according to the first model (KF) deteriorates. The reason for this is that, as described above, the first model (KF) estimates the current velocity vector v of the moving body 2 based on past observation data of the moving body 2. The past observation data of the moving body 2 does not reflect, for example, the sudden movements of the moving body 2 when it makes sudden movements of its own volition, regardless of other moving bodies 2 or obstacles. Therefore, when the moving object 2 makes a sudden movement, the accuracy of the estimation of the first velocity vector v of the moving object 2 by the first model (KF) deteriorates. Consequently, as shown in Figure 5, when the moving object 2 makes a sudden movement, substituting the first velocity vector v of the moving object 2 from the first model (KF) into equation (1) of SFM to predict the trajectory of the moving object 2 deteriorates. In Figure 5, the case where the trajectory of the moving object 2 is predicted by substituting the first velocity vector v of the moving object 2 from the first model (KF) into equation (1) of SFM is indicated by the label "Trajectory by the first model (KF) and SFM". In contrast, the second model (ST-GCN) estimates the second velocity vector v of the moving object 2 based on the skeletal data of the moving object 2. Sudden movements of the moving object 2 are reflected in minute changes in the past skeletal data of the moving object 2. In other words, the second model (ST-GCN) can estimate the second velocity vector v of the moving object 2 based on minute changes in the past skeletal data of the moving object 2. Therefore, when the moving object 2 makes a sudden movement, the second model (ST-GCN), unlike the first model (KF), can estimate the second velocity vector v of the moving object 2 with greater accuracy. As a result, when the moving object 2 makes a sudden movement, substituting the second velocity vector v from the second model (ST-GCN) into equation (1) of the SFM allows for a more accurate prediction of the trajectory of the moving object 2 than substituting the first velocity vector v from the first model (KF).In Figure 5, the trajectory of the moving object 2 estimated by the second model is labeled "Trajectory by the second model (ST-GCN)". Also in Figure 5, the correct trajectory of the moving object 2 is labeled "Correct trajectory".

[0035] Furthermore, the information processing device 1 according to this embodiment can also predict the trajectory of the moving object 2 by continuously substituting the second velocity vector v of the moving object 2 by the second model (ST-GCN) into equation (1) of the SFM. According to this method, the interaction between the moving object 2 and other moving objects 2 or obstacles is taken into consideration by the SFM. In addition, even if the moving object 2 makes a sudden movement, the second velocity vector v estimated with high accuracy by the second model (ST-GCN) is applied to the SFM. However, in order to predict the trajectory of the moving object 2 by continuously substituting the second velocity vector v of the moving object 2 by the second model (ST-GCN) into equation (1) of the SFM, the second model must be trained with high accuracy. For example, the moving object 2 makes a variety of movements. That is, the moving object 2 makes not only sudden movements but also slow movements. In order to accurately estimate the velocity vector v of the moving object 2 that makes such a variety of movements, the second model must be trained with high accuracy. This is time-consuming.

[0036] Therefore, if the control unit 12 determines that the moving body 2 is making a sudden movement, it decides to apply the second velocity vector v of the moving body 2 according to the second model (ST-GCN) to the SFM. Furthermore, if the control unit 12 does not determine that the moving body 2 is making a sudden movement, that is, if it determines that the moving body 2 is not making a sudden movement, it decides to apply the first velocity vector v of the moving body 2 according to the first model (KF) to the SFM. With this configuration, the velocity vector v applied to the SFM is selected from the first velocity vector v and the second velocity vector v depending on whether or not the moving body 2 is making a sudden movement. As a result, the position of the moving body 2 can be predicted with high accuracy. In this embodiment, as a determination of whether or not the moving body 2 is making a sudden movement, the control unit 12 determines whether or not either of the following first and second conditions is met. If the control unit 12 determines that either of the following first and second conditions is met, it determines that the moving body 2 is making a sudden movement.

[0037] <Condition 1> The first condition is that the absolute value α of the current acceleration of the moving body 2 is greater than or equal to the acceleration threshold. When the first condition is met, it corresponds to the case where the moving body 2 is suddenly accelerating or suddenly stopping. The acceleration threshold may be determined experimentally. Here, as shown in Figure 6, when the moving body 2 is suddenly accelerating or suddenly stopping, the past position of the moving body 2 changes rapidly. As described above, the first model (KF) can estimate the velocity of the moving body 2 based on the position information of the moving body 2 at multiple past points in time. Therefore, the control unit 12 may calculate the absolute value α of the current acceleration of the moving body 2 using the velocities of the moving body 2 at multiple points in time estimated by the first model (KF) based on the position information of the moving body 2 at multiple past points in time. For example, the control unit 12 calculates the absolute value α of the current acceleration of the moving body 2 using the current velocity of the moving body 2 estimated by the first model (KF) and the past velocity of the moving body 2 estimated by the first model (KF). As an example of this process, the control unit 12 calculates a velocity difference, which is the value obtained by subtracting the past velocity of the moving body 2, estimated by the first model (KF), from the current velocity of the moving body 2, estimated by the first model (KF). The control unit 12 calculates a first time difference, which is the value obtained by subtracting the second time, when the past velocity of the moving body 2 is observed, from the first time, when the current velocity of the moving body 2 is observed. The control unit 12 may estimate each of the first and second time points based on data input to the first model (KF). The first time, when the current velocity of the moving body 2 is observed, may be the current time. The control unit 12 calculates the current acceleration of the moving body 2 by dividing the velocity difference by the first time difference, and calculates the absolute value α of the current acceleration of the moving body 2.

[0038] The control unit 12 may calculate the absolute value α of the current acceleration of the moving body 2 using the current first velocity vector v of the moving body 2 estimated by the first model (KF) and the past first velocity vector v of the moving body 2 estimated by the first model (KF). As an example of this process, the control unit 12 calculates a velocity vector difference, which is a vector obtained by subtracting the past first velocity vector v of the moving body 2 from the current first velocity vector v of the moving body 2. The control unit 12 calculates a second time difference, which is the value obtained by subtracting the past first velocity vector v of the moving body 2 from the third time when the current first velocity vector v of the moving body 2 is observed. The control unit 12 may estimate the third time and the fourth time based on the data input to the first model (KF). The third time when the first velocity vector v is estimated may be the current time. The control unit 12 calculates the current acceleration vector of the moving body 2 by dividing the velocity vector difference by the second time difference, and calculates the absolute value α of the current acceleration vector of the moving body 2. The control unit 12 calculates the absolute value α of the current acceleration of the moving body 2 by, for example, equation (2). In equation (2), the symbol "v Ｎ The symbol with "→" above it represents the first velocity vector v Ｎ Therefore, the first velocity vector v Ｎ This is the current first velocity vector v of the moving object 2 estimated by the first model (KF). Ｎ The unit is [m / s]. In equation (2), the symbol "v Ｎ－４ The symbol with "→" above it represents the first velocity vector v Ｎ－４ Therefore, the first velocity vector v Ｎ－４ This is the first velocity vector v of the moving object 2 in the past, as estimated by the first model (KF). Ｎ－４ This is the first velocity vector v Ｎ This is the first velocity vector v from four frames prior. Ｎ－４The unit is [m / s]. In equation (2), the frame rate fps is the frequency at which the first model (KF) estimates the first velocity vector v of the moving object 2. In this embodiment, the position information of the moving object 2 obtained by analyzing the stereo image generated by the stereo camera 10 is input to the first model (KF). Therefore, the frame rate fps in equation (2) corresponds to the frame rate at which the stereo camera generates the captured image.

[0039] Here, in equation (2), the current first velocity vector v of the moving object 2 is given by Ｎ And the first velocity vector v Ｎ The first velocity vector v from 4 frames prior Ｎ－４ The absolute value α of the current acceleration of the moving object 2 is calculated by this. However, the past first velocity vector v of the moving object used to calculate the absolute value α of the current acceleration of the moving object 2 is the current first velocity vector v of the moving object 2. Ｎ The first velocity vector v from 4 frames prior Ｎ－４ It is not limited to this. The past first velocity vector v of the moving body 2 used to calculate the absolute value α of the current acceleration of the moving body 2 may be the velocity of the moving body 2 from 5 frames or more ago or 3 frames or less later.

[0040] <Second Condition> The second condition is that the angle θ between the current first velocity vector v of the moving body 2 according to the first model (KF) and the future second velocity vector v of the moving body 2 according to the second model (ST-GCN) is greater than or equal to an angle threshold. The angle threshold may be determined experimentally. When the second condition is met, it corresponds to the case where the moving body 2 makes a sharp turn. Here, as shown in Figure 7, when the moving body 2 makes a sharp turn, the direction of the velocity vector of the moving body 2 changes abruptly from the present to the future. Therefore, it is possible to evaluate whether the moving body 2 will make a sharp turn or not based on the current first velocity vector v of the moving body 2 and the future second velocity vector v of the moving body 2. The control unit 12 calculates the angle θ by, for example, equation (3). In equation (3), the symbol "v ＫＮ The symbol with "→" above it is the current first velocity vector v of the moving object 2 estimated by the first model (KF). In equation (3), the symbol "v ＮＮThe symbol with "→" above it represents the future second velocity vector v of the moving object 2, as estimated by the second model (ST-GCN).

[0041] (Example of operation of the information processing device) Figure 8 is a flowchart showing an example of the operation of the information processing device 1 shown in Figure 1.

[0042] In step S1, the control unit 12 acquires the position information of the moving object 2 at multiple past points in time. For example, the control unit 12 acquires the position information of the moving object 2 at multiple past points in time from stereo images of multiple frames generated by the stereo camera 10.

[0043] In step S2, the control unit 12 generates skeletal data of the moving body 2 at multiple past points in time by analyzing the stereo images of multiple frames generated by the stereo camera 10. The control unit 12 acquires the skeletal data of the moving body 2 at multiple past points in time by generating it.

[0044] In step S3, the control unit 12 estimates the current first velocity vector v of the moving body 2 using the first model (KF). For example, the control unit 12 estimates the current first velocity vector v of the moving body 2 by inputting the position information of the moving body 2 at multiple past points in time obtained in step S1 into the first model (KF). Here, this current first velocity vector v of the moving body 2 is the first velocity vector v in equation (2) in the first condition of step S5 described later. Ｎ It can be used for this purpose. Also, in step S3, the control unit 12, in addition to the current first velocity vector v of the moving body 2, considers the first velocity vector v of equation (2) in the first condition of step S5 described later. Ｎ－４The control unit 12 may also estimate the past first velocity vector of the moving body 2 for use in the calculation. In this case, in step S1, the control unit 12 may acquire position information of the moving body 2 at multiple past points in time for estimating the past velocity of the moving body 2 using the first model (KF). The control unit 12 may calculate the absolute value α of the current acceleration of the moving body 2 in the first condition of step S5 described later, using the current velocity of the moving body 2 and the past velocity of the moving body 2. In this case, in step S3, the control unit 12 may estimate the current velocity of the moving body 2 and the past velocity of the moving body 2 using the first model (KF), in addition to the current first velocity vector v of the moving body 2.

[0045] In step S4, the control unit 12 estimates the future second velocity vector v of the mobile body 2 using the second model (ST-GCN). For example, the control unit 12 inputs the skeletal data of the mobile body 2 at multiple past points in time, acquired in step S2, into the second model (ST-GCN). The control unit 12 inputs the skeletal data of the mobile body 2 at multiple past points in time into the second model and obtains the second velocity vector v of the mobile body 2 output from the second model. By obtaining the second velocity vector v of the mobile body 2 output from the second model, the control unit 12 estimates the future second velocity vector v of the mobile body 2 using the second model (ST-GCN).

[0046] In step S5, the control unit 12 determines whether the moving body 2 makes a sudden movement. In this embodiment, the control unit 12 determines whether the moving body 2 makes a sudden movement by determining whether either the first condition or the second condition is met. If the control unit 12 determines that neither the first condition or the second condition is met, it does not determine that the moving body 2 makes a sudden movement (step S5: NO). In other words, if the control unit 12 determines that neither the first condition or the second condition is met, it determines that the moving body 2 does not make a sudden movement (step S5: NO). If the control unit 12 determines that either the first condition or the second condition is met, it determines that the moving body 2 makes a sudden movement (step S5: YES). If the control unit 12 does not determine that the moving body 2 makes a sudden movement (step S5: NO), it proceeds to step S6. If the control unit 12 determines that the moving body 2 makes a sudden movement (step S5: YES), it proceeds to step S7.

[0047] In step S6, the control unit 12 decides to apply the current first velocity vector v of the moving body 2, which was estimated in step S3, to the SFM. The control unit 12 may select the first velocity vector of the moving body 2 from the first velocity vector v and the second velocity vector of the moving body 2 as the velocity vector to be applied to the SFM.

[0048] In step S7, the control unit 12 decides to apply the future second velocity vector v of the moving body 2, which was estimated in step S4, to the SFM. The control unit 12 may select the second velocity vector of the moving body 2 from the first velocity vector v and the second velocity vector of the moving body 2 as the velocity vector to be applied to the SFM.

[0049] In step S8, the control unit 12 predicts the position information of the moving object 2 from the present to a predetermined time later using the SFM. For example, the control unit 12 predicts the position information of the moving object 2 3 seconds later using the SFM. For example, in step S8, the control unit 12 calculates the repulsive force vector F of equation (1) １ , gravitational vector F ２ and repulsive force vector F ３ This is calculated by a known method. If the process proceeds from step S6 to step S8, the control unit 12 inputs the current first velocity vector v of the moving body 2 estimated in step S3 into equation (1) of the SFM, and further calculates the repulsive force vector F １ , gravitational vector F ２ and repulsive force vector F ３ The following is input into equation (1) of the SFM. If the process proceeds from step S7 to step S8, the control unit 12 inputs the future second velocity vector v of the moving body 2 estimated in step S4 into equation (1) of the SFM, and further inputs the calculated repulsive force vector F １ , gravitational vector F ２ and repulsive force vector F ３ Enter this into the SFM formula (1).

[0050] In step S9, the control unit 12 outputs the position information of the mobile object 2 predicted in step S8 to the control device 5. As a result of step S9 being executed, the control device 5 acquires the predicted position information of the mobile object 2. Based on the predicted position information of the mobile object 2, the control device 5 can safely move the robot 4 in cooperation with the mobile object 2, such as a pedestrian.

[0051] After processing in step S9, the control unit 12 returns to step S1.

[0052] Here, the first velocity vector v of the moving body 2 estimated in step S3 is estimated by the Kalman filter of the first model to become the current velocity vector v of the moving body 2. The second velocity vector v of the moving body 2 estimated in step S4 is estimated by the ST-GCN of the second model to become the future velocity vector v of the moving body 2. For example, the second velocity vector v is the velocity vector v 0.6 seconds from the present. Depending on the processing result of step S5, it is determined whether the current first velocity vector v of the moving body 2 is applied to the SFM (step S6) or the future second velocity vector v of the moving body 2 is applied to the SFM (step S7). In step S8, the position information of the moving body 2 after a predetermined time has elapsed from the present is obtained. This predetermined time is, for example, 3 seconds. This predetermined time is sufficiently longer than the future when the second velocity vector v of the moving body 2 is estimated (for example, 0.6 seconds from the present). Therefore, there is no difference in the result of step S8 regardless of whether step S6 or step S7 is performed.

[0053] In step S3, the control unit 12 may use the position information of the moving body 2 at multiple past points in time as the position information of the center of gravity of the moving body 2 at multiple past points in time, which is input to the first model (KF). This position information of the center of gravity of the moving body 2 input to the first model (KF) is the position information of the center of gravity of the moving body 2 in three-dimensional space. In this case, the control unit 12 detects the position of the center of gravity of the moving body 2 based on the skeletal data of the moving body 2 acquired in step S2. Next, the control unit 12 generates a disparity image from the stereo image generated by the stereo camera 10. The control unit 12 obtains the position information of the center of gravity of the moving body 2 in three-dimensional space by comparing the detected center of gravity of the moving body 2 with the disparity image. Similarly or in a similar manner, in step S4, the control unit 12 may use the position information of the moving body 2 at multiple past points in time as the position information of the center of gravity of the moving body 2 at multiple past points in time, which is input to the second model (ST-GCN).

[0054] Before step S5, the control unit 12 may determine whether the distance from the stereo camera 10 to the moving object 2 is less than or equal to a distance threshold. The control unit 12 may calculate the distance from the stereo camera 10 to the moving object 2 by analyzing the stereo image generated by the stereo camera 10. The distance threshold may be set according to the accuracy of the stereo camera 10. If the distance from the stereo camera 10 to the moving object 2 is less than or equal to the distance threshold, the control unit 12 may proceed to step S5. If the distance from the stereo camera 10 to the moving object 2 exceeds the distance threshold, the control unit 12 does not need to proceed to step S5. In other words, if the distance from the stereo camera 10 to the moving object 2 exceeds the distance threshold, the control unit 12 does not need to predict the position of the moving object 2 using SFM. In this embodiment, the first velocity vector v applied to SFM uses position information of the moving object 2 at multiple past points in time obtained from the stereo image generated by the stereo camera 10. Therefore, if the distance from the stereo camera 10 to the moving object 2 is large, the estimation accuracy of the first velocity vector v of the moving object 2 may deteriorate. Accordingly, the control unit 12 proceeds to step S5 if the distance from the stereo camera 10 to the moving object 2 is below a distance threshold. With this configuration, moving objects 2 for which the estimation accuracy of the first velocity vector v is considered to deteriorate can be excluded from the application of SFM.

[0055] Before step S5, the control unit 12 may determine whether the speed of the moving body 2 exceeds a speed threshold. This speed of the moving body 2 may be the current speed of the moving body 2 estimated by the first model (KF) in the preceding step S3. The speed threshold may be set according to the accuracy of the stereo camera 10. If the speed of the moving body 2 exceeds the speed threshold, the control unit 12 may proceed to step S5. If the speed of the moving body 2 is below the speed threshold, the control unit 12 does not need to proceed to step S5. In other words, if the speed of the moving body 2 is below the speed threshold, the control unit 12 does not need to predict the position of the moving body 2 using the SFM. Here, if the speed of the moving body 2 is below the speed threshold, the estimation accuracy of the first velocity vector v or the second velocity vector v may deteriorate. Therefore, if the speed of the moving body 2 exceeds the speed threshold, the control unit 12 proceeds to step S5. This configuration allows us to exclude moving objects 2, for which the estimation accuracy of the first or second velocity vector v is considered to be degraded, from the target to which SFM is applied.

[0056] (Experimental Results) The inventors conducted experiments to demonstrate the effects of this disclosure. First, the dataset used in the experiments will be described with reference to Figures 9 and 10.

[0057] In the experiment, the subjects performed various walking patterns as pedestrians. The types of walking performed by the pedestrians (subjects) included walking without sudden changes in trajectory, as shown in Figure 9, and walking with sudden changes in trajectory, as shown in Figure 10.

[0058] As shown in Figure 9, data for pedestrians without sudden changes in trajectory includes data for pedestrians walking in a straight line, data for pedestrians walking diagonally, and data for pedestrians walking laterally. Data for pedestrians walking in a straight line is from images with 118 frames. Data for pedestrians walking diagonally is from images with 103 frames. Data for pedestrians walking laterally is from images with 56 frames.

[0059] As shown in Figure 10, data for pedestrians with sudden changes in trajectory include data for pedestrians turning 90° and walking, data for pedestrians turning 45° and walking, and data for pedestrians suddenly stopping and then suddenly starting. The data for pedestrians turning 90° and walking is from images with 26 frames. The data for pedestrians turning 45° and walking is from images with 27 frames. The data for pedestrians suddenly stopping and then suddenly starting is from images with 22 frames.

[0060] The inventors set the following experimental conditions: • The absolute value α of the current acceleration of moving object 2 (a pedestrian in the experiment) in the first condition is calculated using the pedestrian's current velocity and the pedestrian's velocity four frames prior. • The acceleration threshold in the first condition is 0.25 m / s². ２ The following will be determined. - The angle threshold for the second condition will be 10°. - Pedestrians within 11m of the point where the image was captured will be subject to determination under both the first and second conditions. This is to reduce noise. - Pedestrians whose current speed is 0.05m / s or less will be excluded from determination under both the first and second conditions. This is to reduce noise.

[0061] Figure 11 shows the experimental results. In Figure 11, the number of times a correct judgment was made is given by the number of frames. When the type of walking is "no sudden changes in trajectory," the number of times neither the first nor the second condition is met is the number of times. In other words, when the type of walking is "no sudden changes in trajectory," the pedestrian (subject) does not make sudden movements, so the correct judgment result is that neither the first nor the second condition is met. When the type of walking is "sudden changes in trajectory (turning)," the number of times the second condition is met is the number of times. When the type of walking is "sudden changes in trajectory (sudden stops and sudden starts)," the number of times the first condition is met is the number of times.

[0062] Figure 11 shows the average prediction error [m] of the pedestrian's position after 3 seconds. "KF" is the case when the velocity vector estimated by the Kalman filter of the first model is applied to the SFM. "NN" is the case when the velocity vector estimated by the second model (ST-GCN) is applied to the SFM. "This Disclosure" is the case when the first velocity vector or the second velocity vector is applied to the SFM depending on the determination results of the first and second conditions. In the results shown in Figure 11, the inventors used the position information of the pedestrian's center of gravity as the pedestrian's position information.

[0063] When the type of walking was "no sudden trajectory change," the results of this disclosure were comparable to "KN." Furthermore, when the type of walking was "sudden trajectory change (turning)" or "sudden trajectory change (sudden stop / sudden start)," the results of this disclosure were comparable to "NN." From these results, it can be seen that this disclosure can accurately detect the position of a pedestrian regardless of whether the type of walking is "no sudden trajectory change," "sudden trajectory change (turning)," or "sudden trajectory change (sudden stop / sudden start)."

[0064] In the information processing device 1 according to this embodiment, if the control unit 12 determines that the moving body 2 is making a sudden movement, it applies the second velocity vector v estimated by the second model to the dynamic model. If the control unit 12 does not determine that the moving body 2 is making a sudden movement, it applies the first velocity vector v estimated by the first model to the dynamic model. With this configuration, the position of the moving body 2 can be predicted with high accuracy even when the moving body 2 is making a sudden movement.

[0065] While embodiments of this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art will find it easy to make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component or process can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated. Embodiments relating to this disclosure can also be realized as programs executed by a processor in the device or as storage media recording such programs. These should also be understood to be included within the scope of this disclosure.

[0066] For example, in the information processing device 1 according to the above-described embodiment, the control unit 12 was described as determining whether or not the moving body 2 makes a sudden movement. However, instead of determining whether or not the moving body 2 makes a sudden movement, or as if determining whether or not the moving body 2 makes a sudden movement, the control unit 12 may determine whether or not a value related to the movement of the moving body 2 is greater than or equal to a determination threshold. The control unit 12 may calculate the value related to the movement of the moving body 2 using at least one of the first model (KF) and the second model (ST-GCN). The control unit 12 may calculate the value related to the movement of the moving body 2 using at least one of the first velocity vector and velocity of the moving body 2 estimated by the first model (KF) and the second velocity vector and velocity of the moving body 2 estimated by the second model (ST-GCN). A larger value related to the movement of the moving body 2 may indicate that the movement of the moving body 2 is sudden. The determination threshold may be set according to the type of data used to calculate the value related to the movement of the moving body 2. The control unit 12 determines that the moving body 2 is making a sudden movement if it determines that the value related to the movement of the moving body 2 is greater than or equal to a determination threshold. The control unit 12 does not determine that the moving body 2 is making a sudden movement if it determines that the value related to the movement of the moving body 2 is less than the determination threshold. In other words, the control unit 12 determines that the moving body 2 is not making a sudden movement if it determines that the value related to the movement of the moving body 2 is less than the determination threshold. A person skilled in the art will understand that examples of values ​​related to the movement of the moving body 2 are the absolute value α in the first condition and the angle θ in the second condition described above. If the value related to the movement of the moving body 2 is the absolute value α of the current acceleration of the moving body 2 in the first condition described above, the determination threshold may be an acceleration threshold. If the value related to the movement of the moving body 2 is the angle θ between the current first velocity vector v of the moving body 2 according to the first model (KF) and the future second velocity vector v of the moving body 2 according to the second model (ST-GCN) in the second condition described above, the determination threshold may be an angle threshold.

[0067] For example, in the embodiment described above, the information processing device 1 was described as being mounted on the robot 4. However, the information processing device 1 does not have to be mounted on the robot 4. As another example, the information processing device 1 may be a server. In this case, the information processing device 1 may receive stereo images or captured images from a stereo camera 10 or surveillance camera, etc., provided on the robot 4. The information processing device 1 may predict the position of the moving body 2 using the received stereo images or captured images in the same or similar manner as in the embodiment described above.

[0068] For example, it is also possible to implement an embodiment in which a general-purpose computer functions as the information processing device 1 according to the above embodiment. Specifically, a program describing the processing content that realizes each function of the information processing device 1 according to the above embodiment is stored in the memory of the general-purpose computer, and the processor reads and executes the program. Therefore, this disclosure can also be implemented as a program that can be executed by a processor, or as a non-temporary computer-readable medium that stores the program.

[0069] In one embodiment, (1) the information processing device includes a control unit that predicts the position of a moving body by a dynamic model that represents at least the relationship between the interaction between the moving body and another moving body or obstacle and the velocity vector of the moving body, the control unit estimates a first velocity vector of the moving body by a first model, the first model being a model that estimates a first velocity vector of the moving body based on past observation data of the moving body, the control unit estimates a second velocity vector of the moving body by a second model being a machine learning model that uses the skeletal data of the moving body as training data, and if it is determined that the moving body is making a sudden movement, the control unit applies the second velocity vector to the dynamic model, and if it is not determined that the moving body is making a sudden movement, the control unit applies the first velocity vector to the dynamic model.

[0070] (2) In the information processing device described in (1) above, the first model is a model that does not utilize machine learning.

[0071] (3) In the information processing apparatus described in (1) or (2) above, the first model is a Kalman filter.

[0072] (4) In the information processing device described in any one of (1) to (3) above, the second model is a model that utilizes ST-GCN.

[0073] (5) In the information processing device described in any one of (1) to (4) above, the control unit calculates the absolute value of the current acceleration of the moving body based on the velocity of the moving body at multiple points in time, and determines that the moving body is making a sudden movement if the absolute value of the current acceleration of the moving body is equal to or greater than an acceleration threshold.

[0074] (6) In the information processing device described in any one of (1) to (5) above, the control unit determines that the moving body is making a sudden movement if the angle between the current first velocity vector of the moving body and the future second velocity vector of the moving body is greater than or equal to an angle threshold.

[0075] (7) In the information processing device described in any one of (1) to (6) above, the dynamic model is an SFM.

[0076] (8) In the information processing device described in any one of (1) to (7) above, the control unit generates past observation data of the moving object by analyzing the captured image generated by the stereo camera, and determines whether the moving object is making a sudden movement if the distance from the stereo camera to the moving object is less than or equal to a distance threshold.

[0077] (9) In the information processing device described in any one of (1) to (8) above, the control unit determines whether the moving body makes a sudden movement when the speed of the moving body exceeds a speed threshold.

[0078] In one embodiment, (10) the information processing method includes: predicting the position of a moving body using a dynamic model that represents at least the relationship between the interaction between the moving body and another moving body or obstacle and the velocity vector of the moving body; estimating a first velocity vector of the moving body using a first model; estimating a second velocity vector of the moving body using a second model; applying the second velocity vector to the dynamic model if it is determined that the moving body is making a sudden movement; and applying the first velocity vector to the dynamic model if it is not determined that the moving body is making a sudden movement, wherein the first model is a model that estimates the first velocity vector of the moving body based on past observation data of the moving body, and the second model is a machine learning model that uses the skeletal data of the moving body as training data.

[0079] In one embodiment, (11) the program causes the computer to perform the following actions: predict the position of a moving body using a dynamic model that represents at least the relationship between the interaction between the moving body and other moving bodies or obstacles and the velocity vector of the moving body; estimate a first velocity vector of the moving body using a first model; estimate a second velocity vector of the moving body using a second model; if it is determined that the moving body is making a sudden movement, apply the second velocity vector to the dynamic model; and if it is not determined that the moving body is making a sudden movement, apply the first velocity vector to the dynamic model, wherein the first model is a model that estimates the first velocity vector of the moving body based on past observation data of the moving body, and the second model is a machine learning model that uses the skeletal data of the moving body as training data.

[0080] In this disclosure, the terms "First," "Second," etc., are identifiers used to distinguish the configurations. Configurations distinguished by the terms "First," "Second," etc., in this disclosure may have their numbers swapped. For example, the first velocity vector may swap the identifiers "First" and "Second" with the second velocity vector. The swapping of identifiers occurs simultaneously. The configurations remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by their codes. The terms "First," "Second," etc., in this disclosure should not be used alone to interpret the order of the configurations or to justify the existence of smaller numbered identifiers.

[0081] 1: Information processing device, 2, 2A, 2B, 2C, 2D: Mobile object, 3: Obstacle, 4: Robot, 5: Control device, 10: Stereo camera, 11: Memory unit, 12: Control unit, 20: Second model, 21: Skeleton data, 23: BN, 24: SC layer, 25: TC layer, 26: Residual connector, 27: AP layer, 28: Vector prediction unit

Claims

1. An information processing device comprising a control unit that predicts the position of a moving body by a dynamic model that represents at least the relationship between the interaction between the moving body and other moving bodies or obstacles and the velocity vector of the moving body, wherein the control unit estimates a first velocity vector of the moving body by a first model, the first model being a model that estimates a first velocity vector of the moving body based on past observation data of the moving body, the second velocity vector of the moving body by a second model being a machine learning model that uses the skeletal data of the moving body as training data, and if it is determined that the moving body is making a sudden movement, the second velocity vector is applied to the dynamic model, and if it is not determined that the moving body is making a sudden movement, the first velocity vector is applied to the dynamic model.

2. The information processing apparatus according to claim 1, wherein the first model is a model that does not utilize machine learning.

3. The information processing apparatus according to claim 1 or 2, wherein the first model is a Kalman filter.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the second model is a model that utilizes ST-GCN (Spatio-Temporal Graph Convolutional Networks).

5. The information processing apparatus according to any one of claims 1 to 4, wherein the control unit calculates the absolute value of the current acceleration of the moving body based on the velocity of the moving body at multiple points in time, and determines that the moving body is making a sudden movement if the absolute value of the current acceleration of the moving body is equal to or greater than an acceleration threshold.

6. The information processing apparatus according to any one of claims 1 to 5, wherein the control unit determines that the moving body is making a sudden movement if the angle between the current first velocity vector of the moving body and the future second velocity vector of the moving body is greater than or equal to an angle threshold.

7. The information processing apparatus according to any one of claims 1 to 6, wherein the mechanical model is a Social Force Model (SFM).

8. The information processing apparatus according to any one of claims 1 to 7, wherein the control unit generates past observation data of the moving object by analyzing captured images generated by a stereo camera, and determines whether the moving object is making a sudden movement if the distance from the stereo camera to the moving object is less than or equal to a distance threshold.

9. The information processing apparatus according to any one of claims 1 to 8, wherein the control unit determines whether the moving body makes a sudden movement when the speed of the moving body exceeds a speed threshold.

10. An information processing method comprising: predicting the position of a moving body using a dynamic model that represents at least the relationship between the interaction between the moving body and other moving bodies or obstacles and the velocity vector of the moving body; estimating a first velocity vector of the moving body using a first model; estimating a second velocity vector of the moving body using a second model; applying the second velocity vector to the dynamic model if it is determined that the moving body is making a sudden movement; and applying the first velocity vector to the dynamic model if it is not determined that the moving body is making a sudden movement, wherein the first model is a model that estimates the first velocity vector of the moving body based on past observation data of the moving body, and the second model is a machine learning model that uses the skeletal data of the moving body as training data.

11. A program that causes a computer to perform the following actions: predict the position of a moving object using a dynamic model that represents at least the relationship between the interaction between the moving object and other moving objects or obstacles and the velocity vector of the moving object; estimate a first velocity vector of the moving object using a first model; estimate a second velocity vector of the moving object using a second model; apply the second velocity vector to the dynamic model if it is determined that the moving object is making a sudden movement; and apply the first velocity vector to the dynamic model if it is not determined that the moving object is making a sudden movement, wherein the first model is a model that estimates the first velocity vector of the moving object based on past observation data of the moving object, and the second model is a machine learning model that uses the skeletal data of the moving object as training data.

Images