Method and device for learning and detecting hand gestures in a vehicle

Abstract

The invention relates to a method for learning and detecting a hand gesture of a user in the interior of a vehicle, having the steps of capturing, by means of at least one image capturing unit (1), at least one image (2) of the interior, receiving, by means of a data processing unit (3), the image (2), and estimating, by means of an image processing unit (4), the number N of two-dimensional or three-dimensional joint positions G1 - GN of a hand of the user in the image (2), transferring the joint positions G1 - GN to a classification unit (4'), and learning or detecting the new hand gesture by means of the classification unit (4') on the basis of the joint positions G1 – GN.

Description

[0001] 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 VIENNA (AT)

[0002] Method and device for learning and recognizing hand gestures in a vehicle

[0003] The invention relates to a method and a device for learning and recognizing hand gestures in a vehicle.

[0004] It is known from the prior art to use hand gestures from people in vehicles to trigger specific vehicle actions. For example, there are hand gestures for controlling the radio or the lighting conditions in the vehicle cabin. Cameras that provide images or image sequences of the vehicle's interior can be used to recognize such hand gestures. Pre-trained AI models, especially neural networks, can be used to recognize hand gestures in the images or image sequences that have been previously assigned to the corresponding actions.

[0005] To adapt such an AI model to a specific driver or to enable it to learn new hand gestures from that driver, it is necessary to train the AI ​​model. This requires training data and sufficient computing power. In automotive environments, it is usually not possible to access sufficient computing power, so the only remaining solution is to train the AI ​​models outside the vehicle, which requires a connection to a cloud infrastructure. The training data must be sent to a cloud infrastructure, and the trained models must be fed into the vehicle. This is complex and requires a constant mobile data connection. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) The object of the invention is to provide an alternative to these known methods and devices.In particular, the aim is to enable the rapid adaptation of existing AI models for recognizing hand gestures in a vehicle to new drivers, using the lowest possible computing power and without the use of external cloud infrastructure.

[0006] These and other problems are solved according to the invention using the methods and devices of the independent claims.

[0007] A method according to the invention is designed for recognizing a hand gesture of a human user in the interior of a vehicle. In a first step of the method, an image acquisition unit captures at least one image of the vehicle's interior. The image acquisition unit can be a conventional 2D camera; however, it can also be a 3D camera, for example, a stereo camera with two individual cameras or a time-of-flight (ToF) camera. Such 2D and 3D cameras, as well as their arrangement in a vehicle, are generally known to those skilled in the art.

[0008] The image is subsequently received by a data processing unit. This data processing unit can be a microcontroller or microcomputer and may include a central processing unit (CPU), volatile semiconductor memory (RAM), non-volatile semiconductor memory (ROM, SSD), magnetic storage (hard drive), and / or optical storage (CD-ROM), as well as interface units (Ethernet, USB) and the like. The components of such data processing units are generally known to those skilled in the art.

[0009] In the next step, an image processing unit estimates a number N of two-dimensional or three-dimensional joint positions of the user's hand in the image. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) For this purpose, known pattern recognition algorithms, especially from the field of machine learning, can be used. In particular, the image processing unit can include a neural network that is trained to recognize and estimate joint positions of a hand in a 2D image.

[0010] The neural network in the image processing unit can, for example, be a convolutional neural network (CNN) with one or more convolutional layers followed by a pooling layer. Typically, the input is a two-dimensional matrix of values, namely the pixels of a two-dimensional color image. Accordingly, the neurons in the convolutional layer are arranged in a matrix. The neural network outputs a series of two- or three-dimensional joint positions.

[0011] Supervised training using annotated example images is preferably used to train the neural network in the image processing unit. The training data consists of two-dimensional images of people whose two- or three-dimensional wrist positions are annotated. The output of the neural network can also be a graph, where, starting from a wrist, four finger joints of all five fingers of the hand are connected by edges.

[0012] Neural networks for estimating the two- or three-dimensional positions of human joints in images are known in the prior art. The use of such neural networks in vehicles for estimating key points of vehicle occupants is also known. However, instead of a neural network, any other method from the field of machine learning can also be used for image processing and joint position recognition.

[0013] Detecting wrist positions in two or three dimensions requires significant computing power, as complex patterns in the image data must be recognized. To interpret the image data, the information (color, intensity) of all pixels must be entered into the neural network as an input matrix. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) Depending on the image resolution, for example, 640 x 480 pixels, this results in several hundred thousand input parameters. For the detection of N joint positions, this yields N x 3 output parameters in 3D space. With N = 21 joint positions, the number of output parameters is therefore 63. In two-dimensional space, this still results in N x 2 output parameters, i.e., 42 for N = 21 joint positions. Training a neural network with so many input and output parameters is complex and is therefore usually carried out outside the vehicle.

[0014] In the next step, the estimated joint positions are passed to a classification unit specifically designed for classifying hand gestures based on two- or three-dimensional joint positions. This classification unit categorizes the passed joint positions Gi - GN into a predefined hand gesture. The classification unit may also include a neural network trained to classify hand gestures based on joint positions.

[0015] The neural network in the classification unit can again be a convolutional neural network (CNN) with one or more convolutional layers, followed by a pooling layer. Typically, the input is in the form of a graph, namely the two- or three-dimensional joint positions Gi - GN, which correspond to the five fingers of the human hand, radiating from a central wrist, and are connected by edges.

[0016] Accordingly, the neurons are arranged in the first convolutional layer. The neural network outputs one of several possible hand gestures. Training data was used to train the neural network in the classification unit, assigning two- or three-dimensional joint positions of the user's hand to a specific hand gesture. Each hand gesture can, in turn, be assigned to a specific action of the vehicle.

[0017] The number of input parameters for the neural network in the classification unit is only N x 3, which, with 21 three-dimensional joint positions, is approximately 63.65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 VIENNA (AT). The number of output parameters, namely the predefined hand gestures, is even lower—for example, five, ten, or fifteen hand gestures can be predefined.

[0018] Training a neural network with 63 input and 15 output parameters is simple and can also be done in the vehicle by the user themselves.

[0019] Using only two-dimensional joint positions can be particularly useful when only a small number of clearly distinguishable hand gestures have been pre-stored, for example, only five or ten different hand gestures. In this case, the neural network in the classification unit requires only 42 input parameters and five or ten output parameters.

[0020] Instead of a neural network, any other method for classifying input data into one of several categories can be used for the classification unit. This is particularly recommended when only two-dimensional joint positions are used, the number of hand gestures is small (e.g., less than 10), and the hand gestures are clearly distinguishable. In this case, the classification unit can apply an analytical method, such as regression analysis, to select a predefined hand gesture based on the joint positions. A support vector machine (SVM) or a random forest can also be used for hand gesture classification in this case to reduce the required computing power.

[0021] For example, the classification unit can define several reference hand gestures, each with N x 3 reference joint positions. The classification unit selects the reference hand gesture whose reference joint positions show the smallest deviation from the estimated joint positions for classification.

[0022] The task separation according to the invention allows for optimization of resource requirements: The complex estimation of the joint positions Gi-GN from the captured image is solved in the pre-trained and unchanging first image processing unit.65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) In contrast, the less resource-intensive classification of hand gestures takes place in the classification unit and can be continuously modified or supplemented by the user.

[0023] The estimation of joint positions and the classification of hand gestures are therefore performed by different and separate units. The image processing unit used for joint position estimation has fixed parameters and is trained externally. The classification unit has variable parameters and can be trained locally by the user and adapted to the individual user's hand gestures. A significant advantage of the invention's use of separate units is that new hand gestures can be learned without the captured images having to leave the vehicle. This is important in automotive applications where data privacy and minimizing external dependencies play a crucial role.

[0024] An inventive method for teaching a user a new hand gesture in the interior of a vehicle comprises the following steps:

[0025] As a first step, a data processing unit receives a request from the user to learn a new hand gesture for a specific vehicle action, such as adjusting the side mirror. To do this, the user first thinks of a hand gesture for the action and assigns it a name. This allows the system to be flexibly adapted to the individual needs and preferences of the user for operating the vehicle.

[0026] In the next step, at least one image of the interior is captured by an image capture unit. This image is then received by the data processing unit in the vehicle. An image processing unit within the data processing unit estimates N two- or three-dimensional joint positions Gi - GN of the user's hand within the image.

[0027] The estimated joint positions Gi - GN and the assigned action are used as training data for a classification unit. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 VIENNA (AT) Due to the limited number of joint positions - usually 42 to a maximum of 63 numerical values ​​- and the low complexity of the classification model, this training can be carried out directly in the vehicle and in real time.

[0028] According to the invention, the user may be prompted to repeat the hand gesture several times, in particular three, five, or up to ten times, to enable robust learning of the hand gesture. Alternatively, instead of training a neural network, mean values ​​of the respective joint positions can be calculated and stored as reference joint positions for this hand gesture. After completion of the learning process, the classification unit is able to reliably recognize the new hand gesture and execute the corresponding action.

[0029] According to the invention, the image processing unit can estimate up to N = 21 three-dimensional joint positions of the user's hand, in particular the position of the wrist and the four joint positions of the user's five fingers.

[0030] The methods according to the invention offer a number of advantages. Since the classification unit takes over the task of recognizing hand gestures, the computational effort is reduced. All processing steps take place locally in the vehicle, so sensitive image data does not need to be transferred to external servers. The ability to train new hand gestures during operation allows users to customize the system to their individual needs. This makes it possible to build a user-specific set of hand gestures. These hand gestures can be linked to various actions in the vehicle. Since only the joint positions are required for training, the memory requirement remains low.

[0031] According to the invention, the image acquisition unit can capture a sequence of images, in particular a sequence of five to ten images in rapid succession, for example within a period of half a second to two seconds. In this case, the image processing unit can use the sequence of images to estimate the three-dimensional joint positions Gi - GN of the user's hand. This can improve the accuracy of the estimate.65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) According to the invention, it can also be provided that the image processing unit determines a temporally successive sequence (Gi - GN), (GI - GN)', (GI - GN)" of joint positions from the temporally successive sequence of images, transmits the temporal sequence of joint positions to the classification unit, and the classification unit classifies the temporal sequence into a predefined dynamic hand gesture or uses it to learn a new dynamic hand gesture. This ensures that not only static hand gestures but also dynamic hand gestures (i.e., hand movements) are learned and recognized. For example, a swiping or rotating movement of the user's hand can be defined as a hand gesture to trigger a specific action.

[0032] In practice, the methods according to the invention can be used in various contexts.

[0033] Examples of applications according to the invention include the operation of infotainment systems (playing music, receiving calls, entering navigation destinations), the control of comfort functions (regulating the temperature, opening windows or sunroofs), the setting of assistance systems (activating safety-relevant actions such as an emergency call) and the like.

[0034] The invention further relates to a computer-readable storage medium comprising instructions that cause a data processing unit to execute a method according to the invention.

[0035] The invention further relates to a device for recognizing predefined hand gestures of a user in the interior of a vehicle, comprising an image acquisition unit configured to capture an image of the interior and a data processing unit configured to receive the image. The image processing unit is configured to estimate a number N of two-dimensional or three-dimensional joint positions Gi - GN of the user's hand in the image. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) According to the invention, a classification unit is provided which is configured to receive the estimated joint positions Gi - GN and which is configured to classify the joint positions Gi - GN into one of the predefined hand gestures.

[0036] According to the invention, it can be provided that the image acquisition unit is configured to capture a sequence of images, the image processing unit is configured to estimate a temporally successive sequence (Gi - GN), (GI - GN)', (GI - GN)" of joint positions from the sequence of images, and the classification unit is configured to classify the temporal sequence into a predefined hand gesture.

[0037] Finally, the invention also relates to a device for teaching a user a new hand gesture in the interior of a vehicle, comprising a data processing unit configured to receive a request from the user to learn a hand gesture for a specific action of the vehicle, an image acquisition unit configured to capture at least one image of the interior, a data processing unit configured to receive the image, and an image processing unit configured to estimate a number N of two-dimensional or three-dimensional joint positions Gi - GN of a user's hand in the image. The image processing unit is preferably configured as a neural network.A classification unit is provided, which is trained to receive the joint positions Gi - GN, and which is trained to use the joint positions Gi - GN to learn the new hand gesture. The classification unit is preferably designed as a neural network.

[0038] According to the invention, the image acquisition unit can again be configured to capture a sequence of images, the image processing unit can be configured to estimate a temporally successive sequence (Gi - GN), (GI - GN)', (GI - GN)" of joint positions from the sequence of images, and the classification unit can be configured to use the temporal sequence to learn the new hand gesture. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) The invention further relates to a vehicle comprising a device according to the invention.

[0039] Further features of the invention will become apparent from the claims, the exemplary embodiments and the figures.

[0040] The invention is explained below using an exemplary, non-exclusive embodiment.

[0041] Fig. 1a shows a schematic example of a device according to the invention in a vehicle;

[0042] Fig. 1b shows a schematic example of a generated graph when carrying out a method according to the invention;

[0043] Fig. 2 shows a schematic flowchart of a method according to the invention for recognizing predefined hand gestures of a user in a vehicle;

[0044] Fig. 3 shows a schematic flowchart of a method according to the invention for learning new hand gestures by a user in a vehicle.

[0045] Fig. 1a shows a schematic example of a device according to the invention in a vehicle. The device comprises an image acquisition unit 1 in the form of a camera, which is designed to capture a two-dimensional image 2. Furthermore, the device comprises a data processing unit 3 with an image processing unit 4 (not shown) and a classification unit 4'. The data processing unit 3 is connected to various actuators 5, 5' and display devices in the vehicle.

[0046] Image processing unit 4 comprises a neural network trained to estimate the three-dimensional joint positions of a human hand from a two-dimensional image. Two-dimensional image data of human hands with three-dimensionally annotated key points were used to train this network. The images used for training thus contain images of human hands whose joint points are annotated with three-dimensional relative coordinates. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) In operation, image processing unit 4 recognizes the three-dimensional joint positions of a human hand visible in an image 2, which, for example, is a color image with a resolution of 1024 x 768 pixels.

[0047] Fig. 1b shows a schematic example of the data object generated by the image processing unit 4 during the execution of a method according to the invention. The estimated three-dimensional joint positions Gi, G2, ... GN are available in the form of Cartesian coordinates and are stored by the image processing unit 4 as a data object in the form of a graph.

[0048] The generated graph has a root in the shape of the wrist G1 and five linearly arranged edges, each with four joint points. In total, the graph thus has 21 three-dimensional joint positions. Each joint position has three coordinate values, resulting in a total of 63 stored numerical values.

[0049] Fig. 2 shows a schematic flowchart of a method according to the invention for recognizing predefined hand gestures of a user in a vehicle. In a first step, an image acquisition unit 1 captures an image 2 of the vehicle's interior. The image 2 is transmitted to a data processing unit 3 in the vehicle. An image processing unit 4, which is designed as a neural network and is part of the data processing unit 3, processes the image 2 and estimates the three-dimensional positions of the recognizable joint positions G1 - GN of the user's hand. The joints are the wrist and four joints of each of the five fingers of the user's hand, so that N = 21. If not all joints are recognizable in the image, the positions of the recognizable joints are estimated, and the remaining joint positions are extrapolated from these.

[0050] In the next step, the estimated joint positions G1–GN are passed to a classification unit 4', which is also located in the data processing unit 3. Classification unit 4' is designed as a separate neural network, which was previously trained to assign joint positions to one of several predefined hand gestures Hi–HM. Approximately ten hand gestures were predefined and assigned to corresponding vehicle actions. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) In the next step, classification unit 4' classifies the joint positions Gi–GN as one of these predefined hand gestures Hi. Data processing unit 3 then triggers the corresponding actuator 5' to execute the action assigned to the hand gesture, for example, increasing the interior temperature.

[0051] Fig. 3 shows a schematic flowchart of a method according to the invention for teaching a user new hand gestures in a vehicle. In a first step, a data processing unit 3 receives a request from the user to learn a hand gesture Hj for a specific action. For example, this could be a hand gesture for moving the driver's seat.

[0052] The user is prompted to perform the hand gesture. An image capture unit 1 then takes a picture 2 of the vehicle's interior. This picture 2 is transmitted to a data processing unit 3 within the vehicle. An image processing unit 4, designed as a neural network and part of the data processing unit 3, processes the image 2 and estimates the three-dimensional positions of the user's hand joints (Gi-GN) as they are visible in the image.

[0053] The estimated joint positions Gi - GN are passed as training data to a classification unit 4', which is also located in the data processing unit 3. In this embodiment, the classification unit 4' is implemented as a neural network and uses the estimated joint positions Gi - GN to train the hand gesture Hj. These steps are repeated several times for the user-defined hand gesture Hi, for example, 3 times, 5 times, 10 times, or 15 times, to ensure robust recognition of the hand gesture by the classification unit.

[0054] As a result of the method, the classification unit 4' is trained to robustly recognize the hand gesture Hi based on the joint positions. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) In alternative embodiments of the invention, the classification unit 4' comprises an analytical regression model which stores reference joint positions Gi' - GN' for each of the predefined hand gestures and compares the estimated joint positions Gi - GN with the reference joint positions for classifying the hand gesture, selecting the hand gesture whose normal distance is smallest. To learn a new hand gesture, this embodiment may provide that the user performs a hand gesture several times, with mean values ​​of the estimated joint positions being calculated and stored as new reference joint positions for this new hand gesture.

[0055] However, the invention is not limited to these described embodiments, but also includes further embodiments of the present invention within the scope of the following patent claims.

Claims

65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 VIENNA (AT) Patent claims 1. Method for recognizing a predefined hand gesture of a user in the interior of a vehicle, comprising the steps: a. Recording, by an image recording unit (1), at least of an image (2) of the interior, b. Receipt, by a data processing unit (3), of the image (2), c. Estimation, by an image processing unit (4), of a number N of two-dimensional or three-dimensional joint positions Gi - GN of a user's hand in the image (2), characterized by the fact that d. the estimated joint positions Gi - GN are transferred to a classification unit (4'), and e. the classification unit (4') classifies the joint positions Gi - GN into a predefined hand gesture.

2. Method for learning a user's hand gesture in the interior of a vehicle, comprising the steps: a. Receipt, by a data processing unit (3), of a request from the user to learn a hand gesture for a specific action of the vehicle, b. Recording, by an image recording unit (1), at least one image (2) of the interior, c. Receipt, by the data processing unit (3), of the image (2), d. Estimation, by an image processing unit (4), of a number N of two-dimensional or three-dimensional joint positions Gi - GN of a user's hand in the image (2), characterized by the fact that e. the joint positions Gi - GN are passed to a classification unit (4') as training or reference data, and f. the classification unit (4') uses the joint positions Gi - GN to learn the new hand gesture. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 VIENNA (AT) 3. Method according to claim 2, characterized in that steps b) - f) are repeated several times, in particular three, five, or up to ten times, to enable robust learning of the new hand gesture.

4. Method according to one of claims 1 to 3, characterized in that the image acquisition unit (1) records a sequence of images (2, 2', 2"), in particular a sequence of five to ten images (1 , 1 ', 1 “) in short temporal succession.

5. Method according to claim 4, characterized in that the image processing unit (4) uses the sequence of images (2, 2', 2") to achieve an improved estimation of the two- or three-dimensional joint positions Gi - GN ZU.

6. Method according to one of claims 4 or 5, characterized in that the image processing unit (4) determines a temporally successive sequence (Gi - GN), (GI - GN)', (GI - GN)" of joint positions from the sequence of images (2, 2', 2"), passes the temporal sequence to the classification unit (4'), and the classification unit (4') classifies the temporal sequence into a predefined hand gesture or uses it to learn a new hand gesture.

7. Method according to any one of claims 1 to 6, characterized in that the image processing unit (4) estimates a number of up to N = 21 joint positions of the user's hand, wherein in particular the position of the wrist and the four joint positions of the user's five fingers are.

8. Computer-readable storage medium comprising instructions that cause a data processing unit (3) to execute a method according to any one of claims 1 to 7. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 Vienna (AT) 9. Device for recognizing predefined hand gestures of a user in the interior of a vehicle, comprising: a. an image acquisition unit (1) designed to capture at least one image (2) of the interior, b. a data processing unit (3) designed to receive the image (2), c. an image processing unit (4) configured to estimate a number N of two-dimensional or three-dimensional joint positions Gi - GN of a user's hand in the image (2), characterized by the fact that d. a classification unit (4') is provided which is designed to receive the estimated joint positions Gi - GN, wherein e. the classification unit (4') is designed to classify the estimated joint positions Gi - GN into a hand gesture.

10. Device according to claim 9, characterized in that a. the image acquisition unit (1 ) is designed to acquire a sequence of images (2, 2', 2"), b. the image processing unit (4) is designed to estimate a temporally successive sequence (Gi - GN), (GI - GN)', (GI - GN)" of joint positions from the sequence of images (2, 2', 2"), and c. the classification unit (4') is designed to classify the temporal sequence into a predefined hand gesture. 65100 / AG / - emotion3D GmbH, Rainergasse 1 / 8, 1040 VIENNA (AT) 11. Device for learning a hand gesture by a user in the interior of a vehicle, comprising: a. a data processing unit (3) which is trained to receive a request from the user to learn a hand gesture for a specific action of the vehicle, b. an image acquisition unit (1) designed to capture at least one image (2) of the interior, c. a data processing unit (3) designed to receive the image (2), d. an image processing unit (4) configured to estimate a number N of two-dimensional or three-dimensional joint positions Gi - GN of a user's hand in the image (2), characterized by the fact that e. a classification unit (4') is provided which is designed to receive the joint positions Gi - GN, wherein f. the classification unit (4') is trained to use the joint positions Gi - GN to learn the new hand gesture.

12. Device according to claim 11, characterized in that a. the image acquisition unit (1 ) is designed to acquire a sequence of images (2, 2', 2"), b. the image processing unit (4) is designed to estimate a temporally successive sequence (Gi - GN), (GI - GN)', (GI - GN)" of joint positions from the sequence of images (2, 2', 2"), and c. the classification unit (4') is trained to use the temporal sequence for learning the new hand gesture.

13. Vehicle comprising a device according to any one of claims 9 to 12.

Images