Method, apparatus and computer program for detecting position of upper limb joint

Abstract

The present invention relates to a method for detecting the position of an upper limb joint, comprising the steps of: when an RGB image obtained by photographing the upper body of a subject is received, inputting the RGB image obtained by photographing the upper body into a first artificial intelligence model trained to output the pixel coordinates of a plurality of joints and the nose, and outputting the pixel coordinates of the plurality of joints and the nose of the subject; inputting the pixel coordinates output in the step of outputting the pixel coordinates and the RGB image to a second artificial intelligence model, so as to output depth information of the plurality of joints and the nose; and calculating the three-dimensional coordinates of the plurality of joints and the nose by using the pixel coordinates and the depth information.

Description

Method, device, and computer program for detecting the position of upper limb joints

[0001] This patent application claims priority to Korean Patent Application No. 10-2024-0031205 filed with the Korean Intellectual Property Office on March 5, 2024, and the disclosures of said patent application are incorporated herein by reference.

[0002] The present invention was carried out under the support of the Ministry of Science and ICT under project unique number 2710002317 and project number RS-2023-00248118, the project management agency for the above project is the Pan-Governmental Full-Cycle Medical Device Research and Development Center, the research project name is "Leading the 4th Industrial Revolution and Future Medical Environment," the research project title is "Advanced Smart Board Medical Device for Improving and Evaluating Upper Extremity Function in Patients with Brain Diseases and Development of Non-Face-to-Face Rehabilitation Treatment Technology Using Artificial Intelligence," the project performing agency is Neofect Co., Ltd., and the research period is April 1, 2023 – December 31, 2025.

[0003] The present invention relates to a method, apparatus, and computer program for detecting the position of an upper limb joint, and in particular to a method, apparatus, and computer program for detecting the position of an upper limb joint that detects the position of a three-dimensional joint in real time using a smartphone camera.

[0004] In recent years, rapid advancements in artificial intelligence and computer vision technologies have significantly improved human posture estimation and joint position detection technologies. These technologies are being utilized in various fields, including medicine, sports, and entertainment, and play a crucial role in the field of rehabilitation medicine, particularly in analyzing patient movements and monitoring treatment effectiveness.

[0005] Conventional 3D joint position detection systems have primarily been implemented using expensive specialized equipment. For example, motion capture systems or specialized cameras equipped with depth sensors have been used. While these systems provide high accuracy, their application in daily use or remote medical environments has been limited due to high costs and complex installation processes.

[0006] Meanwhile, joint position estimation technology using 2D images has also been developed. This technology estimates the 2D coordinates of joints from images captured by a standard RGB camera and has the advantages of high accessibility and relatively simple implementation. However, there was a limitation in that it was difficult to determine the accurate 3D position due to a lack of depth information with 2D estimation alone.

[0007] Recently, research on estimating 3D joint positions from a single RGB image has been actively underway. While this approach has the advantage of extracting 3D information without a separate depth sensor, it often still requires high computational power or faces difficulties in real-time processing.

[0008] Furthermore, many existing systems focus on estimating the pose of the entire body, which has limited their application in the field of rehabilitation therapy, where precise analysis specialized for upper limb joints is required. This is because accurately detecting and analyzing minute movements and postural changes of the upper limbs is crucial in rehabilitation.

[0009] Furthermore, most high-performance 3D joint position estimation systems are designed to run on high-performance computers or servers, which has limited real-time processing and the provision of immediate feedback in mobile environments. This has made it difficult for patients to continuously monitor and improve their movements in daily life.

[0010] Against this backdrop, the need has arisen for technology capable of performing high-performance 3D upper limb joint position detection in real time using widely available smartphones. This has the potential to enhance the effectiveness of rehabilitation therapy, enable continuous monitoring of patients in their daily lives, and improve the quality of remote medical services.

[0011] Therefore, there is a need for methods, devices, and computer programs for detecting the position of upper limb joints that can address these technical requirements and limitations.

[0012] [Prior Art Literature]

[0013] [Patent Literature]

[0014] Korean Patent Publication No. 10-2023-0096063

[0015] The present invention aims to provide a method, apparatus, and computer program for detecting the position of an upper limb joint that enables accurate 3D joint position estimation without expensive specialized equipment and can provide immediate feedback through real-time processing in a mobile environment.

[0016] To achieve the above objective, the present invention is characterized in that, in a method for detecting the position of an upper limb joint, the method comprises: a step of inputting an RGB image of a subject's upper body to a first artificial intelligence model trained to output pixel coordinates of a plurality of joints and a nose when an RGB image of an upper body is input, thereby outputting pixel coordinates of a plurality of joints and a nose of a subject; a step of inputting the pixel coordinates output in the step of outputting pixel coordinates and the RGB image to a second artificial intelligence model to output depth information of a plurality of joints and a nose; and a step of calculating three-dimensional coordinates of a plurality of joints and a nose using the pixel coordinates and depth information.

[0017] Preferably, the step of outputting the pixel coordinates may output at least one of the left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist coordinates as a plurality of joint coordinates.

[0018] Preferably, the step of outputting the pixel coordinates may use a Vision Transformer as the first artificial intelligence model.

[0019] Preferably, the step of outputting the depth information may involve generating a heatmap in which pre-measured depth values ​​are assigned to pixel coordinates of a plurality of joints and the nose, and training the second artificial intelligence model to output a heatmap in which pre-measured depth values ​​are assigned to pixel coordinates of a plurality of joints and the nose when an RGB image and pixel coordinates of a plurality of joints and the nose are input.

[0020] Preferably, the step of outputting the depth information may use an artificial intelligence model with a UNet structure as the second artificial intelligence model.

[0021] Preferably, the step of calculating the three-dimensional coordinates can calculate the three-dimensional coordinates (X,Y,Z) of each joint or nose by inputting pixel information and depth information of each joint or nose into the following mathematical formula 1.

[0022] [Mathematical Formula 1]

[0023]

[0024] (Here, And, and, depth is depth information, x is the x-value of the pixel coordinate, y is the y-value of the pixel coordinate, and f x ,f y ε₀ and y₀, respectively, represent the camera's focal length along the x and y axes, indicating the distance from the camera lens to the image sensor. c x , c yThese are the positions that serve as the center in the horizontal direction and the center in the vertical direction of the image, respectively, and are the coordinates that represent the optical center of the camera in the image plane.

[0025] Preferably, the method may further include the step of evaluating the horizontal state of the shoulders through the three-dimensional coordinates of the left and right shoulders, and providing feedback to the user to adjust the horizontal state of the shoulders if the horizontal state of the shoulders is evaluated as misaligned.

[0026] Preferably, the step of providing the feedback can define a vector (V) connecting both shoulders using the three-dimensional coordinates (L) of the left shoulder and the three-dimensional coordinates (R) of the right shoulder, as shown in Equation 2 below.

[0027] [Mathematical Formula 2]

[0028]

[0029] Preferably, in claim 8, the step of providing the feedback is, using the following mathematical formulas 3 to 5, the angle formed by the normal vector (n) of the xy-plane and the vector (V) connecting both shoulders ( The horizontal state of the shoulder can be evaluated by calculating ).

[0030] [Mathematical Formula 3]

[0031]

[0032] [Mathematical Formula 4]

[0033]

[0034] [Mathematical Formula 5]

[0035]

[0036] In addition, the present invention is further characterized by a device for detecting the position of an upper limb joint, comprising: a processor including one or more cores; and a memory; wherein the processor inputs an RGB image of the upper body of a subject to output pixel coordinates of a plurality of joints and a nose to a first artificial intelligence model trained to output pixel coordinates of a plurality of joints and a nose when an RGB image of the upper body is input, and inputs the pixel coordinates output in the step of outputting the pixel coordinates and the RGB image to a second artificial intelligence model to output depth information of a plurality of joints and a nose, calculates three-dimensional coordinates of a plurality of joints and a nose using the pixel coordinates and depth information, evaluates the horizontal state of the shoulders through the three-dimensional coordinates of the left shoulder and the right shoulder, and provides feedback to the user to adjust the horizontal state of the shoulders when the horizontal state of the shoulders is evaluated to be misaligned.

[0037] In addition, the present invention is a computer program that includes instructions stored in a computer-readable storage medium to cause a computer to perform the following operations, wherein the operations include: an operation of inputting an RGB image of a subject’s upper body to a first artificial intelligence model trained to output pixel coordinates of a plurality of joints and a nose when an RGB image of a subject’s upper body is input, and outputting pixel coordinates of a plurality of joints and a nose of a subject; an operation of inputting the pixel coordinates output by the operation of outputting pixel coordinates and the RGB image to a second artificial intelligence model to output depth information of a plurality of joints and a nose of a subject; and an operation of calculating three-dimensional coordinates of a plurality of joints and a nose of a subject using the pixel coordinates and depth information of a subject. The operation of outputting depth information is further characterized by generating a heatmap in which pre-measured depth values ​​are assigned to the pixel coordinates of a plurality of joints and a nose of a subject, and training the second artificial intelligence model to output a heatmap in which pre-measured depth values ​​are assigned to the pixel coordinates of a plurality of joints and a nose of a subject when an RGB image and pixel coordinates of a plurality of joints and a nose of a subject are input.

[0038] The present invention has the advantage of being able to estimate the position and depth information of a joint through image data.

[0039] In addition, the present invention has the advantage of being able to provide immediate feedback through real-time processing in a mobile environment using video captured by a smartphone.

[0040] In addition, the present invention has the advantage of increasing the efficiency of rehabilitation treatment and improving the patient's treatment process.

[0041] Figure 1 shows a flowchart of a method for detecting the position of an upper limb joint according to an embodiment of the present invention.

[0042] FIG. 2 shows a schematic diagram of a method for detecting the position of an upper limb joint according to an embodiment of the present invention.

[0043] FIG. 3 shows a heatmap generated in the step of outputting depth information according to an embodiment of the present invention.

[0044] Figure 4 shows a structural diagram of the UNet structure of a second artificial intelligence model used in the step of outputting depth information according to an embodiment of the present invention.

[0045] FIG. 5 illustrates an example of calculating the angle between a normal vector (n) and a vector (V) connecting both shoulders in the step of providing feedback according to an embodiment of the present invention.

[0046] FIG. 6 illustrates an example of providing feedback to a subject in the step of providing feedback according to an embodiment of the present invention.

[0047] FIG. 7 shows a configuration diagram of a device for detecting the position of an upper limb joint according to an embodiment of the present invention.

[0048] FIG. 8 shows a schematic diagram of a computing environment according to an embodiment of the present invention.

[0049] The present invention is characterized in that it comprises a method for detecting the position of an upper limb joint, wherein the method includes: a step of inputting an RGB image of a subject’s upper body to a first artificial intelligence model trained to output pixel coordinates of a plurality of joints and a nose when an RGB image of an upper body is input, thereby outputting pixel coordinates of a plurality of joints and a nose of a subject; a step of inputting the pixel coordinates output in the step of outputting pixel coordinates and the RGB image to a second artificial intelligence model to output depth information of a plurality of joints and a nose; and a step of calculating three-dimensional coordinates of a plurality of joints and a nose using the pixel coordinates and depth information.

[0050] The present invention will be described in detail below with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by exemplary embodiments. Identical reference numerals in each drawing indicate components that perform substantially the same function.

[0051] The purpose and effects of the present invention may be naturally understood or become clearer through the following description, and the purpose and effects of the present invention are not limited solely to the description below. Furthermore, in describing the present invention, if it is determined that a detailed description of known technology related to the present invention may unnecessarily obscure the essence of the present invention, such detailed description will be omitted.

[0052] The terms used in this invention are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the description of the invention, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0053] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0054] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this invention.

[0055] In interpreting the components, they are interpreted to include a margin of error even without a separate explicit indication. In the case of descriptions regarding temporal relationships, for example, where the temporal sequence is described using 'after,' 'following,' 'next,' 'before,' etc., cases that are not continuous are included unless 'immediately' or 'directly' is used.

[0056] Hereinafter, the technical configuration of the present invention will be described in detail with reference to the attached drawings.

[0057] FIG. 1 shows a flowchart of a method for detecting the position of an upper limb joint according to an embodiment of the present invention. Referring to FIG. 1, the method for detecting the position of an upper limb joint may include a step of outputting pixel coordinates (S110), a step of outputting depth information (S130), a step of calculating three-dimensional coordinates (S150), and a step of providing feedback (S170).

[0058] FIG. 2 shows a schematic diagram of a method for detecting the position of an upper limb joint according to an embodiment of the present invention. Referring to FIG. 2, first, the method for detecting the position of an upper limb joint can acquire an RGB image through a smartphone camera. Next, the method for detecting the position of an upper limb joint can obtain pixel coordinates of the joint by estimating the position of the joint from the input RGB image through a first artificial intelligence model. Next, the method for detecting the position of an upper limb joint can generate a heatmap by inserting depth information into the pixel coordinates of the joint. Subsequently, the method for detecting the position of an upper limb joint can estimate depth information of each joint by inputting the RGB image and the pixel information of the joint into a second artificial intelligence model. Finally, the method for detecting the position of an upper limb joint can evaluate upper limb movement through the pixel coordinates and depth information of each joint, and can provide feedback to the user based on the evaluated upper limb movement results.

[0059] The method for detecting the position of upper limb joints can detect the two-dimensional position and depth information of the joints using RGB images captured by a smartphone camera. Through this, the method can provide posture monitoring, warnings, and feedback to subjects during non-face-to-face and home-based upper limb rehabilitation training without the need for separate intervention by an occupational therapist. In particular, the method can monitor changes in the subject's posture in real time during rehabilitation training, providing evaluation and feedback based on the degree of postural deviation during training, focusing on torso posture. Furthermore, the method can transmit evaluation and feedback results to the user's mobile device, providing an environment where the user can check them in real time via a mobile application.

[0060] The step of outputting pixel coordinates (S110) can output pixel coordinates of multiple joints and the nose of a subject by inputting an RGB image of the subject's upper body to a first artificial intelligence model trained to output pixel coordinates of multiple joints and the nose when an RGB image of the upper body is input. The RGB image may refer to a video captured by a smartphone camera.

[0061] The step of outputting pixel coordinates (S110) can output at least one of the coordinates of the left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist as a plurality of joint coordinates.

[0062] The step of outputting pixel coordinates (S110) may use a Vision Transformer as the first artificial intelligence model.

[0063] The step of outputting pixel coordinates (S110) can train a first artificial intelligence model, and the training data may be an RGB image with the pixel coordinates of each joint or nose labeled.

[0064] The first artificial intelligence model requires a large amount of computation and time for training, but once training is complete, it can rapidly produce results with a small amount of computation. Therefore, the present invention can be utilized in a mobile environment. In another embodiment, the present invention is provided in the form of an API, so that computations using artificial intelligence are performed on the server, and only the results are provided to the mobile device.

[0065] The step of outputting depth information (S130) can output depth information of multiple joints and nose by inputting the pixel coordinates outputted in the step of outputting pixel coordinates and the RGB image into the second artificial intelligence model.

[0066] FIG. 3 shows a heatmap generated in the step of outputting depth information according to an embodiment of the present invention. Referring to FIG. 3, the step of outputting depth information (S130) generates a heatmap in which pre-measured depth values ​​are assigned to pixel coordinates of a plurality of joints and a nose, and can train the second artificial intelligence model to output a heatmap in which pre-measured depth values ​​are assigned to pixel coordinates of a plurality of joints and a nose when an RGB image and pixel coordinates of a plurality of joints and a nose are input.

[0067] FIG. 4 shows a structural diagram of the UNet structure of the second artificial intelligence model used in the step (S130) of outputting depth information according to an embodiment of the present invention. Referring to FIG. 4, the step (S130) of outputting depth information may use an artificial intelligence model with a UNet structure as the second artificial intelligence model.

[0068] The UNet structure can have a deep neural network structure in which the encoder and decoder parts form a U-shape symmetrically. The UNet structure performs the process of extracting features from input images and restoring them to the desired output form, and is designed to effectively estimate joint depth information in particular.

[0069] Specifically, the encoder part consists of successive feature extraction steps, each step including a 3x3 convolution operation, a ReLU activation function, and a 2x2 max pooling operation. The input image (256x256x3) passes through the first convolution layer, increasing the number of channels in the feature map and reducing its spatial size through max pooling. As this process is repeated, feature maps of sizes 128x128, 64x64, 32x32, and 16x16 are sequentially generated.

[0070] The decoder part performs the process of progressively reconstructing features extracted from the encoder. Each step begins with a 2x2 upsampling operation to increase the spatial size of the feature map, and refines the features through 3x3 convolution and the ReLU activation function. A particularly important point is that at each decoder step, the feature maps generated at the corresponding step of the encoder are combined through a concatenation operation. This skip connection allows detailed spatial information captured by the encoder to be directly utilized in the reconstruction process of the decoder.

[0071] At the bottom (bottleneck) of the network, the most compressed feature representation of size 16x16 is generated, which contains the most abstract features of the input image. Subsequently, as the size is gradually increased through the decoder, a final output of size 256x256, identical to the original image, is generated. In the final layer, depth values ​​at each pixel location are predicted using 1x1 convolution and a sigmoid activation function.

[0072] The UNet structure can accurately estimate joint depth information in RGB images through these structural features, and preserves detailed depth information by utilizing multi-scale features through skip connections. Furthermore, it is a structure suitable for real-time processing as it enables effective feature extraction with a relatively shallow number of layers.

[0073] The step of calculating 3D coordinates (S150) can calculate 3D coordinates of multiple joints and nose using pixel coordinates and depth information.

[0074] The step of calculating three-dimensional coordinates (S150) can calculate the three-dimensional coordinates (X,Y,Z) of each joint or nose by inputting pixel information and depth information of each joint or nose into the following mathematical formula 1.

[0075] [Mathematical Formula 1]

[0076]

[0077] (Here, And, and, depth is depth information, x is the x-value of the pixel coordinate, y is the y-value of the pixel coordinate, and f x ,f y ε₀ and y₀, respectively, represent the camera's focal length along the x and y axes, indicating the distance from the camera lens to the image sensor. c x , c y These are the positions that serve as the center in the horizontal direction and the center in the vertical direction of the image, respectively, and are the coordinates that represent the optical center of the camera in the image plane.

[0078] The step of providing feedback (S170) evaluates the horizontal state of the shoulders through the three-dimensional coordinates of the left shoulder and the right shoulder, and if the horizontal state of the shoulders is evaluated as being misaligned, it can provide feedback so that the user can adjust the horizontal state of the shoulders.

[0079] The step of providing feedback (S170) can define a vector (V) connecting both shoulders using the 3D coordinates (L) of the left shoulder and the 3D coordinates (R) of the right shoulder, as shown in Equation 2 below.

[0080] [Mathematical Formula 2]

[0081]

[0082] FIG. 5 illustrates an example of calculating the angle between a normal vector (n) and a vector (V) connecting both shoulders in the step of providing feedback according to an embodiment of the present invention. Referring to FIG. 5, the step of providing feedback (S170) calculates the angle formed by the normal vector (n) of the xy-plane and the vector (V) connecting both shoulders using the following mathematical formulas 3 to 5. The horizontal state of the shoulder can be evaluated by calculating ). That is, the step of providing feedback (S170) can evaluate whether the shoulder is horizontal in the up, down, left, and right directions in three dimensions.

[0083] [Mathematical Formula 3]

[0084]

[0085] [Mathematical Formula 4]

[0086]

[0087] [Mathematical Formula 5]

[0088]

[0089] FIG. 6 illustrates an example of providing feedback to a subject in the step (S170) of providing feedback according to an embodiment of the present invention. Referring to FIG. 6, if the upper and lower horizontal alignment of the subject's right shoulder and left shoulder is not aligned, the step (S170) of providing feedback may generate feedback such as "Please raise your right shoulder to align it." In another embodiment, if the left and right (rotational) horizontal alignment of the subject's right shoulder and left shoulder is not aligned, the step (S170) of providing feedback may generate feedback such as "Please rotate your shoulder counterclockwise to maintain a proper posture."

[0090] FIG. 7 shows a configuration diagram of a device (100) for detecting the position of an upper limb joint according to an embodiment of the present invention. Referring to FIG. 7, the configuration of the device (100) for detecting the position of an upper limb joint illustrated is merely a simplified example. In one embodiment of the present invention, the device (100) for detecting the position of an upper limb joint may include other configurations for performing the computing environment of the device (100), and only some of the disclosed configurations may constitute the device (100).

[0091] A device (100) for detecting the position of an upper limb joint may include a processor (110) including one or more cores, a memory (120), and a network (130).

[0092] The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in memory (120) and perform data processing for machine learning according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the learning of a network function. For example, a CPU and a GPGPU can work together to process the learning of a network function and data classification using the network function. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process the learning of a network function and data classification using the network function. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0093] The processor (110) can output pixel coordinates of multiple joints and the nose of a subject by inputting an RGB image of the subject's upper body to a first artificial intelligence model trained to output pixel coordinates of multiple joints and the nose when an RGB image of the upper body is input. The processor (110) can perform the step (S110) of outputting the aforementioned pixel coordinates.

[0094] The processor (110) can input the pixel coordinates output in the step of outputting pixel coordinates and the RGB image into the second artificial intelligence model to output depth information of multiple joints and the nose. The processor (110) can perform the step (S130) of outputting the depth information described above.

[0095] The processor (110) can calculate the three-dimensional coordinates of multiple joints and the nose using pixel coordinates and depth information. The processor (110) can perform the step (S150) of calculating the aforementioned three-dimensional coordinates.

[0096] The processor (110) evaluates the horizontal state of the shoulders through the three-dimensional coordinates of the left shoulder and the right shoulder, and if the horizontal state of the shoulders is evaluated as being misaligned, it can provide feedback to allow the user to adjust the horizontal state of the shoulders. The processor (110) can perform the step (S170) of providing feedback.

[0097] The memory (120) can store any form of information generated or determined by the processor (110) and any form of information received by the network (130).

[0098] The memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (120) on the internet. The description of the memory described above is merely an example, and the present disclosure is not limited thereto.

[0099] The network (130) may use any type of known wired or wireless communication system. The network (130) may receive RGB images, etc. from a related device or related system.

[0100] The network (130) can transmit and receive information, user interfaces, etc., processed by the processor (110) through communication with other terminals. For example, the network (130) can provide a user interface generated by the processor (100) to a client (e.g., a user terminal). In addition, the network (130) can receive external input from a user authorized as a client and transmit it to the processor (110). At this time, the processor (110) can process operations such as outputting, modifying, changing, or adding information provided through the user interface based on the external input from the user received from the network (130).

[0101] Meanwhile, a device (100) for detecting the position of an upper limb joint according to one embodiment of the present disclosure may include a server as a computing system that transmits and receives information through communication with a client. At this time, the client may be any type of terminal capable of accessing the server.

[0102] In an additional embodiment, the device (100) for detecting the position of an upper limb joint may include any type of terminal that receives data resources generated from any server and performs additional information processing.

[0103] A computer program for detecting the position of an upper limb joint, which is another embodiment of the present invention, may include an operation of outputting pixel coordinates, an operation of outputting depth information, an operation of calculating three-dimensional coordinates, and an operation of providing feedback.

[0104] The operation of outputting pixel coordinates can output pixel coordinates of multiple joints and the nose of a subject by inputting an RGB image of the subject's upper body into a first artificial intelligence model trained to output pixel coordinates of multiple joints and the nose when an RGB image of the upper body is input. The operation of outputting pixel coordinates refers to the operation performed in the aforementioned step of outputting pixel coordinates (S110).

[0105] The operation of outputting depth information is an operation of outputting pixel coordinates, and the pixel coordinates output and the RGB image are input into the second artificial intelligence model to output depth information of multiple joints and the nose. The operation of outputting depth information refers to the operation performed in the aforementioned step of outputting depth information (S130).

[0106] The operation of calculating 3D coordinates can calculate the 3D coordinates of multiple joints and the nose using pixel coordinates and depth information. The operation of calculating 3D coordinates refers to the operation performed in the aforementioned step of calculating 3D coordinates (S150).

[0107] The action of providing feedback evaluates the horizontal state of the shoulders through the three-dimensional coordinates of the left and right shoulders, and if the horizontal state of the shoulders is evaluated as misaligned, it can provide feedback so that the user can adjust the horizontal state of the shoulders. The action of providing feedback refers to the action performed in the aforementioned step of providing feedback (S170).

[0108] FIG. 8 shows a schematic diagram of a computing environment according to an embodiment of the present invention.

[0109] Although the present disclosure has been described as generally being implementable by a computing device, those skilled in the art will understand that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers, and / or as a combination of hardware and software.

[0110] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, those skilled in the art will be well aware that the method of the present disclosure may be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0111] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0112] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0113] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0114] An exemplary environment for implementing various aspects of the present disclosure, including a computer (1000), is shown, wherein the computer (1000) includes a processing unit (1020), a system memory (1030), and a system bus (1010). The system bus (1010) connects system components, including the system memory (1030) (but not limited thereto), to the processing unit (1020). The processing unit (1020) may be any processor among various commercial processors. Dual processors and other multiprocessor architectures may also be used as the processing unit (1020).

[0115] The system bus (1010) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. The system memory (1030) includes read-only memory (ROM) (1034) and random access memory (RAM) (1032). The basic input / output system (BIOS) is stored in non-volatile memory (1034), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1000) at times such as during startup. The RAM (1032) may also include high-speed RAM, such as static RAM, for caching data.

[0116] The computer (1000) also includes an internal hard disk drive (HDD) (1050) (e.g., EIDE, SATA)—this internal hard disk drive (1050) may also be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1060) (e.g., for reading from or writing to a removable diskette), and an optical disk drive (1070) (e.g., for reading from a CD-ROM disk or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1050), the magnetic disk drive (1060), and the optical disk drive (1070) may each be connected to the system bus (1010) via a hard disk drive interface, a magnetic disk drive interface, and an optical drive interface. Interfaces for implementing external drives include at least one or both of the Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0117] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1000), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0118] A number of program modules, including an operating system (1092), one or more application programs (1094), other program modules (1096), and a database (1098), may be stored in the drive and RAM (1032). All or part of the operating system, applications, modules, and / or data may also be cached in RAM (1032). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0119] The user can input commands and information into the computer (1000) through one or more wired / wireless input devices (1042), such as pointing devices like a keyboard and a mouse. Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1020) via an input / output interface (1040) connected to the system bus (1010), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0120] A monitor or other type of display device is also connected to the system bus (1010) via an interface such as a video adapter. In addition to the monitor, the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and others.

[0121] A computer (1000) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1082), via wired and / or wireless communication. The remote computer(s) (1082) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and may include many or all of the components generally described for the computer (1000). The logical connection includes a wired / wireless connection to a local area network (LAN) and / or a larger network, e.g., a wide area network (WAN). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a global computer network, e.g., the Internet.

[0122] When used in a LAN networking environment, the computer (1000) is connected to a local network (not shown) via a wired and / or wireless communication network interface or adapter (not shown). The adapter (not shown) may facilitate wired or wireless communication to the LAN (not shown), and the LAN (not shown) may also include a wireless access point installed therein to communicate with the wireless adapter (not shown). When used in a WAN networking environment, the computer (1000) may include a modem (not shown), be connected to a communication computing device on the WAN (not shown), or have other means to establish communication over the WAN (not shown), such as through the Internet. The modem (not shown), which may be internal or external and wired or wireless, is connected to the system bus (1010) via a serial port interface (not shown). In a networked environment, the program modules described for the computer (1000) or parts thereof may be stored in a remote memory / storage device (not shown). You will be well aware that the illustrated network connection is exemplary and that other means of establishing communication links between computers can be used.

[0123] The computer (1000) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0124] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0125] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0126] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and model steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of the present disclosure.

[0127] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0128] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0129] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0130] The embodiments of the present invention described above are not implemented only through devices and methods, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present invention, or a recording medium on which such a program is recorded. Such a recording medium may be executed not only on a server but also on a user terminal.

[0131] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

[0132] The present invention has the advantage of being able to estimate the position and depth information of a joint through image data.

[0133] In addition, the present invention has the advantage of being able to provide immediate feedback through real-time processing in a mobile environment using video captured by a smartphone.

[0134] In addition, the present invention has the advantage of increasing the efficiency of rehabilitation treatment and improving the patient's treatment process.

Claims

1. A method for detecting the position of an upper limb joint, A step of inputting an RGB image of the subject's upper body into a first artificial intelligence model trained to output pixel coordinates of multiple joints and the nose when an RGB image of the upper body is input, and outputting pixel coordinates of multiple joints and the nose of the subject; A step of inputting the pixel coordinates output in the step of outputting the pixel coordinates and the RGB image into a second artificial intelligence model to output depth information of a plurality of joints and the nose; and A step of calculating the 3D coordinates of a plurality of joints and the nose using the pixel coordinates and depth information above; A method that includes 2. In Paragraph 1, The step of outputting the above pixel coordinates is, A method of outputting at least one of the left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist coordinates as multiple joint coordinates.

3. In Paragraph 1, The step of outputting the above pixel coordinates is, A method of using a Vision Transformer as the first artificial intelligence model.

4. In Paragraph 1, The step of outputting the depth information above is, Generate a heatmap by assigning pre-measured depth values ​​to the pixel coordinates of multiple joints and the nose, and A method of training the second artificial intelligence model to output a heatmap with pre-measured depth values ​​assigned to the pixel coordinates of the multiple joints and nose when an RGB image, multiple joints, and the nose pixel coordinates are input.

5. In Paragraph 1, The step of outputting the depth information above is, A method of using an artificial intelligence model with a UNet structure as the second artificial intelligence model above.

6. In Paragraph 1, The step of calculating the above three-dimensional coordinates is, A method of calculating the 3D coordinates (X,Y,Z) of each joint or nose by inputting pixel information and depth information of each joint or nose into the following mathematical formula 1. [Mathematical Formula 1] (Here, And, and, depth is depth information, x is the x-value of the pixel coordinate, y is the y-value of the pixel coordinate, and f x ,f y ε₀ and y₀, respectively, represent the camera's focal length along the x and y axes, indicating the distance from the camera lens to the image sensor. c x , c y These are the positions that serve as the center in the horizontal direction and the center in the vertical direction of the image, respectively, and are the coordinates that represent the optical center of the camera in the image plane.

7. In Paragraph 1, A method further comprising the step of evaluating the horizontal state of the shoulders through the three-dimensional coordinates of the left and right shoulders, and providing feedback to the user to adjust the horizontal state of the shoulders if the horizontal state of the shoulders is evaluated as misaligned.

8. In Paragraph 7, The step of providing the above feedback is, A method of defining a vector (V) connecting both shoulders using the 3D coordinates (L) of the left shoulder and the 3D coordinates (R) of the right shoulder, as shown in mathematical formula 2 below. [Mathematical Formula 2] 9. In Paragraph 8, The step of providing the above feedback is, Using the following mathematical formulas 3 to 5, the angle formed by the normal vector (n) of the xy-plane and the vector (V) connecting both shoulders ( A method of evaluating the horizontal state of the shoulder by calculating ). [Mathematical Formula 3] [Mathematical Formula 4] [Mathematical Formula 5] 10. A device for detecting the position of an upper limb joint, A processor comprising one or more cores; and Memory; Includes, The above processor is, A first artificial intelligence model trained to output pixel coordinates of multiple joints and the nose when an RGB image of the upper body is input is input to output pixel coordinates of multiple joints and the nose of the subject, and an RGB image of the subject's upper body is input to output pixel coordinates of multiple joints and the nose of the subject. The pixel coordinates output in the step of outputting the pixel coordinates and the RGB image are input into a second artificial intelligence model to output depth information of multiple joints and the nose, and Using the above pixel coordinates and depth information, the 3D coordinates of multiple joints and the nose are calculated, and A device that evaluates the horizontal state of the shoulders through the three-dimensional coordinates of the left and right shoulders, and provides feedback to the user to adjust the horizontal state of the shoulders if the horizontal state of the shoulders is evaluated as misaligned.

11. A computer program stored on a computer-readable storage medium and comprising instructions that cause a computer to perform the following operations, wherein the operations are: An operation of inputting an RGB image of the subject's upper body into a first artificial intelligence model trained to output pixel coordinates of multiple joints and the nose when an RGB image of the upper body is input, and outputting pixel coordinates of multiple joints and the nose of the subject; An operation of inputting the pixel coordinates output by the operation of outputting the above pixel coordinates and the above RGB image into a second artificial intelligence model to output depth information of a plurality of joints and the nose; and The operation of calculating the 3D coordinates of a plurality of joints and the nose using the pixel coordinates and depth information; is included, The operation of outputting the above depth information is, A computer program stored on a computer-readable storage medium, wherein the second artificial intelligence model is trained to generate a heatmap with pre-measured depth values ​​assigned to pixel coordinates of multiple joints and the nose, and outputs the heatmap with pre-measured depth values ​​assigned to pixel coordinates of multiple joints and the nose when an RGB image and pixel coordinates of multiple joints and the nose are input.

Images