Device and method for classifying posture in image data by using end-to-end model

Abstract

The present disclosure relates to a posture classification method executed by one or more processors, the posture classification method comprising the steps of: training an end-to-end posture classification model; and performing posture classification on an object included in original image data by using the trained end-to-end posture classification model. The step of performing the posture classification includes the steps of: receiving original image data; generating preprocessed image data by preprocessing the received original image data; generating feature data from the preprocessed image data by means of an image feature extraction module including one or more convolutional layers; generating object recognition information and key point information from the feature data; and generating posture information about the object from the feature data, the object recognition information, and the key point information.

Description

Device and method for classifying poses in image data using an end-to-end model

[0001] The present disclosure relates to a method for classifying the pose of an object in image data using an end-to-end model.

[0002] The present invention is derived from research conducted as part of the Ministry of Science and ICT’s AI semiconductor SW integrated platform technology development (Project No.: 1711198587, Project No.: RS-2023-00277060, Project Management Agency: Korea Institute of Information and Communication Technology Planning and Evaluation, Research Project Name: Open Edge AI Semiconductor Design and SW Platform Technology Development, Project Performing Agency: Altist Co., Ltd., Research Period: 2023.06.01 ~ 2027.12.31).

[0003] Meanwhile, the Korean government, the provider of the problem, has no property interest in all aspects of the present invention.

[0004] In the field of computer vision, deep learning is used for purposes such as image classification, object detection, segmentation, and pose estimation. Among these, pose estimation is a technology that detects objects included in images or videos, and visualizes them by identifying and connecting key points that constitute the objects.

[0005] Pose classification is a technique that classifies the pose of an object within an image using the locations of keypoints, which are the outputs of pose estimation. Conventionally, modules for pose classification and pose estimation have been configured separately, and the pose classification module classifies the pose using the output of the pose estimation module.

[0006] At this point, since the decision on which data from the pose estimation module's output to use as input for the pose classification module occurs subjectively, there is a problem in that the pose classification accuracy of the entire system is lowered due to this subjectivity.

[0007] Furthermore, the separation of these attitude estimation and attitude classification modules increases the complexity of the overall system and may lead to problems such as significantly increased maintenance and repair requirements.

[0008] Therefore, there is a demand in the industry for technology to perform attitude classification by introducing end-to-end models.

[0009] The present disclosure aims to provide an end-to-end attitude classification model that improves the accuracy of the attitude classification model and reduces maintenance and management requirements.

[0010] The problems that this disclosure aims to solve are not limited to those described above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below.

[0011] A pose classification method executed by one or more processors according to one aspect of the present disclosure, wherein the pose classification method comprises: a step of learning an end-to-end pose classification model; and a step of performing pose classification of an object included in original image data using the previously learned end-to-end pose classification model; wherein the step of performing pose classification may comprise: a step of receiving original image data; a step of performing preprocessing on the received original image data to generate preprocessed image data; a step of generating feature data from the preprocessed image data by an image feature extraction module including one or more convolutional layers; a step of generating object recognition information and keypoint information from the feature data; and a step of generating pose information of the object from the feature data, the object recognition information and keypoint information.

[0012] In addition, the above feature data may be characterized by including a plurality of feature maps.

[0013] Additionally, the feature data may be characterized by including: a first feature map corresponding to a first convolution layer which is the last layer of the image feature extraction module; a second feature map corresponding to a second convolution layer which is the layer preceding the first convolution layer; and a third feature map corresponding to a third convolution layer which is the layer preceding the second convolution layer.

[0014] In addition, the plurality of feature maps may be characterized by including a first feature map containing low-dimensional features of the original image data and a second feature map containing high-dimensional features of the original image data.

[0015] In addition, the step of generating pose information of the object may be characterized by including the step of generating combined input data by combining the feature data, the object recognition information, and the keypoint information.

[0016] In addition, the step of generating the combined input data may be characterized by including the step of connecting the feature data, the object recognition information, and the keypoint information.

[0017] In addition, the step of connecting the feature data and the object recognition information and keypoint information may be characterized by including the step of connecting the object recognition information and keypoint information for each feature map included in the feature data.

[0018] In addition, the step of generating the combined input data may be characterized by performing an attention operation on the feature data, the object recognition information, and the keypoint information.

[0019] In addition, the pose classification model includes an image feature extraction module, an object recognition and keypoint prediction module, and a pose classification module, and the step of training the end-to-end pose classification model may be characterized by training the image feature extraction module, the object recognition and keypoint prediction module, and the pose classification module simultaneously.

[0020] The posture classification method according to the present disclosure can improve the accuracy and consistency of classification results compared to conventional posture classification models, and can facilitate the maintenance and management of posture classification models.

[0021] The effects according to the present disclosure are not limited to those described above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0022] FIG. 1 is a conceptual diagram showing the configuration of a posture classification model according to some embodiments of the present disclosure.

[0023] FIG. 2 is a conceptual diagram showing the configuration of an image feature extraction module (120) according to some embodiments of the present disclosure.

[0024] FIG. 3 is a conceptual diagram showing the flow of data input to a posture classification module according to some embodiments of the present disclosure.

[0025] FIG. 4 is a flowchart illustrating the execution of an object pose classification method within image data using an end-to-end model according to some embodiments of the present disclosure.

[0026] FIG. 5 is a block diagram showing a computing device for providing the posture classification method of the present disclosure.

[0027] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by exemplary embodiments. Unless otherwise defined, all terms used in this specification (including technical and scientific terms) shall be used in a meaning that is commonly understood by those skilled in the art to which this disclosure belongs, but this may vary depending on the intent of those skilled in the art, case law, the emergence of new technology, etc.

[0028] Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. In certain cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant explanatory sections. Accordingly, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0029] FIG. 1 is a conceptual diagram showing the configuration of a posture classification model according to some embodiments of the present disclosure.

[0030] The modules described in the following description of the invention, including FIG. 1, can be executed by one or more processors. For example, these modules can be executed by a processor after being loaded into a memory area accessible to the processor as software modules. As another example, these modules can be implemented and executed in hardware such as an FPGA or an ASIC.

[0031] For the convenience of explanation, the operations of the posture classification model for performing the posture classification method according to the present disclosure are described as being performed by individual modules; however, notwithstanding this description, it should be understood that the individual operations are substantially executed by dedicated hardware or one or more processors.

[0032] Additionally, the components of a posture classification model for performing the posture classification method according to the present disclosure may be implemented within a single computing device, or, as another example, distributed within a plurality of computing devices connected directly and / or indirectly.

[0033] Referring to FIG. 1, a pose classification model (100) according to some embodiments of the present disclosure may include an image data preprocessing module (110), an image feature extraction module (120), an object recognition and keypoint prediction module (130), a pose classification module (140) and / or a visualization module (150).

[0034] In the present disclosure, the pose classification model (100) may be configured as an end-to-end model. That is, one or more modules included in the pose classification model (100) may be learned by a single process. That is, the learning of at least one module included in the pose classification model (100) may be performed integrally. Preferably, the object recognition and keypoint prediction module (130) and the pose classification module (140) may be learned simultaneously. More preferably, the image feature extraction module (120), the object recognition and keypoint prediction module (130), and the pose classification module (140) may be learned simultaneously.

[0035] The configuration and function of each module will be described in detail below with reference to the drawings.

[0036] In some embodiments of the present disclosure, the image data preprocessing module (110) may perform preprocessing on the original image data (10) on which pose classification is to be performed. In some embodiments, the preprocessing may include resizing, tilting, rotation, tonal conversion, padding and / or image segmentation and recombination.

[0037] In some embodiments, the image data preprocessing module (110) may perform preprocessing on the original image data (10) differently during the training process of the pose classification model (100) and during the inference process after training.

[0038] In some embodiments, the image data preprocessing module (110) may perform size transformation, gradient adjustment, rotation, tone transformation, padding and / or image segmentation and recombination on each of the original image data (10) during the learning process. Preferably, the image data preprocessing module (110) may perform size transformation, gradient adjustment, rotation, tone transformation, padding and / or image segmentation and recombination on each of the original image data (10). In another embodiment, the image data preprocessing module (110) may perform preprocessing on the original image data (10) by randomly selecting a preset number of preprocessing techniques from a plurality of preprocessing techniques such as size transformation, gradient adjustment, rotation, tone transformation, padding and / or image segmentation and recombination. Through this, the image data preprocessing module (110) may generate one or more preprocessed image data from one original image data (10).

[0039] In some embodiments, the image data preprocessing module (110) may perform at least one of size transformation and / or padding during the inference process of the pose classification model after training. In one example, the image data preprocessing module (110) may perform size transformation and padding on each of the original image data (10) during the inference process of the pose classification model after training to generate one or more preprocessed image data from a single image.

[0040] In some other embodiments, the image data preprocessing module (110) may perform normalization on the original image data (10). For example, the normalization technique performed by the image data preprocessing module (110) may include min-max normalization or Z-score normalization.

[0041] For example, the image data preprocessing module (110) may perform normalization on the original image data (10) and then perform additional preprocessing such as size transformation, padding, image segmentation, and recombination. In another embodiment, the image data preprocessing module (110) may first perform processing such as size transformation, padding, image segmentation, and recombination on the original image data (10), and then perform normalization on each of the processed image data to generate one or more preprocessed image data.

[0042] Referring again to FIG. 1, an image feature extraction module (120) according to some embodiments of the present disclosure may be a module implemented with software, hardware and / or a combination thereof for generating feature data of a preprocessed image output from an image data preprocessing module (110).

[0043] In some embodiments, the image feature extraction module (120) can extract features from a preprocessed image based on preset feature information.

[0044] In some other embodiments, the image feature extraction module (120) may be configured to be a machine learning-based learnable model and to generate feature information of an input preprocessed image through a learning process.

[0045] In some embodiments, the image feature extraction module (120) may utilize a deep learning model for performing computer vision tasks. For example, the image feature extraction module (120) may be configured using a Convolutional Neural Network, a machine learning model based on a Convolutional Neural Network, and / or a Vision Transformer. However, the structure of the model for configuring the image feature extraction module (120) described above is exemplary, and the configuration of the image feature extraction module (120) is not limited thereto.

[0046] In one embodiment, the image feature extraction module (120) may be implemented using a machine learning model based on a convolutional neural network. In this case, the image feature extraction module (120) may include one or more convolution layers for generating feature data from a preprocessed image.

[0047] In some embodiments, each of one or more convolution layers constituting the image feature extraction module (120) can generate a feature map. In one embodiment, the image feature extraction module (120) can transmit the feature map extracted from the last convolution layer as feature data to modules located at the downstream end. In another embodiment, the image feature extraction module (120) can transmit the feature map extracted from one or more convolution layers as feature data to modules located at the downstream end.

[0048] For example, the image feature extraction module (120) can transmit the first layer feature map, the second layer feature map, and the third layer feature map as feature data to modules located at the rear end. At this time, the selection of the first to third layers can be varied. In one example, multiple layers can be selected as the convolution layers located at the rear end. That is, if the image feature extraction module (120) is composed of N convolution layers, the first layer can be the (N-2)th layer, the second layer can be the (N-1)th layer, and the third layer can be the (N-3)th layer.

[0049] In another example, multiple layers may be selected such that the feature data includes both low-dimensional and high-dimensional features of the image data. For example, the multiple layers may be determined according to a preset number from each of the convolution layers located at the front end, the center end, and the rear end of the image feature extraction module (120).

[0050] In one example, if the image feature extraction module (120) includes 3N convolution layers, the first layer may be determined to be any one of the 1st layer to the Nth layer, the second layer to be any one of the (N+1)th layer to the 2Nth layer, and the third layer to be any one of the (2N+1)th layer to the 3Nth layer.

[0051] Referring again to FIG. 1, feature data generated from an image feature extraction module (120) according to some embodiments of the present disclosure can be transmitted to an object recognition and keypoint prediction module (130).

[0052] In some embodiments, the object recognition and keypoint prediction module (130) may be a module composed of software, hardware and / or a combination thereof configured to predict keypoints related to the position information of an object and the pose of an object within the extracted feature data.

[0053] In some embodiments, a keypoint may refer to a feature point related to the pose of an object included in the image data. For example, if the object included in the image data is an animal, the animal's joints may be keypoints.

[0054] In some embodiments, the object recognition and keypoint prediction module (130) may be implemented using a machine learning-based learnable model.

[0055] In one embodiment, the object recognition and keypoint prediction module (130) may include an object recognition module for performing object recognition from feature data generated from an image feature extraction module (120) and a keypoint prediction module for predicting keypoints from the feature data.

[0056] For example, the object recognition module may be a neural network model for generating bounding box and / or segmentation information to represent the region of existence of an object within the original image data from feature data. In one embodiment, the object recognition module may generate one or more bounding box and / or segmentation information.

[0057] Additionally, for example, the keypoint prediction module may be a neural network model for generating keypoint information of an object within the original image data from feature data. In one embodiment, the keypoint information of the object may be determined at the pixel level.

[0058] In another embodiment, the object recognition and keypoint prediction module (130) can generate one or more bounding box and / or segmentation information and keypoint information as a single neural network model.

[0059] In some embodiments, the object recognition and keypoint prediction module (130) may output only some of the generated object recognition information (e.g., bounding box information) and / or keypoint information. For example, the object recognition and keypoint prediction module (130) may exclude at least some of the bounding boxes and / or keypoints from the output using a Non-Maximum Suppression technique.

[0060] Referring again to FIG. 1, a posture classification model (100) according to some embodiments of the present disclosure may include a posture classification module (140).

[0061] In some embodiments, the pose classification module (140) may be a module that generates a classification result for pose information of an object included in an image. In some embodiments, the pose classification module (140) may be implemented using a machine learning-based learnable model.

[0062] Referring to FIG. 1, in some embodiments, the pose classification module (140) receives feature data generated from the image feature extraction module (120) and object recognition information and / or keypoint information generated from the object recognition and keypoint prediction module (130), and can generate pose information of an object included in the image data therefrom.

[0063] In some embodiments, the pose classification module (140) receives object recognition information (e.g., bounding box information and / or key point information) from the object recognition and key point prediction module (130) and can use it to classify the pose of an object in an image. For example, the object recognition information transmitted from the object recognition and key point prediction module (130) may include pixel information of a bounding box where an object exists within the original image data (10) and / or the location of a pixel determined to have an object based on segmentation. In one embodiment, the object recognition and key point prediction module (130) may transmit one or more object recognition information to the pose classification module (140).

[0064] In some other embodiments, the keypoint information transmitted from the object recognition and keypoint prediction module (130) may include location (e.g., pixel) information of one or more keypoints within the original image data (10).

[0065] In some embodiments, the feature data transmitted from the image feature extraction module (120) to the pose classification module (140) may be a feature map generated from a plurality of convolution layers included in the image feature extraction module (120). For example, the feature data transmitted from the image feature extraction module (120) may include a first layer feature map, a second layer feature map, and a third layer feature map. In this case, the selection of the first to third layers may vary. In one example, the plurality of layers may be selected as the convolution layers located at the rearmost end. That is, if the image feature extraction module (120) is composed of N convolution layers, the first layer may be the (N-2)th layer, the second layer may be the (N-1)th layer, and the third layer may be the (N-3)th layer.

[0066] In another example, the feature data transmitted from the image feature extraction module (120) to the pose classification module (140) may be selected to include both low-dimensional and high-dimensional features of the original image data. For example, a plurality of feature maps may be determined according to a preset number from each of the convolution layers located at the front end, the center end, and the rear end of the image feature extraction module (120).

[0067] In one example, if the image feature extraction module (120) includes 3N convolution layers, the first layer may be determined to be any one of the 1st layer to the Nth layer, the second layer to be any one of the (N+1)th layer to the 2Nth layer, and the third layer to be any one of the (2N+1)th layer to the 3Nth layer.

[0068] As described above, the pose classification module (140) according to the present disclosure can classify the pose information of an object included in the image data by utilizing not only the object recognition information and keypoint information generated from the object recognition and keypoint prediction module (130) but also the low-dimensional and high-dimensional features of the original image data (10) generated from the image feature extraction module (120) in order to classify the pose of an object existing in the original image data (10). In addition, when training the pose classification model (100), the image feature extraction module (120), the object recognition and keypoint prediction module (130), and the pose classification module (140) are trained simultaneously, so the learning of features for pose classification can be performed more effectively. Furthermore, by configuring the pose classification model (100) as an end-to-end model as in the configuration of FIG. 1, maintenance and improvement of the model can be facilitated.

[0069] Referring again to FIG. 1, a pose classification model (100) according to some embodiments of the present disclosure may further include a visualization module (150). In some embodiments, the visualization module (150) may generate pose information visualization image data (20) by adding object recognition information and keypoint information generated from an object recognition and keypoint prediction module (130) and pose information generated from a pose classification module (140) onto the original image data (10).

[0070]

[0071] FIG. 2 is a conceptual diagram showing the configuration of an image feature extraction module (120) according to some embodiments of the present disclosure.

[0072] Referring to FIG. 2, an image feature extraction module (120) according to some embodiments of the present disclosure may be configured using a machine learning model having a convolutional neural network-based structure.

[0073] In this case, in some embodiments, the image feature extraction module (120) may include a plurality of convolution layers. The image feature extraction module illustrated exemplarily in FIG. 2 may include a first convolution layer (121) through a fourth convolution layer (124). However, the configuration of the convolution layers illustrated in FIG. 2 is exemplary, and the image feature extraction module (120) may be configured with additional detailed configurations and a smaller or larger number of convolution layers.

[0074] In some embodiments, the image feature extraction module (120) can transmit a feature map output from one or more convolution layers as feature data to the object recognition and keypoint prediction module (130) and the pose classification module (140).

[0075] At this time, as the transmitted feature data, in one embodiment, the feature data may be configured using a feature map corresponding to any one of the first convolution layer (121) to the fourth convolution layer (124).

[0076] In another embodiment, feature maps corresponding to a preset number of convolution layers constitute feature data, wherein the feature maps constituting the feature data can be formed by selecting a preset number of convolution layers starting from the outermost layer. For example, if the preset number of convolution layers is 2, a third feature map corresponding to the third convolution layer (123) and a fourth feature map corresponding to the fourth convolution layer (124) may be included in the feature data.

[0077]

[0078] FIG. 3 is a conceptual diagram showing the flow of data input to a posture classification module according to some embodiments of the present disclosure.

[0079] Referring to FIG. 3, in some embodiments, the image feature extraction module (120) can generate feature data (11) including one or more feature maps from original image data (10).

[0080] In some embodiments, the generated feature data (11) is transmitted to an object recognition and keypoint prediction module (130) to generate object recognition information and keypoint information (12).

[0081] In some embodiments, the generated feature data (11), object recognition information, and keypoint information (12) can be combined by an input combining module (140a), converted into combined input data, and then transmitted to a posture classification module (140).

[0082] In one embodiment, the input combining module (140a) can perform the combination of feature data (11), object recognition information, and key point information (12) by concatenating feature data (11) and object recognition information and key point information (12).

[0083] In another embodiment, the input combining module (140a) can perform the combination of feature data and object recognition information and key point information by concatenating object recognition information and key point information (12) for each feature map included in the feature data (11).

[0084] In another embodiment, the input combining module (140a) can combine the feature data (11) and the object recognition information and key point information (12) by performing a preset operation between the feature data (11) and the object recognition and key point prediction module (12).

[0085] For example, the input combining module (140a) can perform an attention operation between each feature map included in the feature data (11) and the object recognition information and keypoint information (12) to combine the object recognition information and keypoint information (12) to generate combined input data and transmit it to the detail classification module (140). For example, based on the attention score generated through the attention operation between each feature map and the object recognition information and keypoint information (12), a weighted sum can be performed on each feature map to consider the importance of each feature map, and combined input data can be generated. At this time, additional operations such as padding may be performed to enable the operation between the feature map and the object recognition information and keypoint information.

[0086]

[0087] FIG. 4 is a flowchart illustrating the execution of an object pose classification method within image data using an end-to-end model according to some embodiments of the present disclosure.

[0088] Referring to FIG. 4, in some embodiments of the present disclosure, original image data may be input into a pose classification model (S100). The original image data input in step (S100) may be converted into a preprocessed image by performing preprocessing through step (S200). Feature data is extracted from the preprocessed image (S300), and the extracted feature data may be used for object recognition and keypoint prediction (S400). After the execution of step (S400), the feature data extracted in step (S300), the object recognition information and keypoint information extracted in step (S400) are combined and input into a pose classification model to generate pose information of an object included in the image data (S500). The generated pose information, object recognition information, and keypoint information may be combined with the original image data and visualized (S600).

[0089]

[0090] FIG. 5 is a block diagram showing a computing device for providing the posture classification method of the present disclosure.

[0091] The computing device (800) may include a memory (810), a processor (820), a communication unit (830), and an input / output interface (840), and as shown in FIG. 5, the computing device (800) may be configured to communicate information and / or data through a network using the communication unit (830).

[0092] The memory (810) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (810) may include a permanent mass storage device such as random access memory (RAM), read-only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. As another example, a permanent mass storage device such as ROM, SSD, flash memory, disk drive, etc. may be included in the computing device (800) as a separate permanent storage device distinct from the memory. Additionally, an operating system and at least one program code may be stored in the memory (810).

[0093] These software components may be loaded from a computer-readable recording medium separate from the memory (810). This separate computer-readable recording medium may include a recording medium that can be directly connected to the computing device (800), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. As another example, the software components may be loaded into the memory (810) via a communication unit (830) rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory (810) based on a computer program installed by files provided through the communication unit (830) by developers or a file distribution system that distributes installation files for applications.

[0094] Here, the program may include program instructions, data files, and data structures, either individually or in combination. The program may be designed and created using machine code or high-level language code. The program may be specifically designed to implement the aforementioned attitude classification method, or it may be implemented using various functions or definitions that are known and available to a person skilled in the art of computer software. The program for implementing the aforementioned attitude classification method may be recorded on a recording medium readable by a processor.

[0095] Memory can store a program that performs the aforementioned operation and the operation described below, and the processor can execute the program stored in memory. When there are multiple processors and memory, it is possible for them to be integrated on a single chip or to be provided in physically separate locations. Memory may include volatile memory such as S-RAM (Static Random Access Memory, S-RAM) and D-RAM (Dynamic Random Access Memory) for temporarily storing data. Additionally, memory may include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory) for long-term storage of control programs and control data.

[0096]

[0097] The processor (820) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to another user terminal (not shown) or another external system by memory (810) or communication unit (830).

[0098] The processor may include various logic circuits and arithmetic circuits, process data according to a program provided from memory, and generate control signals based on the processing results. In this case, the memory and the processor may each be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

[0099] The processor (820) may include one or more processors. In this case, the one or more processors may be homogeneous or heterogeneous processors. For example, the processor (820) may include heterogeneous processors such as a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU) and / or a neural processing unit (NPU). For convenience of explanation, one or more homogeneous or heterogeneous processors may be referred to as 'processors' within this specification.

[0100] The communication unit (830) may provide a configuration or function for a user terminal (not shown) and a computing device (800) to communicate with each other via a network, and may provide a configuration or function for the computing device (800) to communicate with an external system (e.g., a separate cloud system). For example, control signals, commands, data, etc. provided under the control of the processor (820) of the computing device (800) may be transmitted to the user terminal and / or the external system through the communication unit (830) and the network, and through the communication unit of the user terminal and / or the external system.

[0101] The wired communication unit may include various wired communication units such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication units such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0102] In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, the wireless communication unit may include a wireless communication unit that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G.

[0103] The wireless communication unit may include a wireless communication interface comprising an antenna and a transmitter that transmit a Wi-Fi signal. Additionally, the wireless communication unit may further include a Wi-Fi signal conversion module that modulates a digital control signal output from the control unit through the wireless communication interface into an analog wireless signal under the control of the control unit.

[0104] The wireless communication unit may include a wireless communication interface comprising an antenna and a receiver for receiving a Wi-Fi signal. Additionally, the wireless communication unit may further include a Wi-Fi signal conversion module for demodulating an analog wireless signal received through the wireless communication interface into a digital control signal.

[0105] The short-range communication unit is for short-range communication and can support short-range communication by using at least one of Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0106] Additionally, the input / output interface (840) of the computing device (800) may be a means for interfacing with a device (not shown) for input or output that is connected to the computing device (800) or that the computing device (800) may include.

[0107] The input unit is for receiving information and data, including audio information (or signals), text, etc., from a network or a user, and may include at least one microphone and at least one of a user input unit. Data collected from the input unit may be analyzed and processed into a user control command.

[0108] The user input unit is intended to receive information from a user, and when information is input through the user input unit, the control unit can control the operation of the device to correspond to the input information. Such a user input unit may include a hardware physical key (e.g., a button, dome switch, jog wheel, jog switch, etc. located on at least one of the front, rear, and side of the device) and a software touch key. As an example, the touch key may consist of a virtual key, soft key, or visual key displayed on a touchscreen-type display unit through software processing, or may consist of a touch key placed on a part other than the touchscreen. Meanwhile, the virtual key or visual key may have various forms and may be displayed on the touchscreen, for example, as a graphic, text, icon, video, or a combination thereof.

[0109] In FIG. 5, the input / output interface (840) is shown as an element configured separately from the processor (820), but is not limited thereto, and the input / output interface (840) may be configured to be included in the processor (820). The computing device (800) may include more components than those in FIG. 5. However, there is no need to clearly illustrate most of the prior art components.

[0110] The processor (820) of the computing device (800) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals and / or a plurality of external systems.

[0111] The methods and / or various embodiments described above may be realized in digital electronic circuits, computer hardware, firmware, software, and / or combinations thereof. Various embodiments of the present disclosure may be executed by a data processing device, for example, one or more programmable processors and / or one or more computing devices, or may be implemented as a computer program stored on a computer-readable recording medium and / or a computer program stored on a computer-readable recording medium. The computer program described above may be written in any form of programming language, including a compiled language or an interpreted language, and may be distributed in any form, such as a standalone program, a module, a subroutine, etc. The computer program may be distributed through a single computing device, a plurality of computing devices connected through the same network, and / or a plurality of computing devices distributed to be connected through a plurality of different networks.

[0112] The methods and / or various embodiments described above may be performed by one or more processors configured to execute one or more computer programs that process, store, and / or manage any functions, functions, etc. by operating based on raw samples or generating output data. For example, the methods and / or various embodiments of the present disclosure may be performed by special-purpose logic circuits such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and an apparatus and / or system for performing the methods and / or embodiments of the present disclosure may be implemented as a special-purpose logic circuit such as an FPGA or an ASIC.

[0113] One or more processors executing a computer program may include one or more processors of a general-purpose or special-purpose microprocessor and / or any type of digital computing device. The processor may receive instructions and / or data from each of read-only memory and random access memory, or receive instructions and / or data from read-only memory and random access memory. In the present invention, components of a computing device performing the methods and / or embodiments may include one or more processors for executing instructions and one or more memory devices for storing instructions and / or data.

[0114] According to one embodiment, a computing device may exchange data with one or more mass storage devices for storing data. For example, the computing device may receive and / or receive data from a magnetic disc or an optical disc, and may transfer data to a magnetic disc or an optical disc. A computer-readable storage medium suitable for storing instructions and / or data associated with a computer program may include, but is not limited to, any form of non-volatile memory including semiconductor memory devices such as EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable PROM), and flash memory devices. For example, a computer-readable storage medium may include magnetic discs such as internal hard disks or removable disks, optical magnetic discs, CD-ROMs, and DVD-ROMs.

[0115] To provide interaction with a user, the computing device may include, but is not limited to, a display device for providing or displaying information to the user (e.g., CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), etc.) and a pointing device (e.g., keyboard, mouse, trackball, etc.) on which the user can provide input and / or commands, etc. on the computing device. That is, the computing device may further include any other type of device for providing interaction with the user. For example, the computing device may provide any form of sensory feedback to the user for interaction with the user, including visual feedback, auditory feedback and / or tactile feedback. In this regard, the user may provide input to the computing device through various gestures such as visual, vocal, and motion.

[0116] In the present invention, various embodiments may be implemented in a computing system comprising backend components (e.g., data servers), middleware components (e.g., application servers), and / or frontend components. In this case, the components may be interconnected by any form or medium of digital data communication, such as a communication network. For example, the communication network may include a Local Area Network (LAN), a Wide Area Network (WAN), etc.

[0117] A computing device based on the exemplary embodiments described herein may be implemented using hardware and / or software configured to interact with a user, including a user device, a user interface (UI) device, a user terminal, or a client device. For example, the computing device may include a portable computing device such as a laptop computer. Additionally or alternatively, the computing device may include, but is not limited to, Personal Digital Assistants (PDAs), tablet PCs, game consoles, wearable devices, Internet of Things (IoT) devices, Virtual Reality (VR) devices, Augmented Reality (AR) devices, etc. The computing device may further include other types of devices configured to interact with a user. Additionally, the computing device may include a portable communication device suitable for wireless communication over a network such as a mobile communication network (e.g., a mobile phone, a smartphone, a wireless cellular phone, etc.). A computing device may be configured to communicate wirelessly with a network server using wireless communication technologies and / or protocols such as radio frequency (RF), microwave frequency (MWF) and / or infrared frequency (IRF).

[0118] In an embodiment according to the present disclosure, functions related to artificial intelligence may be implemented through a processor and memory. In this case, the processor may be any one of a general-purpose processor such as a CPU (Center Processing Unit), AP (Application Processor), DSP (Digital Signal Processor), a graphics-dedicated processor such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit), and an artificial intelligence-dedicated processor such as an NPU (Neural Network Processing Unit). The processor may process raw samples according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the processor is an artificial intelligence-dedicated processor, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model. In some embodiments according to the present disclosure, functions related to artificial intelligence may be implemented through a plurality of processors.

[0119] In an embodiment according to the present disclosure, a predefined operation rule or artificial intelligence model may be configured to perform machine learning. Here, being configured to perform machine learning means that the predefined operation rule or artificial intelligence model is configured to perform a desired characteristic (or objective) by learning using a plurality of training data based on a learning algorithm. Such learning may be performed on the device itself in which the artificial intelligence according to the present disclosure is implemented, or it may be performed through a separate server and / or system.

[0120] Artificial intelligence models can be implemented as neural networks (or artificial neural networks) and can operate based on statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network can refer to a model in which artificial neurons (nodes), which form a network through synaptic connections, change the strength of synaptic connections through learning to possess problem-solving capabilities. A neural network can be composed of multiple neural network layers; for example, a neural network may include an input layer, a hidden layer, and an output layer. Each of the multiple neural network layers may include at least one node and at least one weight, and neural network operations can be performed through operations between the results of previous (precious) layers and the weights. At least one weight possessed by the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, at least one weight may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Neural networks can infer a result to be predicted from an arbitrary input.

[0121] The learning methods of artificial intelligence models can be classified according to the learning approach into supervised learning, where raw samples and output data are provided as training data and the correct answer (output data) corresponding to the problem (raw samples) is predetermined; unsupervised learning, where only raw samples are provided without output data and the correct answer (output data) corresponding to the problem (raw samples) is not predetermined; and reinforcement learning, where a reward is granted whenever an action is taken from the current state and learning proceeds in a direction that maximizes this reward. Alternatively, they can be classified according to the architecture, which is the structure of the learning model.

[0122] In the embodiments of the present disclosure, the artificial intelligence model is a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory Network (LSTM), Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA for Natural Language Processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics, Visual Understanding, Video Synthesis for Vision Processing, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation for ResNet Data Intelligence, At least one of various artificial intelligence structures and algorithms, such as data creation, may be used. The examples described above are merely examples of artificial intelligence structures and algorithms used according to the embodiments of the present disclosure and do not limit the artificial intelligence structures and algorithms used according to the embodiments of the present disclosure.

[0123] Meanwhile, the embodiments disclosed in this specification may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium. A computer-readable recording medium may include all types of recording media that store instructions decipherable by a computer. Examples include ROM, RAM, magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0124] The foregoing describes specific embodiments for implementing the present disclosure. The present disclosure will include not only the embodiments described above, but also embodiments that are simply modified or can be easily modified. Furthermore, the present disclosure will include technologies that can be easily modified and implemented using the embodiments described above.

[0125] Various embodiments of the present invention, including specific structural and functional details, are exemplary. Accordingly, the embodiments of the present disclosure are not limited to those described above and may be implemented in various other forms. Furthermore, the terms used in the present invention are intended to describe some embodiments and are not to be interpreted as limiting the embodiments. For example, singular words and the above may be interpreted to include plural forms unless the context clearly indicates otherwise.

[0126] In this invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which such concepts belong. Furthermore, commonly used terms, such as those defined in advance, should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology.

[0127] Although the present invention has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to this specification.

Claims

1. A posture classification method executed by one or more processors, wherein the posture classification method comprises: Step of training an end-to-end pose classification model; and The method includes the step of performing pose classification of objects included in the original image data using the aforementioned previously trained end-to-end pose classification model, The step of performing the above posture classification is, Step of receiving original image data; A step of performing preprocessing on the received original image data to generate preprocessed image data; A step of generating feature data from the preprocessed image data by an image feature extraction module comprising one or more convolutional layers; A step of generating object recognition information and keypoint information from the above feature data; and Characterized by including the step of generating pose information of the object from the above feature data, the above object recognition information, and keypoint information. Detail classification method.

2. In Paragraph 1, The above feature data is, Characterized by including multiple feature maps, Detail classification method.

3. In Paragraph 2, The above feature data is, A first feature map corresponding to the first convolution layer, which is the last layer of the image feature extraction module; A second feature map corresponding to a second convolution layer, which is a layer preceding the first convolution layer; and Characterized by including a third feature map corresponding to a third convolution layer, which is a layer preceding the second convolution layer. Detail classification method.

4. In Paragraph 2, The above plurality of feature maps are, Characterized by including a first feature map including low-dimensional features of the original image data and a second feature map including high-dimensional features of the original image data. Detail classification method.

5. In Paragraph 1, The step of generating posture information of the above object is, Characterized by including the step of generating combined input data by combining the above feature data, the above object recognition information, and the keypoint information. Detail classification method.

6. In Paragraph 5, The step of generating the above combined input data is, Characterized by including the step of connecting the above feature data and the above object recognition information and keypoint information, Detail classification method.

7. In Paragraph 6, The step of connecting the above feature data and the above object recognition information and keypoint information is, Characterized by including the step of connecting the object recognition information and keypoint information for each feature map included in the above feature data. Detail classification method.

8. In Paragraph 5, The step of generating the above combined input data is, Characterized by performing an attention operation on the above feature data and the above object recognition information and keypoint information. Detail classification method.

9. In Paragraph 5, The above posture classification model is, It includes an image feature extraction module, an object recognition and keypoint prediction module, and a pose classification module, and The step of training the above-mentioned end-to-end pose classification model is, Characterized by simultaneously learning the image feature extraction module, the object recognition and keypoint prediction module, and the pose classification module. Detail classification method.

10. A posture classification device that performs the posture classification method according to claim 1.

11. A computer-readable, non-transient recording medium having a program recorded thereon for executing the posture classification method according to claim 1.

Images