Method for reflecting user data in real time for pose estimation in behavior experiment

Abstract

The present specification provides a method by which a server reflects user data in real time for a pose estimation AI model in a behavior experiment, the method comprising the steps of: analyzing a video input through a terminal, and estimating a pose of a target by using the pose estimation AI model; performing scoring for each frame of the video; performing chunk scoring on the basis of the scoring for each frame; performing image curation of a chunk on the basis of the chunk scoring; labeling a selected frame through the image curation; and reflecting data of the labeled frame to the pose estimation AI model.

Description

Method for incorporating real-time user data for behavioral experiment pose estimation

[0001] This specification relates to a system and method for reflecting user data in real time into an artificial intelligence model for pose estimation in animal behavior experiments.

[0002] Animal behavior analysis refers to the process of understanding and analyzing animal behavior patterns during animal experiments, including non-clinical trials. This enhances the reliability of experimental results through the observation and evaluation of animal behavior and is utilized in various studies employing animal models. However, traditional animal behavior analysis has largely relied on the subjective observations of experts, which has resulted in limitations in the consistency and reproducibility of findings. To automate this process, AI-based behavior analysis solutions have been developed. Representative solutions include EthoVision and DeepLabCut; these systems automatically analyze animal behavior, providing higher accuracy and efficiency than methods that relied on human observation.

[0003] AI-based analysis solutions generally adopt pose estimation methods rather than image-based analysis to extract features robust to changes in the experimental environment (e.g., background images) and optical conditions (e.g., illumination, saturation). Pose estimation identifies behavioral patterns by estimating the positions of key body parts of an animal and is a critical factor determining the overall accuracy of the analysis solution. Therefore, improving the accuracy of pose estimation is a critical task that determines the performance of automated animal behavior analysis.

[0004] Currently, the most accurate pose estimation method utilizes AI models. However, because AI models predict results based on data, they have a limitation in that they exhibit lower accuracy in unseen environments compared to seen environments. To address this, methods involving training models using large-scale datasets that include diverse environments have been primarily used. This approach enables the AI ​​model to achieve high accuracy in unlearned environments by learning from various environmental changes.

[0005] However, in the case of animal behavior analysis, compared to human behavior analysis, there is a problem in that the size of available datasets is relatively small and domain differences are significant due to the diversity of experimental environments. Since behavioral characteristics and physical appearance vary depending on the animal species, breed, and experimental conditions, it is difficult to overcome the domain differences arising from diverse experimental environments by using only general datasets. Accordingly, a new methodology is needed to overcome the domain gap between training data and actual user data.

[0006] The purpose of this specification is to implement a system and method for inferring results from user data through an AI model and simultaneously performing a performance evaluation, thereby reflecting user data in the AI ​​model in real time.

[0007] The technical problems that this specification aims to solve are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this specification belongs from the detailed description of the specification below.

[0008] One aspect of the present specification is a method for a server to reflect user data in real time for a pose estimation AI model in a behavioral experiment, comprising: a step of analyzing an image received through a terminal and estimating the pose of a target using the pose estimation AI model; a step of performing scoring for each frame of the image; a step of performing chunk scoring based on the frame-by-frame scoring; a step of performing image curation of the chunk based on the chunk scoring; a step of performing labeling for a selected frame through the image curation; and a step of reflecting the data of the labeled frame into the pose estimation AI model.

[0009] Additionally, the step of performing scoring for each frame of the above video may include: a step of extracting a background image for each frame; a step of generating a foreground mask for each frame based on the background image; and a step of performing scoring for each frame based on the foreground mask and the result of the pose estimation.

[0010] In addition, the step of performing scoring for each frame based on the foreground mask and the result of the pose estimation may be performed by scoring based on the degree of agreement between the foreground mask and the result of the pose estimation.

[0011] Additionally, the step of performing the chunk scoring above may be performed based on the average score of the frames included in the chunk.

[0012] Additionally, the step of performing image curation of the chunk may include: a step of selecting labeling candidates based on the chunk scoring and the scores of the frames included in the chunk; a step of additionally selecting the labeling candidates through the pose estimation AI model; a step of additionally sampling random frames from the labeling candidates; and a step of removing similar frames based on the labeling candidates.

[0013] In addition, the step of additionally sampling random frames from the labeling candidates may involve selecting 70% of the frames from the labeling candidates from frames with low scores, and selecting the remaining 30% by random sampling.

[0014] In addition, the step of reflecting the data of the labeled frame into the pose estimation AI model may perform fine-tuning of the pose estimation AI model using the data of the labeled frame.

[0015] In addition, the step of reflecting the data of the labeled frames into the pose estimation AI model is processed in real time for the video through a streaming method, and can be analyzed and processed on a frame-by-frame basis rather than the entire video uploaded by the user.

[0016] Another aspect of the present specification comprises, in a server for reflecting user data in real time for a pose estimation AI model in a behavioral experiment, the pose estimation AI model for extracting key points from an image transmitted from a terminal; an automated evaluation module for improving the accuracy of the pose estimation AI model; and a process for functionally controlling the pose estimation AI model and the automated evaluation module; wherein the process

[0017] By analyzing the input video through the terminal, the pose of the target is estimated using the pose estimation AI model, and scoring is performed frame by frame of the video through the automated evaluation module. Based on the frame-by-frame scoring, chunk scoring is performed, and based on the chunk scoring, image curation of the chunk is performed. Through the image curation, labeling is performed on the selected frames, and the data of the labeled frames can be reflected in the pose estimation AI model.

[0018]

[0019] According to an embodiment of the present specification, a system and method for reflecting user data in real time into an AI model can be implemented by inferring results through an AI model and simultaneously performing a performance evaluation on user data.

[0020] The effects obtainable in this specification are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which this specification belongs from the description below.

[0021]

[0022] FIG. 1 is a block diagram for illustrating an electronic device related to the present specification.

[0023] FIG. 2 is a block diagram of an AI device according to one embodiment of the present specification.

[0024] FIG. 3 illustrates a system for reflecting user data in real time to which the present specification may be applied.

[0025] FIG. 4 illustrates a method for reflecting real-time user data for behavioral experiment pose estimation to which the present specification may be applied.

[0026] FIG. 5 illustrates background image extraction to which the present specification can be applied.

[0027] FIG. 6 illustrates frame-by-frame scoring to which the present specification can be applied.

[0028] FIG. 7 illustrates a Foreground Mask to which the present specification can be applied.

[0029] FIG. 8 is an example of image curation to which the present specification may be applied.

[0030] The accompanying drawings, included as part of the detailed description to aid in understanding the present specification, provide embodiments of the present specification and explain the technical features of the present specification together with the detailed description.

[0031]

[0032] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the concept and technical scope of this specification.

[0033] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0034] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0035] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0036] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0037]

[0038] FIG. 1 is a block diagram for illustrating an electronic device related to the present specification.

[0039] The above electronic device (100) may include a wireless communication unit (110), an input unit (120), a sensing unit (140), an output unit (150), an interface unit (160), a memory (170), a control unit (180), and a power supply unit (190), etc. Since the components illustrated in FIG. 1 are not essential for implementing the electronic device, the electronic device described herein may have more or fewer components than those listed above.

[0040] More specifically, among the above components, the wireless communication unit (110) may include one or more modules that enable wireless communication between the electronic device (100) and a wireless communication system, between the electronic device (100) and another electronic device (100), or between the electronic device (100) and an external server. Additionally, the wireless communication unit (110) may include one or more modules that connect the electronic device (100) to one or more networks.

[0041] This wireless communication unit (110) may include at least one of a broadcast receiving module (111), a mobile communication module (112), a wireless internet module (113), a short-range communication module (114), and a location information module (115).

[0042] The input unit (120) may include a camera (121) or video input unit for inputting a video signal, a microphone (122) or audio input unit for inputting an audio signal, and a user input unit (123, e.g., a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected from the input unit (120) may be analyzed and processed into a control command by the user.

[0043] The sensing unit (140) may include one or more sensors for sensing at least one of information within the electronic device, information about the surrounding environment surrounding the electronic device, and user information. For example, the sensing unit (140) may include at least one of a proximity sensor (141), an illumination sensor (142), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor: infrared sensor), a fingerprint sensor (finger scan sensor), an ultrasonic sensor, an optical sensor (e.g., see camera (121)), a microphone (see 122), a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the electronic device disclosed in this specification can utilize information sensed by at least two of these sensors in combination.

[0044] The output unit (150) is intended to generate output related to sight, hearing, or touch, and may include at least one of a display unit (151), an audio output unit (152), a haptic module (153), and an optical output unit (154). The display unit (151) may form a layered structure with a touch sensor or be formed integrally to implement a touch screen. Such a touch screen functions as a user input unit (123) that provides an input interface between the electronic device (100) and the user, and at the same time can provide an output interface between the electronic device (100) and the user.

[0045] The interface section (160) serves as a passage for various types of external devices connected to the electronic device (100). This interface section (160) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port. In response to an external device being connected to the interface section (160), the electronic device (100) can perform appropriate control related to the connected external device.

[0046] Additionally, the memory (170) stores data that supports various functions of the electronic device (100). The memory (170) can store a number of application programs (or applications) running on the electronic device (100), data for the operation of the electronic device (100), and commands. At least some of these application programs may be downloaded from an external server via wireless communication. Also, at least some of these application programs may exist on the electronic device (100) from the time of shipment for the basic functions of the electronic device (100) (e.g., phone incoming and outgoing functions, message receiving and outgoing functions). Meanwhile, the application programs may be stored in the memory (170), installed on the electronic device (100), and driven by the control unit (180) to perform the operation (or function) of the electronic device.

[0047] In addition to operations related to the application program, the control unit (180) typically controls the overall operation of the electronic device (100). The control unit (180) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. that are input or output through the components described above, or by running an application program stored in memory (170).

[0048] Additionally, the control unit (180) can control at least some of the components examined together with FIG. 1 in order to run an application program stored in memory (170). Furthermore, the control unit (180) can operate at least two or more of the components included in the electronic device (100) in combination with each other to run the application program.

[0049] The power supply unit (190) receives external power and internal power under the control of the control unit (180) and supplies power to each component included in the electronic device (100). This power supply unit (190) includes a battery, and the battery may be a built-in battery or a replaceable battery.

[0050] At least some of the above components may operate in cooperation with each other to implement the operation, control, or control method of an electronic device according to various embodiments described below. Additionally, the operation, control, or control method of the electronic device may be implemented on the electronic device by running at least one application program stored in the memory (170).

[0051] In this specification, the electronic device (100) may be collectively referred to as a server, and the server may include a cloud server. Additionally, the terminal may include all or part of the configuration of the electronic device (100), and may include a tablet PC.

[0052]

[0053] FIG. 2 is a block diagram of an AI device according to one embodiment of the present specification.

[0054] The AI ​​device (20) may include an electronic device including an AI module capable of performing AI processing, or a terminal including the AI ​​module. Additionally, the AI ​​device (20) may be configured to be included as at least a part of the configuration of the electronic device (100) shown in FIG. 1 to perform at least a part of the AI ​​processing together.

[0055] The above AI device (20) may include an AI processor (21), memory (25) and / or a communication unit (27).

[0056] The above AI device (20) is a computing device capable of learning a neural network and can be implemented as various electronic devices such as a terminal, desktop PC, laptop PC, tablet PC, etc.

[0057] The AI ​​processor (21) can train a neural network using a program stored in memory (25). In particular, the AI ​​processor (21) may include a large-scale pre-trained pose estimation model. For example, the pose estimation model can predict key points of a target in video frames streamed in real time. Through the pose estimation model, the AI ​​processor (21) continuously analyzes streamed frames to provide pose estimation results and can automatically evaluate frame-by-frame accuracy to perform automatic labeling on frames with low reliability. Additionally, the pose estimation model can be further fine-tuned using the collected labeling data to improve performance specialized for experimental environments.

[0058] Meanwhile, the AI ​​processor (21) that performs the functions described above may be a general-purpose processor (e.g., CPU), but may be an AI-dedicated processor for artificial intelligence learning (e.g., GPU, graphics processing unit).

[0059] The memory (25) can store various programs and data required for the operation of the AI ​​device (20). The memory (25) can be implemented as non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD). The memory (25) is accessed by the AI ​​processor (21), and the reading / writing / modification / deletion / updating of data by the AI ​​processor (21) can be performed. Additionally, the memory (25) can store a neural network model (e.g., a deep learning model) generated through a learning algorithm for data classification / recognition according to one embodiment of the present specification.

[0060] Meanwhile, the AI ​​processor (21) may include a data learning unit that learns a neural network for data classification / recognition. For example, the data learning unit may learn a deep learning model by acquiring training data to be used for learning and applying the acquired training data to a deep learning model.

[0061] The communication unit (27) can transmit the AI ​​processing results by the AI ​​processor (21) to an external electronic device.

[0062] Here, external electronic devices may include other terminals.

[0063] Meanwhile, although the AI ​​device (20) illustrated in FIG. 2 is described by functionally separating it into an AI processor (21), memory (25), and communication unit (27), the aforementioned components may be integrated into a single module and referred to as an AI module or an artificial intelligence (AI) model.

[0064] FIG. 3 illustrates a system for reflecting user data in real time to which the present specification may be applied.

[0065] Referring to FIG. 3, the server (300) can receive video data in real time from the terminal (310). The server (300) may include a pose estimation AI model (320) and an automated evaluation module (330).

[0066] The terminal (310) serves as an interface for the user to interact with the server. It can be implemented as a device such as a smartphone, tablet, or computer, and receives video data for behavior analysis from the user and transmits it to the server (300), receiving results related to pose estimation from the server (300) and outputting them to the user. Through the terminal (310), the user can receive animal behavior analysis services from the server (300) via the web. Since the transmission of this video data operates in a streaming manner, it is possible to transmit it immediately to the server (300) frame by frame without the need to upload the entire video. This allows for the initiation of rapid analysis and pose estimation, and the user can check the initial results in real time without any waiting time.

[0067] The server (300) is a device that receives and processes video data streamed from the terminal (310). The server (300) performs real-time data analysis and evaluation including a pose estimation AI model (320) and an automated evaluation module (330), and can process requests from multiple users in parallel. For example, the server (300) can generate pose estimation results frame by frame and continuously improve the performance of the pose estimation AI model (320) through additional labeling and model retraining as needed.

[0068] The pose estimation AI model (320) can extract key points from a real-time video stream transmitted from a terminal (310). For example, this model, which has been trained in advance with a large dataset, can predict the pose of a target by processing data frame by frame. The extracted pose information can be transmitted to an automated evaluation module (330) for further verification and evaluation.

[0069] To improve the accuracy of the pose estimation AI model (320), the server (300) can collect additional data on frames with low reliability through the automated evaluation module (330) and, if necessary, retrain the pose estimation AI model (320) through fine-tuning.

[0070] The automated evaluation module (330) can evaluate accuracy based on pose data generated by the pose estimation AI model (320). For example, the automated evaluation module (330) can calculate the reliability of the pose as a score for each frame and determine whether additional labeling is required for frames with low scores. Through this, low-reliability frames can be selected, and data can be generated to improve the performance of the pose estimation AI model (320) through a labeling process.

[0071] Additionally, the automated evaluation module (330) can determine the quality of the entire video by performing evaluations not only on a frame-by-frame basis but also on a chunk-by-chunk basis. It can improve accuracy by performing image curation on chunks that do not meet the standard score and automatically and / or manually labeling the necessary frames. For example, in this specification, a chunk may refer to a video frame of a certain length (in minutes). Finally, the automated evaluation module (330) enables the provision of more precise results in real time by supporting the continuous improvement of the pose estimation AI model (320).

[0072] FIG. 4 illustrates a method for reflecting real-time user data for behavioral experiment pose estimation to which the present specification may be applied.

[0073] Referring to FIG. 4, the server (300) can reflect real-time user data in the pose estimation AI model (320).

[0074] The server (300) analyzes the video input from the terminal (310) to estimate the pose (S4010). For example, the server (300) receives video data streamed or uploaded from a user and estimates the pose of the target for each frame through a pose estimation AI model (320). Pose estimation includes a process of predicting key points (joint positions, etc.), and this result can serve as the basis for subsequent evaluation and labeling processes. The pose estimation AI model (320) can generate frame-by-frame estimation results in real time based on pre-trained data.

[0075] The server (300) performs scoring for each frame of the video (S4020). For example, the server (300) can extract a background image for each frame of the video, measure accuracy by comparing the foreground mask of each frame with the estimated keypoint through an automated evaluation module (310), and calculate a score for each frame based on this. For example, the server (300) can extract a background image of the video using a median of median algorithm. This background image extraction may be a process of generating an image to be considered as a background by calculating the median value of pixels that appear commonly in multiple frames.

[0076] FIG. 5 illustrates background image extraction to which the present specification can be applied.

[0077] Referring to FIG. 5, the server (300) can extract a background image based on a median of median algorithm from frames of an image. The server (300) can generate an image with only the elements constituting the background remaining by selecting multiple frames of an image and calculating the median value of each pixel position. Through this, the server (300) can remove moving elements, such as objects in the image (e.g., a mouse), and extract only a static background that does not change.

[0078] More specifically, the server (300) can divide the entire frame into window sizes and then calculate an intermediate value by comparing the same pixel location values ​​of the frames included in each window. Through the intermediate value calculated in this way, a single background image can be generated, and finally, the background images generated from multiple windows can be processed once again in the same way to complete the final background image. For example, as shown in the result image at the bottom of FIG. 5, an image is returned in which only the background remains and the moving object (mouse) is removed. This method is efficient because it can generate a background image of similar quality with less memory than a general median algorithm.

[0079] Afterwards, the server (300) can perform scoring for each frame of the video through the automated evaluation module (310).

[0080] FIG. 6 illustrates frame-by-frame scoring to which the present specification can be applied.

[0081] Referring to FIG. 6, the automated evaluation module (310) generates a Foreground Mask for each frame based on the background image (S6010). For example, the automated evaluation module (310) converts the background image to grayscale and then sequentially converts each frame of the image to grayscale. The automated evaluation module (310) calculates the pixel difference between the converted frames and the background image, and considers pixels where the difference is greater than or equal to a specific threshold (e.g., 50) as Foreground and sets them as a mask (e.g., marked as True). Through this process, the automated evaluation module (310) can effectively separate only the area where the dynamic object is located.

[0082] FIG. 7 illustrates a Foreground Mask to which the present specification can be applied.

[0083] Referring to FIG. 7, the automated evaluation module (310) can generate a Foreground Mask for the input frame.

[0084] Referring again to FIG. 6, the automated evaluation module (310) performs frame-by-frame scoring by comparing the Foreground Mask with the pose estimation result (S6020). The automated evaluation module (310) checks whether each key point of the estimated pose is located inside the Foreground Mask and can calculate a score based on the ratio of the number of key points included in the Foreground Mask to the total number of key points. For example, if 9 out of 11 key points of the estimated pose are included in the Foreground Mask, the score for the corresponding frame can be calculated as 9 / 11 = 0.82.

[0085] The automated evaluation module (310) stores the calculated frame-by-frame scores (S6030). Subsequently, the server (300) can calculate the average score in chunks. For example, the score of each frame is stored in a separate array, and this data can be used in the chunk scoring and image curation steps.

[0086] Referring again to FIG. 4, the server (300) performs chunk scoring based on frame-by-frame scores through an automated evaluation module (310) (S4030). For example, the automated evaluation module (310) can calculate an average score for chunks divided into fixed time units (e.g., 5 minutes). Each chunk consists of multiple frames, and the chunk score can be determined through the average of the frame scores. The chunk score indicates how reliable the pose estimation result of the corresponding segment is, and chunks lower than the threshold value can be selected as targets for image curation.

[0087] More specifically, the automated evaluation module (310) can generate a chunk array by dividing the frames of the video into fixed time units. For example, if the chunk is 5 minutes long, it contains 9,000 frames per chunk based on 30fps. Frame-by-frame scores already calculated for each frame can be stored in the chunk array, and a chunk-unit average score can be calculated based on this array. Alternatively, the automated evaluation module (310) can sort the frame scores stored in the chunk array and remove noise and outliers by removing the 5% intervals of the minimum and maximum values. Afterward, the average of the remaining values ​​can be calculated and used as the final chunk score. Through this, real-time labeling can be performed for very long videos or live experiments.

[0088] The server (300) performs image curation of chunks based on chunk scores (S4040). The server (300) can perform image curation for chunks with scores lower than a reference value. In this process, the server (300) selects frames with low reliability and randomly adds frames at a certain ratio to select the final frames to be labeled.

[0089] FIG. 8 is an example of image curation to which the present specification may be applied.

[0090] Referring to FIG. 8, the server (300) selects labeling candidates based on the frame score according to chunk scoring (S8010). First, the server (300) can determine the number of labeling frames required for a chunk according to the number of required labels per score that is predefined. For example, 100 labeling frames may be required if the chunk score is 0.7 to 0.8, 200 if it is 0.6 to 0.7, and 400 if it is 0.6 or less. Subsequently, the server (300) can apply a predefined threshold value (e.g., 0.80) to the score of individual frames and filter out frames that are scored below this threshold value. The frames filtered in this way may be selected as labeling candidates based on the number of labeling frames required according to the chunk score.

[0091] The server (300) additionally selects labeling candidates through the pose estimation model AI model (320) (S8020). The server (300) can perform additional evaluations based on the confidence score provided by the pose estimation model AI model (320). For example, frames with a confidence score less than or equal to a pre-set threshold value (e.g., 0.70) can be additionally selected as labeling candidates.

[0092] The server (300) additionally samples random frames from the labeling candidates (S8030). The server (300) may randomly sample frames to compensate for a shortage of frames to be labeled. For example, 30% of the total number of required labeling frames may be additionally selected randomly regardless of the score. Alternatively, a certain multiple of the selected labeling candidates may be selected by random sampling, excluding the selected labeling candidates. Frames selected randomly in this way can serve to compensate for various data patterns that may be omitted based on the score.

[0093] The server (300) removes similar frames based on labeling candidates (S8040). The server (300) can remove duplicate or similar frames by utilizing a similarity comparison algorithm on the labeling candidate frames. Through this, frames to be labeled can be selected. Subsequently, the number of selected frames can be adjusted according to the required number of labels for each chunk score. Referring again to FIG. 4, the server (300) performs labeling on the selected frames through image curation (S4050). For example, the server (300) can perform automatic labeling and / or manual labeling on the selected frames. Automatic labeling is a task that quickly generates labels through a pre-trained AI model, and if necessary, additional accuracy can be secured through manual labeling. Manual labeling can be performed through crowdsourcing or review by internal experts.

[0094] The server (300) applies the labeled data to the pose estimation AI model (320) (S4060). For example, the server (300) can fine-tun the model by applying the labeled data to the existing pose estimation AI model (320). The retrained pose estimation AI model (320) is updated on the server (300), and can provide more precise pose estimation results in real time for incoming data. Through this, the server (300) can support the pose estimation AI model (320) to continuously learn and improve.

[0095] In this specification, the server (300) can immediately proceed with the evaluation, selection of frames for labeling, and labeling learning processes based on streaming transmission from the moment the video is uploaded in order to ensure real-time performance. In addition, by utilizing image curation techniques and auto-labeling, the time required for labeling can be reduced and the efficiency of the entire process can be maximized. Through this design, the user can obtain improved pose estimation results in a short period of time.

[0096] Furthermore, this specification aims to ensure high accuracy in various environments. Performance can be maximized by reflecting user data in the model in real time to overcome domain differences between existing training data and actual user data. In addition, the proposed method includes an algorithm capable of operating regardless of video length, allowing for stable operation even when uploading completed footage or transmitting streaming footage. Consequently, a flexible system can be provided that can process data without issues even if video length information is unclear.

[0097] The foregoing specification may be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drives (HDDs), Solid State Disks (SSDs), Silicon Disk Drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include implementations in the form of carrier waves (e.g., transmission over the Internet). Accordingly, the above detailed description should not be interpreted restrictively in all respects and should be considered exemplary. The scope of this specification should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of this specification are included within the scope of this specification.

[0098] Furthermore, although the above description has focused on the services and embodiments, this is merely illustrative and does not limit the scope of this specification. Those skilled in the art will understand that various modifications and applications not exemplified above are possible without departing from the essential characteristics of the services and embodiments. For example, each component specifically shown in the embodiments may be modified and implemented. Differences related to such modifications and applications should be interpreted as being included within the scope of this specification as defined in the appended claims.

Claims

1. A method for a server to reflect user data in real time for a pose estimation AI model in a behavioral experiment, A step of analyzing an input image through a terminal and estimating the pose of a target using the pose estimation AI model; A step of performing scoring for each frame of the above image; A step of performing chunk scoring based on the above frame-by-frame scoring; A step of performing image curation of the chunk based on the chunk scoring above; A step of performing labeling on selected frames through the above image curation; and A step of reflecting the data of the labeled frame into the pose estimation AI model; A data reflection method including 2. In Paragraph 1, The step of performing scoring for each frame of the above video Step of extracting background images for each of the above frames; Based on the above background image, a step of generating a Foreground Mask for each frame; and A step of performing scoring for each frame based on the foreground mask and the result of the pose estimation; A data reflection method including 3. In Paragraph 2, The step of performing scoring for each frame based on the foreground mask and the result of the pose estimation above A data reflection method in which scoring is performed based on the degree of agreement between the above Foreground Mask and the result of the above pose estimation.

4. In Paragraph 1, The step of performing the above chunk scoring is A data reflection method performed based on the average score of frames included in the above chunk.

5. In paragraph 4, The step of performing image curation of the above chunk is A step of selecting labeling candidates based on the chunk scoring above, according to the scores of the frames included in the chunk; A step of additionally selecting the labeling candidates through the pose estimation AI model; Step of additionally sampling random frames from the above labeling candidates; and A step of removing similar frames based on the above labeling candidates; A data provision method including 6. In Paragraph 5, The step of additionally sampling random frames from the above labeling candidates is A data provision method that excludes the selected labeling candidates and selects a certain multiple of the selected labeling candidates by random sampling.

7. In Paragraph 1, The step of reflecting the data of the labeled frame into the pose estimation AI model is A step of performing fine-tuning of the pose estimation AI model using the data of the labeled frames; A data reflection method including 8. In Paragraph 7, The step of reflecting the data of the labeled frame into the pose estimation AI model is A data reflection method that processes the above video in real time through a streaming method, and analyzes and processes frame by frame rather than the entire video uploaded by the user.

9. In a server for reflecting user data in real time for a pose estimation AI model in behavioral experiments, The pose estimation AI model for extracting key points from an image transmitted from a terminal; An automated evaluation module for improving the accuracy of the pose estimation AI model; and A process for functionally controlling the pose estimation AI model and the automated evaluation module; comprising, The above process is A server that analyzes an input video through a terminal, estimates the pose of a target using the pose estimation AI model, performs scoring for each frame of the video through the automated evaluation module, performs chunk scoring based on the frame-by-frame scoring, performs image curation of the chunk based on the chunk scoring, performs labeling for selected frames through the image curation, and reflects the data of the labeled frames into the pose estimation AI model.

Images