Pruritic behavior detection and evaluation of cats or dogs for drug development and validation

Abstract

Devices, systems, and methods for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation are described. According to one embodiment, a method comprises receiving a series of image frames of the cat or the dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set of keypoints of the cat or the dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or the dog used for the detecting and evaluating the pruritic behavior. The method further comprises processing the series of image frames using an artificial intelligence (AI) workstream having at least one AI model.

Description

[0001] PRURITIC BEHAVIOR DETECTION AND EVALUATION OF CATS OR DOGS FOR DRUG DEVELOPMENT AND VALIDATION

[0002] FIELD

[0003]

[0001] The present disclosure is related to systems and methods for detecting and evaluating cats or dogs for drug development and validation, more specifically to systems and methods for detecting and evaluating cats or dogs for drug development and validation using Artificial Intelligence (Al) technologies.

[0004] BACKGROUND

[0005]

[0002] Clinical data is information which is used for clinical decision-making and research. The clinical data may include a target’s symptoms, disease codes, demographics, laboratory results, etc. Clinical data evaluation is a systematic process for collecting, analyzing, and assessing data to verify the safety and performance of treatment.

[0006]

[0003] For instance, for feline or canine dermatology drug development, a group of cats or dogs may be challenged with a drug to cause itchiness in them. Then, a subset of the group is treated with an experimental treatment, whereas the remainder of the group is given a placebo. For observation, evaluation and development of the treatment, the group of cats or dogs are videotaped for a period of time.

[0007]

[0004] Then, human researchers review the video tapes and evaluate the feline or canine behaviors, such as how many times each cat or dog being observed scratches in that video. Thus, in order to manually review the video tapes of the group of cats or dogs recorded over several weeks or months, it may take several weeks or months to review and evaluate the data which have been collected.SUMMARY

[0008]

[0005] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0009]

[0006] The present disclosure includes multiple embodiments for detecting and evaluating cats or dogs for drug development and validation using Artificial Intelligence (Al) technologies.

[0010]

[0007] According to one embodiment, a method for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation is performed by a computing device having one or more processors. The method comprises receiving a series of image frames of the cat or the dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set of keypoints of the cat or the dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or the dog used for the detecting and evaluating the pruritic behavior.

[0011]

[0008] The method additionally comprises processing the series of image frames using an Al workstream having one or more Al models for assigning corresponding confidence scores for visible keypoints of the cat or the dog in the bounding box in the each image frame, comparing an average confidence score across the visible keypoints of the cat or the dog in the bounding box in the each image frame with a minimum confidence score, evaluating the each image frame in the series of image frames of the cat or the dog obtained during the time interval to determine a pruritic classification of the cat or the dog based on a behavioral response of the cat or thedog detected at the visible keypoints when the average confidence score is greater than the minimum confidence score, and generating the pruritic classification of the cat or the dog based on the behavioral response of the cat or the dog.

[0012]

[0009] According to another embodiment, a non-transitory computer readable medium storing thereon program instructions, which, when executed by one or more processors of a computing device, configure one or more processors for performing a method for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation, the method comprising the steps discussed above.

[0013]

[0010] In yet another embodiment, a system for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation comprises one or more image capturing devices to capture images of the cat or the dog contained in a confinement space and a computing device comprising a memory storing program instructions and one or more processors coupled to the memory to read and execute the program instructions which configure the one or more processors to perform the method discussed above.

[0014]

[0011] In an aspect, the disclosure provides for a method for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation, wherein the method is performed by a computing device having at least one processor, the method comprising:

[0015] receiving a series of image frames of the cat or the dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set of keypoints of the cat or the dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or the dog used for the detecting and evaluating the pruritic behavior; andprocessing the series of image frames using an artificial intelligence (Al) workstream having at least one Al model for:

[0016] assigning corresponding confidence scores for visible keypoints of the cat or the dog in the bounding box in the each image frame;

[0017] comparing an average confidence score across the visible keypoints of the cat or the dog in the bounding box in the each image frame with a minimum confidence score;

[0018] evaluating the each image frame in the series of image frames of the cat or the dog obtained during the time interval to determine a pruritic classification of the cat or the dog based on a pruritic response of the cat or the dog detected at the visible keypoints when the average confidence score is greater than the minimum confidence score; and

[0019] generating the pruritic classification of the cat or the dog based on the pruritic response of the cat or the dog.

[0020]

[0012] In some embodiments, the method further comprises discarding the each image frame when the average confidence score is less than the minimum confidence score, wherein the each image frame is not included in the step of evaluating the each image frame over the time interval to determine the pruritic classification of the cat or the dog.

[0021]

[0013] In some embodiments, wherein the assigning the corresponding confidence scores comprises:

[0022] assigning a first confidence score for the bounding box; and

[0023] performing the assigning the corresponding confidence scores for the visible keypoints of the cat or the dog in the bounding box in the each image frame when thefirst confidence score for the bounding box is greater than a first minimum confidence score.

[0024]

[0014] In some embodiments, wherein the assigning the corresponding confidence scores comprises:

[0025] assigning a first confidence score for the bounding box; and

[0026] discarding the each image frame such that the each image frame is not included in the step of evaluating the each image frame over the time interval when the first confidence score for the bounding box is less than a first minimum confidence score.

[0027]

[0015] In some embodiments, wherein each one of the set of keypoints is defined based on image pixels in the bounding box.

[0028]

[0016] In some embodiments, wherein the average confidence score less than the minimum confidence score is obtained when a number of the visible keypoints wholly present in the each image frame is less than a minimum required number.

[0029]

[0017] In some embodiments, wherein the average confidence score less than the minimum confidence score is obtained when image pixels of the cat or the dog corresponding to the visible keypoints in the each image frame are blurry.

[0030]

[0018] In some embodiments, wherein assigning the average confidence score is obtained by dividing a sum of the corresponding confidence scores by a number of the keypoints in the set.

[0031]

[0019] In some embodiments, wherein the at least one Al model for the pruritic classification is trained by manually annotating training images obtained from cat or dog video footage and feeding the annotated training images into a neural network for training.

[0032]

[0020] In some embodiments, wherein the annotated training images exhibit at least one pruritic behavior comprising scratching with a paw, shaking, grooming, sledding,rubbing, rolling, licking, chewing, or biting, except licking, chewing or biting paws or tail.

[0033]

[0021] In some embodiments, wherein the series of image frames comprises a first set of image frames capturing the cat or the dog that is challenged and subsequently treated with a feline or canine veterinary drug or a second set of image frames capturing another cat or dog that is challenged and subsequently treated with a placebo.

[0034]

[0022] In some embodiments, the method further comprises analyzing efficacy or safety of the feline or canine veterinary drug with a pruritic classification score.

[0035]

[0023] In some embodiments, wherein the pruritic classification of the cat or the dog is determined as pruritic when the pruritic response of the cat or the dog is determined to be pruritic at least once during the time interval.

[0036]

[0024] In some embodiments, wherein the time interval is 60 seconds or less, 30 seconds or less, or 10 seconds or less.

[0037]

[0025] In some embodiments, wherein the processing the series of image frames comprises randomly selecting 65% to 85% of a total image frames in the series of image frames during the time interval prior to the assigning the corresponding confidence scores.

[0038]

[0026] In an aspect, the disclosure provides for a non-transitory computer readable medium storing thereon program instructions, which, when executed by at least one processor of a computing device, configure at least one processor for performing a method for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation, the method comprising:

[0039] receiving a series of image frames of the cat or the dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set ofkeypoints of the cat or the dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or the dog used for the detecting and evaluating the pruritic behavior; and

[0040] processing the series of image frames using an artificial intelligence (Al) workstream having at least one Al model for:

[0041] assigning corresponding confidence scores for visible keypoints of the cat or the dog in the bounding box in the each image frame;

[0042] comparing an average confidence score across the visible keypoints of the cat or the dog in the bounding box in the each image frame with a minimum confidence score;

[0043] evaluating the each image frame in the series of image frames of the cat or the dog obtained during the time interval to determine a pruritic classification of the cat or the dog based on a pruritic response of the cat or the dog detected at the visible keypoints when the average confidence score is greater than the minimum confidence score; and

[0044] generating the pruritic classification of the cat or the dog based on the pruritic response of the cat or the dog.

[0045]

[0027] In some embodiments, wherein the pruritic response of the cat or the dog detected at the visible keypoints are determined based on a movement of the visible keypoints in a way defined to be the pruritic response.

[0046]

[0028] In an aspect, the disclosure provides for a system for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation, the system comprising:

[0047] at least one image capturing device to capture images of the cat or the dog contained in a confinement space; anda computing device comprising a memory storing program instructions and at least one processor coupled to the memory to read and execute the program instructions which configure the at least one processor to perform a method comprising:

[0048] receiving a series of image frames of the cat or the dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set of keypoints of the cat or the dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or the dog used for the detecting and evaluating the pruritic behavior; and

[0049] processing the series of image frames using an artificial intelligence (Al) workstream having at least one Al model for:

[0050] assigning corresponding confidence scores for visible keypoints of the cat or the dog in the bounding box in the each image frame;

[0051] comparing an average confidence score across the visible keypoints of the cat or the dog in the bounding box in the each image frame with a minimum confidence score;

[0052] evaluating the each image frame in the series of image frames of the cat or the dog obtained during the time interval to determine a pruritic classification of the cat or the dog based on a pruritic response of the cat or the dog detected at the visible keypoints when the average confidence score is greater than the minimum confidence score; and

[0053] generating the pruritic classification of the cat or the dog based on the pruritic response of the cat or the dog.

[0054]

[0029] In some embodiments, wherein the assigning the corresponding confidence scores comprises:assigning a first confidence score for the bounding box; and

[0055] performing the assigning the corresponding confidence scores for the visible keypoints of the cat or the dog in the bounding box in the each image frame when the first confidence score for the bounding box is greater than a first minimum confidence score.

[0056]

[0030] In some embodiments, wherein the assigning the corresponding confidence scores comprises:

[0057] assigning a first confidence score for the bounding box; and

[0058] discarding the each image frame such that the each image frame is not included in the step of evaluating the each image frame over the time interval when the first confidence score for the bounding box is less than a first minimum confidence score.

[0059] BRIEF DESCRIPTION OF THE FIGURES

[0060]

[0031] Example embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:

[0061]

[0032] FIG. 1 is a process flow chart illustrating a method for detecting and evaluating a pruritic behavior of a cat or dog for drug development and validation, according to one embodiment of the present disclosure.

[0062]

[0033] FIG. 2 is a flow chart illustrating a method for selecting a valid set of image frames in detecting and evaluating the pruritic behavior of the cat or dog in FIG. 1, according to one embodiment of the present disclosure.

[0063]

[0034] FIG. 3A illustrates an exemplary view of assigning a confidence score to a bounding box of the dog in FIG. 2, according to one embodiment the present disclosure.

[0035] FIG. 3B illustrates an exemplary view of assigning a confidence score to a bounding box of the cat in FIG. 2, according to one embodiment the present disclosure.

[0064]

[0036] FIG. 4A illustrates an exemplary view of a set of keypoints assigned to the dog in FIG. 2, according to one embodiment of the present disclosure.

[0065]

[0037] FIG. 4B illustrates an exemplary view of a set of keypoints assigned to the cat in FIG. 2, according to one embodiment of the present disclosure.

[0066]

[0038] FIG. 5A illustrates an exemplary view of generating an average confidence score across visible keypoints of the dog in FIG. 2, according to one embodiment of the present disclosure.

[0067]

[0039] FIG. 5B illustrates an exemplary view of generating an average confidence score across visible keypoints of the cat in FIG. 2, according to one embodiment of the present disclosure.

[0068]

[0040] FIG. 6 illustrates an exemplary view of a pruritic classification of the dog in FIGS.

[0069] 1 through 5A, according to one embodiment of the present disclosure.

[0070]

[0041] FIG. 7 Illustrates a system for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation, according to one embodiment of the present disclosure.

[0071]

[0042] FIG. 8 is a diagrammatic system view of a data processing system in which any of the embodiments disclosed herein may be performed, according to one embodiment.

[0072]

[0043] Other features of the present embodiments will be apparent from the accompanying drawings and from the detailed description that follows.

[0073] DETAILED DESCRIPTION

[0044] Reference will now be made in detail to the preferred embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. While the disclosure will be described in conjunction with the preferred embodiments, it will be understood that they are not intended to limit the disclosure to these embodiments. On the contrary, the disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the disclosure as defined by the claims. Furthermore, in the detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be obvious to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure embodiments of the present disclosure.

[0074]

[0045] For clinical study of cats or dogs, it would take lots of time and resources because the researchers need to manually review the video tapes of the group of cats or dogs recorded over several weeks or months. Additionally, the detection and evaluation are often done by several human researchers, so sometimes, the result of the study may be mired by their subjectivity. Thus, there is a desire to automate a portion or more of the detection and evaluation process for clinical data so that the time to conduct the observation and evaluation is saved and the subjectivity of the observation is reduced and evaluation is enhanced.

[0075]

[0046] Accordingly, embodiments of the present disclosure propose using artificial intelligence (Al) technology to be exposed with a set of training data so that a trained Al model is enabled to screen a valid set of images from all the images collected as the clinical data. Then, the embodiments of the present disclosure propose to useanother Al model, which has been exposed with another set of training data, to classify the pruritic behaviors or responses of the cats or dogs.

[0076]

[0047] Through automating the detection and evaluation of cats or dogs for drug development and validation using Al technologies, the time to conduct the clinical study may be significantly reduced to several hours or days from the conventional duration of weeks or months. Additionally, the subjectivity of the detection and evaluation is reduced, thus increasing the repeatability of the results.

[0077]

[0048] FIG. 1 is a process flow chart illustrating a method for detecting and evaluating a pruritic behavior of a cat or dog for drug development and validation, according to one embodiment of the present disclosure. It is appreciated that the pruritic behavior refers to actions or responses that occur due to pruritus, which is the medical term for itching. It describes behaviors aimed at alleviating the discomfort caused by itching, such as scratching, rubbing, biting, or licking (commonly observed in both humans and animals). These actions are typically triggered by conditions such as skin irritations, allergies, infections, or systemic diseases.

[0078]

[0049] In operation 104, a series of image frames of a cat or dog obtained during a time interval is received. It is appreciated that the series of image frames may be a part or whole of clinical data or historical clinical data of the cat or dog. The series of image frames may comprise a first set of image frames capturing the cat or dog that is challenged (e g., cause an itchiness using a drug) and subsequently treated with a feline or canine veterinary drug on development and / or a second set of image frames capturing another cat or dog that is challenged and subsequently treated with a placebo. In one embodiment, the time interval is 60 seconds or less, 30 seconds or less, or 10 seconds or less. In one embodiment, the time interval is 3 seconds.

[0050] In one embodiment, each image frame in the series of image frames comprises a set of keypoints of the cat or dog in a bounding box, and the set of keypoints corresponds to anatomical features of the cat or dog used for detecting and evaluating the pruritic behavior. In one embodiment, the pruritic response of the cat or dog detected at visible keypoints are determined based on a movement of the visible keypoints in a way defined to be the pruritic response. For example, a single instance or a combination of instances of the dog scratching with a paw, shaking, grooming, sledding, rubbing, rolling, licking, chewing, or biting may be regarded as a pruritic behavior. However, licking, chewing or biting paws or tail may not be regarded as a pruritic behavior since these behaviors may belong to grooming behaviors. In another example, a single instance or a combination of instances of the cat licking, shaking, grooming, or scratching with a paw may be regarded as a pruritic behavior.

[0079]

[0051] In operation 106 through 114, the series of image frames are processed using an Al workstream having one or more Al models. It is appreciated that the Al workstream may be a process that uses Al-powered tools to streamline tasks and activities in an organization.

[0080]

[0052] In one embodiment, 65% to 85% of total image frames in the series of image frames are randomly selected for the time interval. For example, for processing of 75 image frames in 3 seconds, only 48 image frames may be randomly selected for operation in 106. In another example, 50 frames instead of 75 frames may be selected for the time interval, wherein 48 frames may be randomly selected for operation in 106.

[0081]

[0053] In one embodiment, the Al workstream comprises a classification layer having a screen Al model that is trained to determine whether to include the image frame in pruritic classification of the cat or dog, wherein the screen Al model is to evaluate at least the bounding box and the set of keypoints of the cat or dog and generate a binaryclassification whether to include or not include the image frame in the pruritic classification of the cat or dog. In one embodiment, the screen Al model or the screen algorithm may ignore those cats or dogs which are not part of the clinical study. For example, the screen Al model may be trained to ignore caged cats or dogs while tracking those cats or dogs not caged. In one embodiment, the screen Al model may label a caged dog as such, whereas a dog not caged may be labeled as “canine.” Because of such training, the screen Al model may be able to track a cat or dog being tracked even when the bounding box of the cat or dog is overlapped with the bounding box of a caged cat or dog. Although the caged dog and non-caged dog are distinguished as the non-study animal and the study animal, respectively, in this embodiment, the screen Al model may be trained such that a different feature may be used to distinguish between the study animal and the non-study animal.

[0082]

[0054] In another embodiment, the Al workstream comprises a classification layer having a time series classifier Al model that is trained to determine the pruritic classification of the cat or dog, wherein the time series classifier Al model is to evaluate a valid set of image frames selected from the series of image frames using the screen Al model and generate a binary classification, i.e., pruritic (e.g., T) ornon-pruritic(e.g., ‘0’), of the behavior or response of the cat or dog.

[0083]

[0055] In one embodiment, one or more machine learning models may be trained to process the captured images or videos, i.e., the series of image frames, and to develop into the screen Al model and the time series classifier Al model. For example, for a machine learning model, which is used to process captured or received image frames to detect a dynamic (e.g., in motion relative to its environment and / or relative to previous instances of itself) or static (e.g., standstill) object and define a bounding box for the cat or dog, the set of keypoints and / or a bounding box for each one of thekeypoints, the training data may comprise training image frames containing dynamic or static objects (e.g., of bounding boxes, keypoints, etc.) similar to those the screen Al model would process in practice and process. Some of these training images are used for validation and also include associated annotations based on the correct detection and processing of the dynamic or static object, such as the locations and / or visibility of the bounding boxes and / or keypoints.

[0084]

[0056] In one example, for another machine learning model, which is used to process a valid set of image frames selected by the screen Al model to determine the pruritic classification of the cat or dog being observed, the training data may comprise training images containing dynamic or static objects similar to those the time series classifier Al model would process in practice. Some of these training images are used for validation and also include associated annotations based on the correct classification of the behavior or response of the cat or dog. In one embodiment, the time series classifier Al model is trained by manually annotating training images obtained from cat or dog video footage and feeding the annotated training images into a neural network of the machine learning model for training. When the time series classifier Al model is run inclusively. For example, image frames 1 to 75 is run first, followed by image frames 25 to 100, and then image frame 50 to 125 to cover the continuous behavior or motion of the cat or dog being observed.

[0085]

[0057] Particularly, for each frame in the series of image frames, operations 106 through 110 are performed to sort out a valid set of image frames used for pruritic classification of the cat or dog. Thus, in operation 106, corresponding confidence scores are assigned for visible keypoints of the cat or dog in the bounding box in each image frame. In operation 108, an average confidence score across the visible keypoints of the cat or dog in the bounding box is compared with a minimumconfidence score. In operation 110, those image frames with their respective average confidence scores greater than the minimum or threshold confidence score are kept or included in the valid set of image frames prepared for the pruritic classification of the cat or dog. In one embodiment, those image frames with their average confidence scores less than the minimum or threshold confidence score are discarded or not included in the pruritic classification of the cat or dog. For example, if for a particular image frame, if it is not included for the pruritic classification, the x, y locations of the keypoints and / or bounding boxes may be set with zero, thereby setting the confidence scores of the keypoints and / or bounding boxes to zero.

[0086]

[0058] In operation 112, the valid set of image frames are evaluated to determine the pruritic classification of the cat or dog based on a pruritic response of the cat or dog detected at the visible keypoints. In one embodiment, the pruritic classification of the cat or dog is determined as pruritic when the pruritic response of the cat or dog is determined to be pruritic at least once during the time interval (e.g., 3 second, 1 minute, etc.). It is appreciated that a machine learning model, which is to become the time series classifier Al model, is trained to classify a behavior or response of the cat or dog after the cat or dog is challenged with a drug which causes itchiness, numerous video images of pruritic or non-pruritic behaviors or responses of the cat or dog are fed for pruritic classification, and the classification results of the machine learning model are manually annotated (e.g., second-by-second annotations) to correct the prediction made by the machine learning model.

[0087]

[0059] In one embodiment, the efficacy or safety of a feline or canine veterinary drug is analyzed when the drug is administered to the cat or dog after it has been challenged and the pruritic classification score of the cat or dog after the treatment is evaluated.

[0060] FIG. 2 is a flow chart illustrating a mathod for selecting a valid set of image frames in detecting and evaluating the pruritic behavior of the cat or dog in FIG. 1, according to one embodiment of the present disclosure. FIG. 3A illustrates an exemplary view of assigning a confidence score to a bounding box of the dog in FIG.

[0088] 2, according to one embodiment the present disclosure. FIG. 3B illustrates an exemplary view of assigning a confidence score to a bounding box of the cat in FIG.

[0089] 2, according to one embodiment the present disclosure. FIG. 4A illustrates an exemplary view of a set of keypoints assigned to the dog in FIG. 2, according to one embodiment of the present disclosure. FIG. 4B illustrates an exemplary view of a set of keypoints assigned to the cat in FIG. 2, according to one embodiment of the present disclosure. FIG. 5A illustrates an exemplary view of generating an average confidence score across visible keypoints of the dog in FIG. 2, according to one embodiment of the present disclosure. FIG. 5B illustrates an exemplary view of generating an average confidence score across visible keypoints of the cat in FIG. 2, according to one embodiment of the present disclosure.

[0090]

[0061] In operation 202, a first confidence score 306 is assigned for a bounding box 304 in an image frame for detection of a cat 308 or dog 302, as illustrated in FIGS. 3A and 3B. In this operation, only the cat 308 or dog 302 within the bounding box 304, not the background in the bounding box 304, is processed for the assignment of the first confidence score 306. For example, in FIG. 3A, the first confidence score 306 of 83% is assigned for the bounding box 304 surrounding the dog 302, and the first confidence score 306 of 84% is assigned for the bounding box 304 surrounding the cat 308. It is appreciated that the dog 302 or the cat 308 may be detected by an Al model, such as the screen Al model, in the current image frame being processed and labeled as “canine” or “feline,” respectively, and the bounding box 304 may be definedto indicate the location of the dog 302 or the cat 308 in the image frame. In one embodiment, the first confidence score 306 may be affected by the degree within which the dog 302 or the cat 308 is captured in the image frame or the blurriness of the dog 302 or the cat 308 captured in the image frame.

[0091]

[0062] In operation 204, the first confidence score 306 is compared with a first minimum or threshold confidence score (e.g., 80%). When the first confidence score 306 is less than the first minimum confidence score, the image frame is discarded in operation 208 so that it is not included in the pruritic classification of the dog 302 or the cat 308. In case the first confidence score 306 is greater than the first minimum confidence score, for visible keypoints of the dog 302 or the cat 308, respective confidence scores are assigned in operation 206. For example, in FIG. 4A, eleven (11 ) confidence scores may be assigned for their corresponding eleven (11 ) keypoints of the dog 302, wherein the eleven keypoints may comprise head 404, left ear 406, right ear 408, back start 410, front left paw 412, front right paw 414, back middle 416, back end 418, back left paw 420, back right paw 422, and tail 424. In the same manner, in FIG. 4B, eleven (11) confidence scores may be assigned for their corresponding eleven (11 ) keypoints of the cat 308, wherein the eleven keypoints may comprise head 454, left ear 456, right ear 458, back start 460, front left paw 462, front right paw 464, back middle 466, back end 468, back left paw 470, back right paw 472, and tail 474.

[0092]

[0063] It is appreciated that by using the keypoints (e.g., the visible keypoints) and the line segments connecting the keypoints, the shape and movement of the anatomical features of the dog 302 or the cat 308 (e.g., the head, paw, etc.) may be used to determine the pruritic behavior or response of the dog 302 or the cat 308. It is further appreciated that a machine learning model, which may be trained to become the time series classifier Al model, is trained with a numerous images of the dog 302 or thecat 308 with various shapes and movements of the keypoints and the line segments representing the target anatomical features as well as manual annotations (e.g., data specifying the bounding box 304, keypoints, pruritic behaviors or response, correctness of the pruritic classification, etc.), so that the time series classifier Al model becomes a fast, accurate tool to evaluate the pruritic behavior or response of the dog 302 or the cat 308. In one embodiment, each keypoint is defined based on image pixels in the bounding box 304.

[0093]

[0064] Once the confidence scores for the visible keypoints of the dog 302 or the cat 308 are assigned in operation 206, an average confidence score 502 is generated, as illustrated in FIG. 5A or FIG. 5B. In one embodiment, the average confidence score 502 is calculated by dividing the sum of the confidence scores across the visible keypoints with the number of the visible keypoints. In operation 210, the average confidence score 502 is compared with a second minimum or threshold confidence score (e.g., 30%). In one embodiment, the average confidence score 502 less than the second minimum confidence score is obtained when the number of the visible keypoints wholly present in the image frame is less than a minimum required number. In one embodiment, the average confidence score 502 less than the second minimum confidence score is obtained when the clarity of image pixels of the dog 302 or the cat 308 corresponding to the visible keypoints in the image frame is less than a threshold value.

[0094]

[0065] When the average confidence score 502 is less than the second minimum confidence score, the image frame is discarded in operation 208 so that it is not included in the pruritic classification of the dog 302 or the cat 308. In case the average confidence score 502 (e.g., 37% in FIG. 5A or 38% in FIG. 5B) is greater than the second minimum confidence score (e.g., 30%), for visible keypoints of the dog 302 orthe cat 308, as illustrated in FIG. 5A or FIG. 5B, the image frame is kept or included for pruritic classification of the dog 302 or the cat 308 in operation 212. It is appreciated that by discarding those image frames with their first confidence score less than the first minimum confidence score and their respective average confidence score less than the second minimum confidence score and not including the image frames in the evaluation of the pruritic classification of the dog or cat, the false positives (i.e., calling a non-pruritic response as a pruritic response) or false negatives (i.e. , calling a pruritic response as a non-pruritic response) in the Al prediction of pruritic response of the dog or cat may be reduced.

[0095]

[0066] FIG. 6 illustrates an exemplary view of a pruritic classification of the dog 302 in FIGS. 1 through 5A, according to one embodiment of the present disclosure.

[0096]

[0067] In one embodiment, once the evaluation of the valid set of image frames is done in operation 112 of FIG. 1 , the pruritic classification of the dog 302 is generated based on the pruritic response of the dog 302. For example, FIG. 6 illustrates the pruritic classification results from analyzing twenty minutes of video, where each minute of video contains twenty (20) sequential three (3) second intervals of image frames.

[0097]

[0068] Accordingly, item 1 illustrates study number ‘DS33453’ in ‘phase_1,’ where a one-minute long tape of the dog 302 with animal ID T in room ’10.31’ has been evaluated to classify the pruritic behavior or response of the dog 302. The ‘scratching’ behavior with confidence score of T indicates that the dog 302 is exhibiting pruritic behavior, wherein the confidence score in this case represents the confidence of the prediction for pruritic behavior or response of the dog 302. On the other hand, item 4 illustrates study number ‘DS33453’ in ‘phase_1 ,’ where a one-minute long tape of the dog 302 with animal ID T in room ’10.3T has been evaluated to classify the pruriticbehavior or response of the dog 302. The ‘non-scratching’ behavior with confidence score of T indicates that the dog 302 is exhibiting a non-pruritic behavior.

[0098]

[0069] In one embodiment, the confidence score for the pruritic classification is compared with a third minimum or threshold score (e.g., 90%). When the confidence score is less than the third minimum score, the pruritic classification of the dog 302 may be withheld. It is appreciated that for two-hour period, if the dog 302 exhibits pruritic behaviors 50 out of 120 m inutes, the score of 50 may be reported as the pruritic score of the dog 302, wherein 35 pruritic minutes out of 120 minutes may be considered as the threshold which indicates that the dog 302 may be acceptable to use in the study. Then, the treatment in development may be administered to the dog 302 to evaluate the efficacy of the treatment. For the clinical data being collected, such as a number of files storing series of image frames over the time interval, time stamps such as shown in ‘1_minute_window” in FIG. 6 may be used to track the data and evaluate the study in context. It is appreciated that a pruritic classification of the cat 308 in FIGS. 1 through 5B may be obtained in a similar manner as the example of the dog 302, as illustrated in FIG. 6.

[0099]

[0070] FIG. 7 Illustrates a system for detecting and evaluating a pruritic behavior of a cat or dog for drug development and validation, according to one embodiment of the present disclosure.

[0100]

[0071] In FIG. 7, the system comprises one or more image capturing devices 750 to capture images of the cat or dog contained in a confinement space. It is appreciated that the capturing devices 750 may comprise one or more analog and digital video cameras, digital still cameras, image-capture adapters, video frame grabbers, document scanners, x-ray scanners, phosphor plate readers, etc. It is further appreciated that the images captured by the image capturing device may be clinicaldata (e.g., clinical data of cats or dogs for evaluating the effectiveness of new treatment for pruritus) collected ever a period of time (e.g., days, weeks or months) and stored in a storage device (not shown in the figure). The system further comprises a computing device 700 comprising a memory 704 storing program instructions and one or more processors 702 coupled to the memory 704 to read and execute a set of program instructions 724 which configure the one or more processors 702 to perform a method for detecting and evaluating a pruritic behavior of the cat or dog for drug development and validation, as discussed in FIG. 1 through 6.

[0101]

[0072] In one embodiment, the method comprises receiving a series of image frames of the cat or dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set of keypoints of the cat or dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or dog used for detecting and evaluating the pruritic behavior. The method further comprises processing the series of image frames using an artificial intelligence (Al) workstream having one or more Al models.

[0102]

[0073] In one embodiment, the one or more Al models are trained for assigning corresponding confidence scores for visible keypoints of the cat or dog in the bounding box in each image frame, comparing an average confidence score across the visible keypoints of the cat or dog in the bounding box in each image frame with a minimum confidence score, evaluating each image frame in the series of image frames of the cat or dog obtained during the time interval to determine a pruritic classification of the cat or dog based on a pruritic response of the cat or dog detected at the visible keypoints when the average confidence score is greater than the minimum the dog based on the pruritic response of the cat or dog.

[0074] FIG. 8 is a diagrammatic system view 800 of the computing device 700 of FIG.

[0103] 7 in which any of the embodiments disclosed herein may be performed, according to one embodiment. Particularly, the system view 800 of FIG. 8 illustrates a processor 802 (e.g., the processor 702) , a main memory 804 (e.g., the memory 704), a static memory 806 (e.g., the memory 704), a bus 808, a video display 810, an alpha-numeric input device 812, a cursor control device 814, a drive unit 816, a signal generation device 818, a network interface device 820, a machine readable medium 822, instructions 824 (e.g., the instructions 724), and a network 826, according to one embodiment.

[0104]

[0075] The diagrammatic system view 800 may indicate a personal computer and / or the computing device 700 in which one or more operations disclosed herein are performed. The processor 802 may be microprocessor, a state machine, an application specific integrated circuit, a field programmable gate array, etc. The main memory 804 may be a dynamic random access memory and / or a primary memory of a computer system.

[0105]

[0076] The static memory 806 may be a hard drive, a flash drive, and / or other memory information associated with the computing device 700. The bus 808 may be an interconnection between various circuits and / or structures of the computing device 700. The video display 810 may provide graphical representation of information on the computing device 700. The alpha-numeric input device 812 may be a keypad, keyboard and / or any other input device of text (e.g., a special device to aid the physically handicapped). The cursor control device814 may be a pointing device such as a mouse.

[0106]

[0077] The drive unit 816 may be a hard drive, a storage system, and / or other longer term storage subsystem. The signal generation device 818 may be a bios and / or afunctional operating system of the computing device 700. The machine readable medium 822 may provide instructions on which any of the methods disclosed herein may be performed. The instructions 824 may provide source code and / or data code to the processor 802 to enable any one / or more operations disclosed herein.

[0107]

[0078] The present disclosure includes multiple embodiments for detecting and evaluating cats or dogs for drug development and validation using Artificial Intelligence (Al) technologies. The present disclosure centers on two Al models, i.e. , the screen Al model and the time series classifier Al model, to screen a valid set of images from all the images collected as the clinical data and to classify the pruritic behaviors or responses of the cats or dogs, respectively.

[0108]

[0079] As a result, the time to conduct the clinical study is significantly reduced compared to the conventional clinical study. Additionally, the accuracy of the detection and evaluation increases significantly since the human subjectivity is removed from the study driven by the Al.

[0109]

[0080] It is to be understood that this disclosure is not limited to particular embodiments or embodiments described, as such may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present disclosure will be limited only by the appended claims.

[0110]

[0081] Where a range of values is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit unless the context clearly dictates otherwise, between the upper and lower limit of that range and any other stated or intervening value in that stated range, is encompassed within the disclosure. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges and are also encompassed within the disclosure, subject to anyspecifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the disclosure.

[0111]

[0082] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present disclosure, representative illustrative methods and materials are now described.

[0112]

[0083] All publications and patents cited in this specification are herein incorporated by reference as if each individual publication or patent were specifically and individually indicated to be incorporated by reference and are incorporated herein by reference to disclose and describe the methods and / or materials in connection with which the publications are cited. The citation of any publication is for its disclosure prior to the filing date and should not be construed as an admission that the present disclosure is not entitled to antedate such publication by virtue of prior disclosure. Further, the dates of publication provided may be different from the actual publication dates which may need to be independently confirmed.

[0113]

[0084] It is noted that, as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation.

[0114]

[0085] As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discretecomponents and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present disclosure. Any recited method can be carried out in the order of events recited or in any other order which is logically possible.

[0115]

[0086] Although the foregoing disclosure has been described in some detail by way of illustration and example for purposes of clarity of understanding, it is readily apparent to those of ordinary skill in the art in light of the teachings of this disclosure that certain changes and modifications may be made thereto without departing from the spirit or scope of the appended claims.

[0116]

[0087] Accordingly, the preceding merely illustrates the principles of the disclosure. It will be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its spirit and scope. Furthermore, all examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventors to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, embodiments, and embodiments of the disclosure as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, i.e. , any elements developed that perform the same function, regardless of structure. The scope of the present disclosure, therefore, is not intended to be limited to the exemplary embodiments shown and described herein. Rather, the scope and spirit of present disclosure is embodied by the appended claims.

Claims

1. What is claimed is:

1. A method for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation, wherein the method is performed by a computing device having at least one processor, the method comprising:receiving a series of image frames of the cat or the dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set of keypoints of the cat or the dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or the dog used for the detecting and evaluating the pruritic behavior; andprocessing the series of image frames using an artificial intelligence (Al) workstream having at least one Al model for:assigning corresponding confidence scores for visible keypoints of the cat or the dog in the bounding box in the each image frame;comparing an average confidence score across the visible keypoints of the cat or the dog in the bounding box in the each image frame with a minimum confidence score;evaluating the each image frame in the series of image frames of the cat or the dog obtained during the time interval to determine a pruritic classification of the cat or the dog based on a pruritic response of the cat or the dog detected at the visible keypoints when the average confidence score is greater than the minimum confidence score; andgenerating the pruritic classification of the cat or the dog based on the pruritic response of the cat or the dog.

2. The method of claim 1, further comprising discarding the each image frame when the average confidence score is less than the minimum confidence score, wherein the each image frame is not included in the step of evaluating the each image frame over the time interval to determine the pruritic classification of the cat or the dog.

3. The method of claim 1, wherein the assigning the corresponding confidence scores comprises:assigning a first confidence score for the bounding box; andperforming the assigning the corresponding confidence scores for the visible keypoints of the cat or the dog in the bounding box in the each image frame when the first confidence score for the bounding box is greater than a first minimum confidence score.

4. The method of claim 1, wherein the assigning the corresponding confidence scores comprises:assigning a first confidence score for the bounding box; anddiscarding the each image frame such that the each image frame is not included in the step of evaluating the each image frame over the time interval when the first confidence score for the bounding box is less than a first minimum confidence score.

5. The method of claim 1, wherein each one of the set of keypoints is defined based on image pixels in the bounding box.

6. The method of claim 1, wherein the average confidence score less than the minimum confidence score is obtained when a number of the visible keypoints wholly present in the each image frame is less than a minimum required number.

7. The method of claim 1, wherein the average confidence score less than the minimum confidence score is obtained when image pixels of the cat or the dog corresponding to the visible keypoints in the each image frame are blurry.

8. The method of claim 1 , wherein assigning the average confidence score is obtained by dividing a sum of the corresponding confidence scores by a number of the keypoints in the set.

9. The method of claim 1, wherein the at least one Al model for the pruritic classification is trained by manually annotating training images obtained from cat or dog video footage and feeding the annotated training images into a neural network for training.

10. The method of claim 9, wherein the annotated training images exhibit at least one pruritic behavior comprising scratching with a paw, shaking, grooming, sledding, rubbing, rolling, licking, chewing, or biting, except licking, chewing or biting paws or tail.

11. A non-transitory computer readable medium storing thereon program instructions, which, when executed by at least one processor of a computing device, configure at least one processor for performing a method for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation, the method comprising:receiving a series of image frames of the cat or the dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set of keypoints of the cat or the dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or the dog used for the detecting and evaluating the pruritic behavior; andprocessing the series of image frames using an artificial intelligence (Al) workstream having at least one Al model for:assigning corresponding confidence scores for visible keypoints of the cat or the dog in the bounding box in the each image frame;comparing an average confidence score across the visible keypoints of the cat or the dog in the bounding box in the each image frame with a minimum confidence score;evaluating the each image frame in the series of image frames of the cat or the dog obtained during the time interval to determine a pruritic classification of the cat or the dog based on a pruritic response of the cat or the dog detected at the visible keypoints when the average confidence score is greater than the minimum confidence score; andgenerating the pruritic classification of the cat or the dog based on the pruritic response of the cat or the dog.

12. The non-transitory computer readable medium of claim 11 , wherein the pruritic response of the cat or the dog detected at the visible keypoints are determined based on a movement of the visible keypoints in a way defined to be the pruritic response.

13. A system for detecting and evaluating a pruritic behavior of a cat or a dog for drug development and validation, the system comprising:at least one image capturing device to capture images of the cat or the dog contained in a confinement space; anda computing device comprising a memory storing program instructions and at least one processor coupled to the memory to read and execute the programinstructions which configure the at least one processor to perform a method comprising:receiving a series of image frames of the cat or the dog obtained during a time interval, wherein each image frame in the series of image frames comprises a set of keypoints of the cat or the dog in a bounding box, and wherein the set of keypoints corresponds to anatomical features of the cat or the dog used for the detecting and evaluating the pruritic behavior; andprocessing the series of image frames using an artificial intelligence (Al) workstream having at least one Al model for:assigning corresponding confidence scores for visible keypoints of the cat or the dog in the bounding box in the each image frame;comparing an average confidence score across the visible keypoints of the cat or the dog in the bounding box in the each image frame with a minimum confidence score;evaluating the each image frame in the series of image frames of the cat or the dog obtained during the time interval to determine a pruritic classification of the cat or the dog based on a pruritic response of the cat or the dog detected at the visible keypoints when the average confidence score is greater than the minimum confidence score; andgenerating the pruritic classification of the cat or the dog based on the pruritic response of the cat or the dog.

14. The system of claim 13, wherein the assigning the corresponding confidence scores comprises:assigning a first confidence score for the bounding box; andperforming the assigning the corresponding confidence scores for the visible keypoints of the cat or the dog in the bounding box in the each image frame when the first confidence score for the bounding box is greater than a first minimum confidence score.

15. The system of claim 13, wherein the assigning the corresponding confidence scores comprises:assigning a first confidence score for the bounding box; anddiscarding the each image frame such that the each image frame is not included in the step of evaluating the each image frame over the time interval when the first confidence score for the bounding box is less than a first minimum confidence score.

Images