Behavior analysis method and apparatus, and electronic device and storage medium
By combining multimodal large models and large language models, end-to-end behavior analysis was achieved, solving the problems of low accuracy and low efficiency in existing technologies and improving the accuracy and efficiency of animal behavior analysis.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
- Filing Date
- 2024-11-30
- Publication Date
- 2026-06-04
Smart Images
Figure CN2024135989_04062026_PF_FP_ABST
Abstract
Description
Behavioral analysis methods, devices, electronic equipment and storage media Technical Field
[0001] This application relates to the field of behavior analysis technology, and more specifically, to a behavior analysis method, apparatus, electronic device, and storage medium. Background Technology
[0002] In recent years, machine learning-based methods for quantitative analysis of animal behavior have developed rapidly. By acquiring animal videos, these methods can perform refined behavioral analysis on animals, resulting in a series of scientific research and industrialization achievements.
[0003] However, existing animal behavior analysis methods primarily extract body point features of interest from animal videos and perform behavior analysis using methods including numerical computation, behavioral machine learning classification, and clustering. Among these, the accuracy of existing methods, whether using behavioral machine learning classification or clustering for behavior analysis, is limited. Furthermore, existing methods focus on single, small-scale datasets, failing to extract general behavioral features, and suffer from difficulties in experimental reproducibility. Their multi-stage processing is also time-consuming, resulting in low analytical efficiency.
[0004] As can be seen from the above, the problem of low efficiency in behavior analysis in existing technologies still needs to be solved. Summary of the Invention
[0005] This application provides a behavior analysis method, apparatus, electronic device, and storage medium, which can solve the problem of low accuracy in behavior analysis in related technologies. The technical solutions are as follows:
[0006] According to one aspect of this application, a behavior analysis method includes:
[0007] Obtain multimodal data collected for the target to be detected, and perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a large multimodal model;
[0008] Behavioral analysis is performed on the multimodal data based on a multimodal large model to obtain a behavioral text description of the target to be detected;
[0009] Statistical analysis of the behavioral text description is performed based on a large language model to obtain behavioral analysis results for the target to be detected.
[0010] According to one aspect of this application, a behavior analysis device includes:
[0011] The data acquisition module is used to acquire multimodal data collected for the target to be detected, and to perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a large multimodal model;
[0012] The behavior analysis module is used to perform behavior analysis on the multimodal data based on a multimodal large model to obtain a textual description of the behavior of the target to be detected.
[0013] The statistical analysis module is used to perform statistical analysis on the behavioral text description based on a large language model to obtain behavioral analysis results for the target to be detected.
[0014] In one exemplary embodiment, the data acquisition module includes:
[0015] Data set unit, used to construct a behavioral semantic dataset based on the multimodal data;
[0016] The training unit is used to train the pre-trained model based on the behavioral semantic dataset to obtain a multimodal large model.
[0017] In one exemplary embodiment, the dataset unit includes:
[0018] Clustering subunits are used to perform clustering analysis on the multimodal data to divide the multimodal data into at least two short-time data segments;
[0019] The behavioral semantics subunit is used to set corresponding behavioral semantics for the short-term data segments and generate a behavioral semantics dataset.
[0020] In one exemplary embodiment, the training unit includes:
[0021] The behavior analysis subunit is used to input the behavior semantic dataset into the pre-trained model for behavior analysis, and to obtain behavioral text descriptions for each short-time data segment in the behavior semantic dataset.
[0022] The evaluation subunit is used to evaluate the accuracy of the behavioral text description based on the behavioral semantics in the semantic dataset, and obtain the evaluation result indicating the accuracy of the behavioral description of the pre-trained model for the behavioral semantics of short-time data segments.
[0023] The parameter tuning subunit is used to tune the parameters of the pre-trained model based on the evaluation results and repeat the training until the behavior description accuracy of the behavior analysis of the pre-trained model reaches a preset threshold, thereby obtaining a multimodal large model.
[0024] In one exemplary embodiment, the behavior analysis module includes:
[0025] The problem information unit is used to generate corresponding problem information based on the multimodal data;
[0026] The behavioral text description unit is used to generate a behavioral text description in a preset format corresponding to the multimodal data based on the problem information.
[0027] In one exemplary embodiment, the behavior text description unit includes:
[0028] The first text description subunit is used to generate corresponding behavioral text descriptions for each short-time data segment in the multimodal data;
[0029] The second text description subunit is used to generate a corresponding multimodal data behavior text description for the behavior sequence of each short-time data segment.
[0030] In one exemplary embodiment, the behavioral text description includes behavioral semantic text and motion parameter text;
[0031] The first text description subunit includes:
[0032] Body point sub-unit, used to determine the three-dimensional body point data corresponding to the target to be detected in the short-time data segment;
[0033] The motion parameter subunit is used to predict motion parameters based on the three-dimensional body point data and generate motion parameter text.
[0034] The semantic prediction subunit is used to perform behavioral semantic prediction on the short-time data fragment and generate behavioral semantic text.
[0035] According to one aspect of this application, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to perform the behavior analysis method as described above.
[0036] According to one aspect of this application, a storage medium stores computer-readable instructions thereon, which are executed by one or more processors to implement the behavior analysis method described above.
[0037] The beneficial effects of the technical solution provided in this application are:
[0038] In the above technical solution, this application trains a large multimodal model using multimodal data, generates behavioral text descriptions using the large multimodal model, and then generates analysis results using a large language model. The method of fine-tuning the pre-trained model using multimodal data allows for learning from large-scale multimodal data and enables the large multimodal model to better utilize this data. This achieves the extraction and analysis of general behavioral features, improves the accuracy of behavior classification, recognition, and description, and yields a large multimodal model with high accuracy in behavior description, effectively improving the efficiency of behavior analysis. By combining the large multimodal model and the large language model, behavior analysis is performed on multimodal data in an end-to-end manner, improving the efficiency of behavior analysis and effectively addressing the problem of insufficient behavior analysis capabilities in related technologies. Attached Figure Description
[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 is a schematic diagram of the implementation environment according to this application;
[0041] Figure 2 is a flowchart illustrating a behavior analysis method according to an exemplary embodiment;
[0042] Figure 3 is a flowchart of step 210 in one embodiment of the embodiment corresponding to Figure 2;
[0043] Figure 4 is a flowchart of step 211 in one embodiment of the embodiment corresponding to Figure 3;
[0044] Figure 5 is a flowchart of step 213 in one embodiment of the embodiment corresponding to Figure 3;
[0045] Figure 6 is a flowchart of step 230 in one embodiment of the embodiment corresponding to Figure 2;
[0046] Figure 7 is a flowchart of step 233 in one embodiment of the embodiment corresponding to Figure 6;
[0047] Figure 8 is a flowchart of step 2331 in the embodiment corresponding to Figure 7 in one embodiment;
[0048] Figure 9 is a flowchart illustrating the specific implementation of a behavior analysis method in an application scenario.
[0049] Figure 10 is a structural block diagram of a behavior analysis device according to an exemplary embodiment;
[0050] Figure 11 is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation
[0051] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.
[0052] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.
[0053] As mentioned earlier, existing animal behavior analysis methods primarily extract body point features of interest from animal videos and perform behavior analysis using methods including numerical computation, behavioral machine learning classification, and clustering. However, the accuracy of existing methods, whether through behavioral machine learning classification or clustering, is limited. Furthermore, existing methods focus on small-scale, individual datasets, failing to extract general behavioral features, and suffer from difficulties in experimental reproducibility. Their multi-stage processing is time-consuming, resulting in low analytical efficiency.
[0054] Therefore, the behavior analysis method provided in this application can effectively improve the efficiency of behavior analysis. Accordingly, the behavior analysis method is applicable to behavior analysis devices, which can be deployed on electronic devices. The electronic devices can be computer devices configured with the von Neumann architecture, such as desktop computers, laptops, servers, etc.; the electronic devices can also be electronic devices with central control functions, such as gateways; the electronic devices can also refer to portable mobile electronic devices, such as smartphones, tablets, etc.
[0055] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0056] Figure 1 is a schematic diagram of an implementation environment involved in a behavior analysis method. It should be noted that this implementation environment is merely an example adapted to the present invention and should not be considered as providing any limitation on the scope of the invention.
[0057] The implementation environment includes a data acquisition terminal 110 and a server terminal 130.
[0058] Specifically, the acquisition terminal 110 can be considered a data acquisition device, including but not limited to electronic devices with information acquisition functions such as recorders and cameras, or it can be considered a device that integrates information acquisition and behavior analysis, including but not limited to electronic devices such as laptops and desktop computers that have both information acquisition and behavior analysis functions. That is to say, in some embodiments, the acquisition terminal 110 can acquire multimodal data.
[0059] Server 130 can also be considered a behavior analysis device, including but not limited to electronic devices with behavior analysis capabilities such as smartphones, desktop computers, laptops, and servers. It can also be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide backend services, such as behavior analysis services, etc.
[0060] The server 130 and the acquisition terminal 110 establish a network communication connection in advance via wired or wireless means, and data transmission between the server 130 and the acquisition terminal 110 is realized through this network communication connection. The transmitted data includes, but is not limited to, multimodal data.
[0061] In one application scenario, through the interaction between the data acquisition terminal 110 and the server terminal 130, the data acquisition terminal 110 collects multimodal data and uploads the multimodal data to the server terminal 130 to request the server terminal 130 to provide behavior analysis services.
[0062] At this point, for server 130, after receiving the multimodal data uploaded by acquisition terminal 110, it calls the behavior analysis service, trains a multimodal large model based on the multimodal data, generates a behavioral text description of the target to be detected, and then generates the behavior analysis result of the target to be detected based on the large language model, thus solving the problem of low behavior analysis efficiency in related technologies.
[0063] Please refer to Figure 2. This application embodiment provides a behavior analysis method, which is applicable to electronic devices. The electronic device can be the server 130 in the implementation environment shown in Figure 1.
[0064] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.
[0065] As shown in Figure 2, the method may include the following steps:
[0066] Step 210: Obtain multimodal data collected for the target to be detected, and perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a large multimodal model.
[0067] Multimodal data refers to data collected targeting a specific object, carrying information about the object's behavioral state. Multimodal data can be a single type of data or a dataset containing multiple data types. For example, multimodal data could be video data collected by filming the target, or it could be electroencephalogram (EEG) recordings, environmental data, sound data, etc., collected targeting the target.
[0068] Step 230: Perform behavioral analysis on the multimodal data based on the multimodal large model to obtain a behavioral text description of the target to be detected.
[0069] In the process of behavior analysis, it is necessary to format the acquired multimodal data into textual descriptions to obtain general behavioral textual descriptions in text form as the basis for behavior analysis. In this application, multimodal data in various forms are uniformly integrated into textual behavioral descriptions through behavior analysis.
[0070] Step 250: Perform statistical analysis on the behavioral text description based on the large language model to obtain behavioral analysis results for the target to be detected.
[0071] Specifically, the behavioral text descriptions generated by the multimodal large model are statistically summarized, and the behavioral text descriptions are used as prompt words to input into the large language model, thereby guiding the large language model to output behavioral state information of the target to be detected related to the behavioral text descriptions.
[0072] In one possible implementation, video data of animals is collected and analyzed to generate behavioral analysis results that include information about the animal's state, such as its emotions or the degree of illness.
[0073] Through the above process, this application enables the model to learn from large-scale multimodal data by fine-tuning the pre-trained model using multimodal data. This allows the model to better utilize large-scale data, thereby extracting and analyzing general behavioral features, improving the accuracy of behavior classification, recognition, and description, and obtaining a large multimodal model with high accuracy in behavior description. This effectively improves the efficiency of behavior analysis. By combining the large multimodal model and the large language model, behavior analysis is performed on multimodal data in an end-to-end manner, further improving the efficiency of behavior analysis.
[0074] In an exemplary embodiment, as shown in FIG3, step 210 may include the following steps:
[0075] Step 211: Construct a behavioral semantic dataset based on multimodal data.
[0076] The semantic dataset includes multimodal data fragments with corresponding behavioral semantics.
[0077] In one exemplary embodiment, as shown in FIG4, step 211 may include the following steps:
[0078] Step 2111: Perform cluster analysis on the multimodal data to divide the multimodal data into at least two short-time data segments.
[0079] Short-time data segments are fragments obtained by dividing multimodal data into segments of different durations.
[0080] In one possible implementation, the multimodal data is two-dimensional or three-dimensional pose estimation data that includes body point data. In this case, tools such as moseq and Behavior Atlas are used to perform cluster analysis on the body points to obtain short-time data fragments corresponding to different clusters.
[0081] Step 2113: Set corresponding behavioral semantics for short-term data segments to generate a behavioral semantic dataset.
[0082] It is understandable that different short-term data segments can represent different behavioral processes of the target being detected during the data acquisition process. Therefore, corresponding behavioral semantic labels can be set for short-term data segments.
[0083] In one possible implementation, data is collected during the behavior process of the target to be detected, which has specific behavioral semantics. Short-term data segments in the collected multimodal data are selected and corresponding specific behavioral semantics are set. For example, the running process of a mouse is filmed, and then the running segment of the mouse in the filmed video data is set with the behavioral semantics of "running".
[0084] Through the above process, a behavioral semantic dataset is generated by cluster analysis, which can then guide the training direction of the model during subsequent training, thereby improving the efficiency and accuracy of behavioral analysis of the generated multimodal large model.
[0085] Step 213: Train the pre-trained model using behavioral semantic dataset to perform behavioral analysis and obtain a multimodal large model.
[0086] In an exemplary embodiment, as shown in FIG5, step 213 may include the following steps:
[0087] Step 2131: Input the behavioral semantic dataset into the pre-trained model for behavioral analysis to obtain behavioral text descriptions for each short-term data segment in the behavioral semantic dataset.
[0088] The pre-trained model is a language model that generates corresponding behavioral text descriptions based on multimodal data. By inputting short-term data fragments from the behavioral semantic dataset into the pre-trained model, the pre-trained model generates text describing the behavioral semantics corresponding to the short-term data fragments by detecting the short-term data fragments.
[0089] For example, a video clip of a mouse running can be input into a pre-trained model, which can then generate a text description of the behavior: "The mouse in the video is moving at an extremely high speed, i.e., it is running. Its speed is *** and its acceleration is ***."
[0090] Step 2133: Evaluate the accuracy of behavioral text descriptions based on behavioral semantics in the semantic dataset to obtain the evaluation results of the accuracy of behavioral descriptions of short-time data fragments by the pre-trained model.
[0091] The evaluation results are generated by comparing the behavioral text descriptions generated by the pre-trained model with the pre-defined behavioral semantics.
[0092] Step 2135: Based on the evaluation results, tune the parameters of the pre-trained model and repeat the training until the accuracy of the behavior description of the behavior analysis of the pre-trained model reaches the preset threshold, and obtain a multimodal large model.
[0093] In one possible implementation, the semantic dataset is divided into a training set, a validation set, and a test set. The pre-trained model is trained using the training set, and the accuracy of the behavior description of the pre-trained model is calculated using the validation and test sets. The accuracy in the test set is obtained through evaluation by those skilled in the art.
[0094] In the above process, the pre-trained model is finely trained using refined short-term data fragments to generate a large-scale multimodal target model. Supervised fine-tuning and alignment of the model under training are then performed using behavioral semantics, guiding the model towards generating more accurate and refined behavioral text descriptions. Ultimately, a large-scale multimodal target model capable of refined behavioral analysis is obtained. This allows for large-scale local fine-tuning of the model using high-quality, scientific datasets, improving training efficiency and the accuracy and efficiency of model behavior classification, recognition, and description.
[0095] In an exemplary embodiment, as shown in FIG6, step 230 may include the following steps:
[0096] Step 231: Generate corresponding problem information based on multimodal data.
[0097] The question information is text generated based on the type of multimodal data. By inputting the question information into the multimodal big model as prompt words, the multimodal big model generates behavioral text descriptions corresponding to the question information.
[0098] For example, when the multimodal information is mouse video information, the question "Please tell me what the mouse in the video is doing, and its specific motion parameters, such as speed and acceleration" is input into the multimodal large model.
[0099] Step 233: Generate a behavioral text description in a preset format based on the problem information for the corresponding multimodal data.
[0100] Specifically, the behavioral text descriptions for multimodal data and its short-term data segments are standardized using a preset format. For example, a three-part description is used to generate behavioral text descriptions for each short-term data segment, while a two-part description is used to generate behavioral text descriptions for the entire multimodal data.
[0101] In one possible implementation, the behavioral text description format for each short-term data segment is {detailed description, definition, motion parameters}, and the behavioral text description format for the entire multimodal data is {behavioral sequence, motion parameters}.
[0102] In an exemplary embodiment, as shown in FIG7, step 233 may include the following steps:
[0103] Step 2331: Generate corresponding behavioral text descriptions for each short-time data segment in the multimodal data.
[0104] In one possible implementation, the behavioral text description may include behavioral semantic text and motion parameter text.
[0105] In an exemplary embodiment, as shown in FIG8, step 2331 may include the following steps:
[0106] Step 23311: Determine the three-dimensional body point data corresponding to the target to be detected in the short-time data segment.
[0107] Among them, the three-dimensional body point data is the data obtained by three-dimensional motion acquisition after setting fixed data points for the target to be detected. The three-dimensional body points can quantify the specific motion parameters of the target to be detected during the detection process.
[0108] Step 23313: Based on the three-dimensional body point data, predict the motion parameters and generate motion parameter text.
[0109] Among them, motion parameters are calculated by acquiring the relative positions and motion processes of each data point in the three-dimensional body point data. For example, by performing behavioral semantic prediction on mouse videos, motion parameter text such as "the mouse speed is 10m / min and the acceleration is 0" can be generated.
[0110] In one possible implementation, motion parameters are not limited to velocity and acceleration, but also include parameters such as body part orientation, angle, and body length.
[0111] Step 23315: Perform behavioral semantic prediction on short-term data segments to generate behavioral semantic text.
[0112] Semantic text is textual information used to describe the behavior of the target to be detected in short-term data segments. For example, by performing behavioral semantic prediction on mouse videos, behavioral semantic text such as "the mouse in the video is licking its fur, grooming itself with its front paws, or scratching itself" can be generated.
[0113] Through the above process, precise behavioral semantic text can be generated for short-term data fragments, thereby improving the accuracy of model behavior analysis.
[0114] Step 2333: Generate corresponding multimodal data behavior text descriptions for the behavior sequences of each short-time data segment.
[0115] It should be noted that short-time data segments in multimodal data have temporal relationships. Therefore, behavioral text descriptions of each short-time data segment are summarized by behavioral sequences to generate corresponding multimodal data behavioral text descriptions.
[0116] For example, by performing behavioral semantic prediction on mouse videos, behavioral semantic text can be generated that reads, "The mouse in the video is grooming itself for the first 10 seconds and sniffing for the next 5 seconds. Its movement distance in the video is 0, and its speed remains unchanged."
[0117] Through the above process, fine-grained behavioral analysis of multimodal data can be achieved, which can accurately obtain the behavioral semantic text of multimodal data. This ensures that statistical analysis performed through a large language model can accurately obtain the behavioral information of the target to be detected, thereby improving the accuracy and efficiency of behavioral analysis.
[0118] Figure 9 is a schematic diagram illustrating the specific implementation of a behavior analysis method in an application scenario.
[0119] In this application scenario, a high-performance server with an A100 80G is used to train a large multimodal model and perform behavioral analysis.
[0120] As shown in Figure 9, by acquiring mouse videos, two-dimensional or three-dimensional pose estimation data are collected, and tools such as moseq and Behavior Atlas are used to perform cluster analysis on body points. Based on the clustering results, short video segments with behavioral semantics are selected to generate a semantic dataset.
[0121] A multimodal large model is obtained by training a pre-trained model using a semantic dataset.
[0122] Input the question "Please tell me what the mouse in the video is doing, and its specific motion parameters, such as speed and acceleration" into the multimodal model.
[0123] The multimodal large model generates behavioral text descriptions for short video clips based on the format {detailed description, definition, motion parameters}: "The mouse in the video is licking its fur, grooming with its front paws, or scratching, i.e. grooming. Its velocity is 0 and its acceleration is 0."
[0124] Since videos that are too long or too high in resolution cannot be processed, a model is used to read mouse videos in a loop to obtain short and long video segments.
[0125] The multimodal large model generates behavioral text descriptions for long video clips based on the format of {behavioral sequence, motion parameters} - "The mouse in the video first preens its fur for 3 seconds, then runs for 2 seconds, turns to the right, continues jogging for 2 seconds, and finally stops to sniff for 2 seconds. Its movement distance in the video is ***, and its speed shows a trend of first accelerating and then decelerating."
[0126] By summarizing the behavioral text descriptions generated by the multimodal large model and inputting them into the large language model for data statistical analysis, behavioral analysis results are generated.
[0127] This invention realizes refined behavior analysis of animals using a multimodal large-scale model. A method based on a multimodal large-scale model is proposed to perform behavior analysis on animal behavior video data in an end-to-end manner. Based on a pre-trained multimodal large-scale model, and fine-tuned locally using a high-quality, scientific behavior video dataset, better refined behavior analysis results are obtained than existing techniques. The model structure and parameter scale of the multimodal large-scale model allow the model to better utilize large-scale data, thus enabling the extraction and analysis of general behavioral features. Statistical analysis is performed on the behavioral text descriptions output from the complete video using a large language model, improving the efficiency of behavior analysis.
[0128] The following are embodiments of the apparatus described in this application, which can be used to execute the behavior analysis method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the behavior analysis method involved in this application.
[0129] Please refer to Figure 10. In this embodiment of the application, a behavior analysis device 900 is provided, including but not limited to: a data acquisition module 910, a behavior analysis module 930, and a statistical analysis module 950.
[0130] The data acquisition module 910 is used to acquire multimodal data collected for the target to be detected, and to perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a large multimodal model.
[0131] The Behavior Analysis Module 930 is used to perform behavior analysis on multimodal data based on a multimodal large model to obtain a textual description of the behavior of the target to be detected.
[0132] The statistical analysis module 950 is used to perform statistical analysis on behavioral text descriptions based on a large language model to obtain behavioral analysis results for the target to be detected.
[0133] It should be noted that the behavior analysis device provided in the above embodiments is only illustrated by the division of the above functional modules when performing behavior analysis. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the behavior analysis device will be divided into different functional modules to complete all or part of the functions described above.
[0134] Furthermore, the embodiments of the behavior analysis device and the behavior analysis method provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.
[0135] Please refer to Figure 11. This application embodiment provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc.
[0136] In Figure 11, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.
[0137] Data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 11, but this does not indicate that there is only one bus or one type of bus.
[0138] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.
[0139] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0140] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 400, but not limited thereto.
[0141] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.
[0142] The computer-readable instructions are executed by one or more processors 4001 to implement the behavior analysis methods in the above embodiments.
[0143] Furthermore, this application provides a storage medium storing computer-readable instructions that are executed by one or more processors to implement the behavior analysis method described above.
[0144] This application provides a computer program product including computer-readable instructions stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, thereby enabling the electronic device to implement the behavior analysis method described above.
[0145] Compared with related technologies, this application, through fine-tuning a pre-trained model using multimodal data, enables learning from large-scale multimodal data. This allows the model to better utilize large-scale data, thereby extracting and analyzing general behavioral features, improving the accuracy of behavior classification, recognition, and description, and obtaining a large multimodal model with high behavior description accuracy, effectively improving the efficiency of behavior analysis. By combining the large multimodal model and a large language model, behavior analysis is performed on multimodal data in an end-to-end manner, improving the efficiency of behavior analysis. Cluster analysis generates a behavioral semantic dataset, which can guide the model training direction in subsequent training processes, thereby improving the efficiency and accuracy of the generated large multimodal model in behavior analysis. The pre-trained model is fine-trained using refined short-time data fragments to generate a target large multimodal model. Supervised fine-tuning and alignment of the model to be trained using behavioral semantics guides the model towards generating more accurate and refined behavioral text descriptions, ultimately obtaining a target large multimodal model capable of refined behavior analysis. This allows for large-scale local fine-tuning of the model using high-quality, scientific datasets, improving training efficiency and the accuracy and efficiency of model behavior classification, recognition, and description. Achieving refined behavioral analysis of multimodal data can accurately obtain the behavioral semantic text of multimodal data, ensuring that statistical analysis through large language models can accurately obtain the behavioral information of the target to be detected, thereby improving the accuracy and efficiency of behavioral analysis.
[0146] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0147] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A behavior analysis method, characterized in that, The method includes: Obtain multimodal data collected for the target to be detected, and perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a large multimodal model; Behavioral analysis is performed on the multimodal data based on a multimodal large model to obtain a behavioral text description of the target to be detected; Statistical analysis of the behavioral text description is performed based on a large language model to obtain behavioral analysis results for the target to be detected.
2. The method as described in claim 1, characterized in that, The step of training the pre-trained model based on the multimodal data to obtain a large multimodal model includes: A behavioral semantic dataset is constructed based on the aforementioned multimodal data; Based on the aforementioned behavioral semantic dataset, the pre-trained model is trained using behavioral analysis to obtain a multimodal large model.
3. The method as described in claim 2, characterized in that, The construction of the behavioral semantic dataset based on the multimodal data includes: Cluster analysis is performed on the multimodal data to divide the multimodal data into at least two short-time data segments; The corresponding behavioral semantics are set for the short-term data segments to generate a behavioral semantic dataset.
4. The method as described in claim 2, characterized in that, The step of training a pre-trained model based on the behavioral semantic dataset to obtain a multimodal large model includes: The behavioral semantic dataset is input into the pre-trained model for behavioral analysis to obtain behavioral text descriptions for each short-time data segment in the behavioral semantic dataset. The accuracy of the behavioral text description is evaluated based on the behavioral semantics in the semantic dataset, and the evaluation result indicating the accuracy of the pre-trained model in describing the behavioral semantics of short-time data segments is obtained. Based on the evaluation results, the parameters of the pre-trained model are tuned and the training is repeated until the accuracy of the behavior description of the behavior analysis of the pre-trained model reaches a preset threshold, thereby obtaining a multimodal large model.
5. The method as described in claim 1, characterized in that, The step of performing behavioral analysis on the multimodal data based on a multimodal large model to obtain a behavioral text description of the target to be detected includes: Based on the multimodal data, corresponding problem information is generated; Based on the problem information, a behavioral text description in a preset format corresponding to the multimodal data is generated.
6. The method as described in claim 5, characterized in that, The step of generating a behavioral text description corresponding to the multimodal data based on the problem information includes: Generate corresponding behavioral text descriptions for each short-time data segment in the multimodal data; Generate corresponding multimodal data behavior text descriptions for the behavior sequences of each short-time data segment.
7. The method as described in claim 6, characterized in that, The behavioral text description includes behavioral semantic text and motion parameter text; The generation of corresponding behavioral text descriptions for each short-time data segment in the multimodal data includes: Determine the three-dimensional body point data corresponding to the target to be detected in the short-time data segment; Based on the three-dimensional body point data, motion parameters are predicted, and motion parameter text is generated. Behavioral semantic prediction is performed on the short-term data fragments to generate behavioral semantic text.
8. A behavior analysis device, characterized in that, include: The data acquisition module is used to acquire multimodal data collected for the target to be detected, and to perform behavior analysis training on the pre-trained model based on the multimodal data to obtain a large multimodal model; The behavior analysis module is used to perform behavior analysis on the multimodal data based on a multimodal large model to obtain a textual description of the behavior of the target to be detected. The statistical analysis module is used to perform statistical analysis on the behavioral text description based on a large language model to obtain behavioral analysis results for the target to be detected.
9. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein, The memory stores computer-readable instructions; The computer-readable instructions are executed by one or more of the processors, causing the electronic device to implement the behavior analysis method as described in any one of claims 1 to 7.
10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by one or more processors to implement the behavior analysis method as described in any one of claims 1 to 7.