Enhanced decision making through few-shot adaptation of artificial intelligence models for action detection

Low-Rank Adaptation (LoRA) with a feature augmentation module enhances the training of VLMs for action detection, addressing computational inefficiencies and overfitting, thereby improving accuracy and efficiency in few-shot learning scenarios.

WO2026236301A1PCT designated stage Publication Date: 2026-11-12NEC LABORATORIES AMERICA INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/026808
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2026-04-29
Filing Date
2026-05-06
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Large-scale pre-trained Video-Language Models (VLMs) face inefficiencies in real-time applications due to their size and computational demands, and adapting them with scarce labeled data leads to overfitting and a mismatch between pre-training objectives and person-centric action detection tasks, requiring fine-grained understanding and localization of specific actions within complex scenes.

Method used

Employing Low-Rank Adaptation (LoRA) to inject trainable low-rank matrices into specific layers of VLMs, combined with a learnable feature augmentation module to enhance data diversity and guide adaptation within the frozen model, optimizing the training process for action detection with few-shot learning.

Benefits of technology

This approach reduces computational resources and overfitting, improving the accuracy and data efficiency of machine learning models for action detection while maintaining performance on downstream tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026026808_12112026_PF_FP_ABST
    Figure US2026026808_12112026_PF_FP_ABST
Patent Text Reader

Abstract

Systems and methods for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection. Augmented feature maps can be obtained by injecting feature augmentations of detected features from video frames into a feature space of a vision encoder of a machine learning model. The vision encoder can be trained to learn low rank adaptation parameters based on the augmented feature maps. Entity level features can be extracted from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters. Similarity between the entity level features and action text embeddings obtained from the video frames can be maximized by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.
Need to check novelty before this filing date? Find Prior Art

Description

ENHANCED DECISION MAKING THROUGH FEW-SHOT ADAPTATION OF ARTIFICIAL INTELLIGENCE MODELS FOR ACTION DETECTIONRELATED APPLICATION INFORMATION

[0001] This application claims priority to U.S. Provisional App. No. 63 / 802,678, filed on May 9, 2025 and U.S. Utility App. No. 19 / 662,703, filed April 29, 2026, both of which are incorporated herein by reference in its entirety.BACKGROUNDTechnical Field

[0002] The present invention relates to optimizing training of artificial intelligence (AI) models, and more particularly to enhanced decision making through few-shot adaptation of artificial intelligence models for action detection.Description of the Related Art

[0003] Large-scale pre-trained models, particularly Video-Language Models (VLMs), have demonstrated remarkable capabilities in understanding multimodal data, achieving state-of-the art results on various video and language tasks. These models can be leveraged for downstream applications like text-to-image generation as they deliver strong performance even without task-specific training with massive datasets.SUMMARY

[0004] According to an aspect of the present invention, a method is provided, including, obtaining augmented feature maps by injecting feature augmentations of detected features24161PCT Page 1 of 35from video frames into a feature space of a vision encoder of a machine learning model, training the vision encoder to learn low rank adaptation parameters based on the augmented feature maps, extracting entity level features from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters, and maximizing similarity between the entity level features and action text embeddings obtained from the video frames by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.

[0005] According to another aspect of the present invention, a system is provided, including, a memory device, one or more processor devices operatively coupled with the memory device to perform operations including, obtaining augmented feature maps by injecting feature augmentations of detected features from video frames into a feature space of a vision encoder of a machine learning model, training the vision encoder to learn low rank adaptation parameters based on the augmented feature maps, extracting entity level features from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters, and maximizing similarity between the entity level features and action text embeddings obtained from the video frames by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.

[0006] According to yet another aspect of the present invention, a non-transitory computer program product including a computer-readable storage medium including a program code, wherein the program code when executed on a computer causes the24161PCT Page 2 of 35computer to perform operations including, obtaining augmented feature maps by injecting feature augmentations of detected features from video frames into a feature space of a vision encoder of a machine learning model, training the vision encoder to learn low rank adaptation parameters based on the augmented feature maps, extracting entity level features from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters, and maximizing similarity between the entity level features and action text embeddings obtained from the video frames by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.BRIEF DESCRIPTION OF DRAWINGS

[0008] The disclosure will provide details in the following description of preferred embodiments with reference to the following figures wherein:

[0009] FIG. 1 is a block diagram that shows a system for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention;

[0010] FIG. 2 is a block diagram that shows a computer system for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection is illustratively depicted in accordance with an embodiment of the present invention;24161PCT Page 3 of 35

[0011] FIG. 3 is a block diagram that shows components of a computer system for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention;

[0012] FIG. 4 is a block diagram that shows components of a computer system for learning low rank adapter parameters for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention;

[0013] FIG. 5 is a block diagram that shows components of the vision encoder to generate a feature map for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention;

[0014] FIG. 6 is a flow diagram that shows a high-level overview of enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention; and

[0015] FIG. 7 is a block diagram that shows a practical application for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention.DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0016] In accordance with embodiments of the present invention, systems and methods are provided for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection.24161PCT Page 4 of 35

[0017] In the present embodiments, augmented feature maps can be obtained by injecting feature augmentations of detected features from video frames into a feature space of a vision encoder of a machine learning model. The vision encoder can be trained to learn low rank adaptation parameters based on the augmented feature maps. Entity level features can be extracted from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters. Similarity between the entity level features and action text embeddings obtained from the video frames can be maximized by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.

[0018] Large-scale pre-trained models, particularly Video-Language Models (VLMs), can be trained on millions of video and text pairs, focusing on entire scenes rather than specific person-centered actions. This large-scale training helps them capture strong spatial and temporal features, which can be adapted for detecting person-centered actions using only a small amount of additional training data.

[0019] However, a key drawback of VLMs is their size and computational demands, making them inefficient for real-time, cost-effective applications. VLMs processes patches from all the frames in the entire video frame at every layer resulting in computation requirement to increase with the number of layers and number of patches. Adapting these large models effectively, especially when labeled data for the target task is scarce, presents significant hurdles.

[0020] Fine-tuning the entire VLM, with its billions of parameters, on a small dataset inevitably leads to severe overfitting, hindering generalization.24161PCT Page 5 of 35

[0021] Additionally, many powerful VLMs are pre-trained on tasks like video captioning or text-video retrieval, which typically involve reasoning about the global scene content. In contrast, action detection demands fine-grained understanding, requiring the localization of specific individuals and the identification of their potentially concurrent actions within complex scenes involving multiple people. This represents a significant granularity mismatch between pre-training objectives and the downstream task.

[0022] Moreover, while data augmentation is a standard technique to combat limited data, conventional augmentations applied at the input level (e.g., video transformations) may not be sufficient or optimally targeted for adapting the internal representations of a large frozen model. Generating diverse and task-relevant variations within the feature space could provide a more effective signal for adaptation, particularly for bridging the gap between scene-level pre-training and person-centric action detection.

[0023] Further, spatial temporal action detection requires fine grained labels for the location of the person with their actions. This kind of annotation is both time-consuming and costly.

[0024] The present embodiments resolve these issues by optimizing the training of machine learning models such as visual language models for learning action detection from just a few examples. Low-Rank Adaptation (LoRA) can be employed for parameterefficient fine-tuning while keeping the machine learning models frozen to update low-rank matrices injected into specific layers of the machine learning model. To enhance data diversity and guide adaptation effectively, a learnable feature augmentation module can be applied internally within the frozen machine learning model. As such, the present embodiments improve accuracy of machine learning models to learn actions while24161PCT Page 6 of 35increasing data efficiency by reducing the computational resources (e.g., compute and storage) needed to train the machine learning models.

[0025] Embodiments described herein may be entirely hardware, entirely software or including both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0026] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer readable medium may include any apparatus that stores, communicates, propagates, or transports the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. The medium may include a computer-readable storage medium such as a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk, etc.

[0027] Each computer program may be tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the24161PCT Page 7 of 35storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.

[0028] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code to reduce the number of times code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0029] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.

[0030] Referring now in detail to the figures in which like numerals represent the same or similar elements and initially to FIG. 1, a block diagram shows a system for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention.

[0031] In an embodiment using a system 100, monitored entities 140 can include entity 141, system component 143, and autonomous vehicle 145. The monitored entities 140 can generate an input dataset that include image / video 102. The image / video 102 can be transmitted to an analytic server 106 that can implement enhanced decision making through few-shot adaptation of artificial intelligence models for action detection 600. The analytic24161PCT Page 8 of 35server 106 can obtain an optimized machine learning (ML) model 118 The optimized ML model 118 can be utilized to perform downstream tasks 120.

[0032] System 100 can be utilized to perform downstream tasks 120 based on the input image / video 102 and user queries 128 from a decision-making entity 127. The downstream tasks 120 can include entity identification 121, system maintenance 123, and vehicle control 125. The analytic server 106 can generate a corrective action for the downstream tasks 120 to be sent to respective computing systems for the monitored entities 140 through a network.

[0033] In entity identification 121, the image / video 102 (e.g., location images, scene images, entity images such as parts of the entity, etc.) related to the entity 141 can be processed by the analysis server 106 to answer user queries 128. The user queries 128 can be relevant to the entity 141 such as their attributes (e.g., position, direction of movement, color of clothing, etc.), relationship with other entities within a scene (e.g., proximity, behavior, etc.), relationship with the environment, etc. The optimized ML model 118 can predict future attributes, and relationships of the entity 141.

[0034] Based on the predictions of the optimized ML model 118, a corrective action can be generated. The corrective action can include notifying the decision making entity 127 of the predictions about the entity 141 based on their image / video 102, generating resolutions to an issue caused by the entity (e.g., the entity 141 as a disabled vehicle in a traffic scene and the resolution is the deployment of a repair technician, etc.) of the image / video 102 to help with the decision making process of the decision making entity 127, etc. For example, an entity 141 (e.g., a worker, autonomous worker, etc.) in a warehouse performs task A in workflow D in manufacturing widget X. Outputs of entity24161PCT Page 9 of 35141 for task A is above the expected yield in the scheduled work period. Actions performed by entity 141 can then be learned and replicated to other entities 141 to increase yield in task A in workflow D. In another example, output of a first entity 141 has been below the expected yield and actions performed by a second entity 141 has been above the expected yield. As such, the layout of the warehouse can be updated to redirect the work to the second entity 141, and have first entity 141 flagged for maintenance.

[0035] In system maintenance 123, image / video 102 (e.g., system logs, test cases, hardware status images, etc.) related to the system component 143 can be processed to answer user queries 128. The user queries 128 can be relevant on how to properly maintain the system component 143 based on the image / video 102, the detected entities, and their corresponding detected actions. A corrective action can be generated by the analytic server 106 which can include the answer to the user queries 128 (e.g., determine causes to bandwidth issues, etc.) to maintain the system component 143. Based on the corrective action (e.g., adding bandwidth, blocking packets from an identified internet protocol (IP) address to resolve malicious attacks, restarting hardware, etc.) the network system can be autonomously maintained.

[0036] In vehicle control 125, image / video 102 (e.g., vehicle part status, traffic scene image, etc.) related to the autonomous vehicle 145 can be processed to answer user queries 128. The user queries 128 can be relevant to how to control the autonomous vehicle 145 given its environment based on the image / video 102, the detected entities, and their corresponding detected actions. A corrective action can be generated by the analytic server 106 which can include the answer to the user queries 128 to control the proper performance of the autonomous vehicle 145. Based on the corrective action (e.g., stopping, speeding up,24161PCT Page 10 of 35changing direction, etc.) the autonomous vehicle 145 can be autonomously controlled using appropriate control devices (e.g., advanced driver assistance systems, braking device, accelerator device, cooling device, etc.) within the autonomous vehicle. In an embodiment, the autonomous vehicle 145 can be controlled in response to avoid a predicted event based on a generated trajectory such as multi-vehicle collision, accidents, detected road hazards, etc.

[0037] In another embodiment, in vehicle control 125, the autonomous vehicle 145 can be controlled to verify and test the functionality of the various components (e.g., advanced driver assistance systems, braking device, accelerator device, cooling device, etc.) of the autonomous vehicle 145 by autonomously controlling the components and generate test data that can be used to fine-tune / train the optimized ML model 118.

[0038] Other downstream tasks and practical applications are contemplated.

[0039] The analytic server 106 can include a processor device 113, data storage device 116, memory 112, communications subsystem 111, peripheral devices 114, and input / output (I / O) bus 115. The analytic server 106 is an implementation of a computer system. Other implementations are contemplated. The computer system is shown in more detail in FIG. 2.

[0040] Referring now to FIG. 2, a block diagram shows a computer system for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection is illustratively depicted in accordance with an embodiment of the present invention.

[0041] The computing device 200 illustratively includes the processor device 113, an input / output (I / O) subsystem 190, a memory 112, a data storage device 116, and a24161PCT Page 11 of 35communications subsystem 111, and / or other components and devices commonly found in a server or similar computing device. The computing device 200 may include other or additional components, such as those commonly found in a server computer (e.g., various input / output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory 112, or portions thereof, may be incorporated in the processor device 113 in some embodiments.

[0042] The processor device 113 may be embodied as any type of processor capable of performing the functions described herein. The processor device 113 may be embodied as a single processor, multiple processors, a Central Processing Unit(s) (CPU(s)), a Graphics Processing Unit(s) (GPU(s)), a single or multi-core processor(s), a digital signal processor(s), a microcontroller(s), or other processor(s) or processing / controlling circuit(s).

[0043] The memory 112 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory 112 may store various data and software employed during operation of the computing device 200, such as operating systems, applications, programs, libraries, and drivers. The memory 112 is communicatively coupled to the processor device 113 via the I / O subsystem 115, which may be embodied as circuitry and / or components to facilitate input / output operations with the processor device 113, the memory 112, and other components of the computing device 200. For example, the I / O subsystem 115 may be embodied as, or otherwise include, memory controller hubs, input / output control hubs, platform controller hubs, extracted control circuitry, firmware devices, communication24161PCT Page 12 of 35links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems to facilitate the input / output operations. In some embodiments, the I / O subsystem 115 may form a portion of a system-on-a-chip (SOC) and be incorporated, along with the processor device 113, the memory 112, and other components of the computing device 200, on a single extracted circuit chip.

[0044] The data storage device 116 may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. The data storage device 116 can store program code for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection 600. Any or all of these program code blocks may be included in a given computing system.

[0045] The communications subsystem 111 of the computing device 200 may be embodied as any network interface controller or other communication circuit, device, or collection thereof, capable of enabling communications between the computing device 200 and other remote devices over a network. The communications subsystem 111 may be configured to employ any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, WiFi®, WiMAX, etc.) to effect such communication.

[0046] As shown, the computing device 200 may also include one or more peripheral devices 114. The peripheral devices 114 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, the peripheral devices 114 may include a display, touch screen, graphics24161PCT Page 13 of 35circuitry, keyboard, mouse, speaker system, microphone, network interface, and / or other input / output devices, interface devices, GPS, camera, and / or other peripheral devices.

[0047] Of course, the computing device 200 may also include other elements (not shown), as readily contemplated by one of skill in the art, as well as omit certain elements. For example, various other sensors, input devices, and / or output devices can be included in computing device 200, depending upon the particular implementation of the same, as readily understood by one of ordinary skill in the art. For example, various types of wireless and / or wired input and / or output devices can be employed. Moreover, additional processors, controllers, memories, and so forth, in various configurations can also be utilized. These and other variations of the computing device 200 are readily contemplated by one of ordinary skill in the art given the teachings of the present invention provided herein.

[0048] As employed herein, the term “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software or combinations thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem can include one or more on-board memories (e.g., caches, dedicated memory arrays, read only memory, etc.). In some embodiments, the hardware processor subsystem can include one or more memories that can be on or off board or that24161PCT Page 14 of 35can be dedicated for use by the hardware processor subsystem (e g., ROM, RAM, basic input / output system (BIOS), etc.).

[0049] In some embodiments, the hardware processor subsystem can include and execute one or more software elements. The one or more software elements can include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0050] In other embodiments, the hardware processor subsystem can include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry can include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PL As).

[0051] These and other variations of a hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0052] Referring now to FIG. 3, a block diagram shows components of a computer system for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention.

[0053] In an embodiment, image / video 102 can be processed to train a video language model 320 and obtain an optimized ML model 118. The image / video 102 can be processed to extract entity bounding boxes 301, video frames 303 and action labels 305. An attention mask generator 307 can process the entity bounding boxes 301 to generate attention mask 311 for detected entities within the entity bounding boxes 301. The attention mask 311, video frames 303 and action labels 305 can be utilized by the video language model 320 for training.24161PCT Page 15 of 35

[0054] The video language model 320 can include a vision encoder 321 and a text encoder 329. The vision encoder 321 can utilize low rank adaptation (LoRA) parameters 323 to generate augmented feature map 325 based on the video frames 303 and attention mask 311. The augmented feature map 325 can be processed to obtain region of interest alignment 327 which is an estimate of regions of interest that provides context for an entity performing an action. The text encoder 329 can process action labels 305 to generate action text embeddings 330.

[0055] The action text embeddings 330 and the ROI alignment 327 can be integrated into an alignment loss 331 for training the video language model 320 and obtain the optimized ML model 118.

[0056] Referring now to FIG. 4, a block diagram shows components of a computer system for learning low rank adapter parameters for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention.

[0057] In an embodiment, a feature augmentation model 401 can generate augmentations for video features for the vision encoder 321. The feature augmentation module 401 can be injected into the intermediate layers 409 of the vision encoder 321. The feature augmentation module 401 can include a learnable embedding table 403 and parameter generator 405 for performing feature-wise linear modulation (FiLM) and learn modulation parameters 407. The augmented features generated with the feature augmentation module 401 can be utilized by the model trainer 320 to minimize a training objective including an anchor task loss 411, a distillation loss 413, a weighted augmented loss 415 and an entropy maximization loss 417 to learn the LoRA parameters 323.24161PCT Page 16 of 35

[0058] Referring now to FIG. 5, a block diagram shows components of the vision encoder to generate the feature map for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention.

[0059] In an embodiment, video frames 303 can be processed by the vision encoder 321 to generate the augmented feature map 325. The vision encoder 321 can utilize the learned LoRA parameters 323 to generate the augmented feature map 325. The vision encoder 321 can extract video frame patches 501 from the video frames 303. The video frame patches 501 can be processed by the intermediate layers 409 of the vision encoder 321. An attention mask 311 can be injected into the intermediate layers 409 and generate feature vectors. The feature vectors can be processed by a patch scoring module 503 and the top-k patches 507 can be selected by the patch selection module 505. The top-K patches 507 can be processed by the output layers 509 of the vision encoder 321 to generate the feature map 325.

[0060] Referring now to FIG. 6, a flow diagram shows a high-level overview of enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention.

[0061] In an embodiment, augmented feature maps can be obtained by injecting feature augmentations of detected features from video frames into a feature space of a vision encoder of a machine learning model. The vision encoder can be trained to learn low rank adaptation parameters based on the augmented feature maps. Entity level features can be extracted from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters. Similarity between the entity level features and action text24161PCT Page 17 of 35embeddings obtained from the video frames can be maximized by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.

[0062] In the present embodiments, Low-Rank Adaptation (LoRA) can be employed to mitigate overfitting and reduce computational cost in the few-shot setting for detecting actions with machine learning models. LoRA injects trainable low-rank matrices into specific layers. For a pre-trained weight matrix W0∈ ℝd×k, the adapted weight W is parameterized as W = Wo+ BA, where B ∈ ℝd×randA ∈ ℝr×kare trainable low-rank matrices with rank r « min(d, k), while W0remains frozen.

[0063] LoRA can be applied to the query (Wq) and value (Wv) projection matrices within the multihead self-attention (MHSA) modules of the vision encoder’s transformer blocks, specifically from layer index ltoraonwards. This focuses adaptation on higher-level visual features. Let 0 denote the frozen pre-trained parameters of the VLM 320 and φlorarepresent the set of all trainable LoRA parameters 323 (the matrices A and B across the adapted layers). The optimized ML model 118 computes outputs based on both 0 and φlora.

[0064] In block 610, injecting feature augmentations of detected features from video frames into the feature space of a vision encoder.

[0065] In an embodiment, to enhance generalization from limited data, augmentations can be introduced directly in the feature space of the vision encoder 321. A feature augmentation module 401 based on Feature-wise Linear Modulation (FiLM) can be inserted after an intermediate layers 409 laugof the frozen vision encoder 321 backbone. Let f(l)∈ ℝB×T′×Cbe the output feature map of layer laugfor a batch of B input sequences, where T' is the sequence length of visual tokens and C is the feature dimension.24161PCT Page 18 of 35

[0066] In block 611, feature augmentations can be generated based on the modulating parameters predicted by a parameter generator based on a learnable embedding table.

[0067] In an embodiment, the feature augmentation module 401 generates N distinct augmented versions of these features. The feature augmentation module 401 can include: a learnable embedding table 403 E ∈ ℝN×D, where N is the number of augmentation types and Deis the embedding dimension. Each row etrepresents the i-th augmentation embedding.

[0068] To generate the augmentations, a parameter generator 405can be utilized and implemented as a small trainable MultiLayer Perceptron (MLP) with parameters \| / . For each embeddingit predicts modulation parameters 407. Yi ’Pt)' (. Yi > Pi) — 9tp.ei)’Yi > PiG(1) The parameters \| / and the embedding table E constitute the modulation parameters 407 φaugof this module. The parameter generator 405 g^is initialized such that Yi ~ 1and Pt ~ 0 initially for all i = 1,..., N.

[0069] In block 613, detected features from the video frames can be modulated to generate augmented feature maps.

[0070] In an embodiment, the parameter generator 405 can perform a FiLM application step where each (γi, βi) modulates the original features f(l)to produce the i-th augmented feature map where O denoteselement-wise multiplication with broadcasting over the batch and sequence dimensions.

[0071] In block 620, the vision encoder can be trained to learn the low rank adaptation parameters and the modulation parameters based on the augmented feature maps and the detected features from the video frames.24161PCT Page 19 of 35

[0072] In an embodiment, during training, the original features f(l)and all N augmented feature maps 325 {f(l)aug,i}Ni=1are Passed through the subsequent layers of the network (which include the trainable LoRA parameters 323 φlora). This yields output logits zorig∈ ℝB×Kcorresponding to the original features and a set of augmented logits {zaug,i∈where K is the number of output classes. The overall trainable parameters during optimization are φ = φlora∪ φaug(e.g., modulation parameters 407 with the LoRA parameters 323).

[0073] In block 621, loss functions including an anchor task loss, a distillation loss, a weighted augmented loss, and an entropy maximization loss can be computed based on the augmented feature maps and the detected features.

[0074] In an embodiment, the Anchor Task Loss 411 (ℒBCE) can be defined as the standard supervised loss applied to the predictions from the original, unaugmented features zorigFor multi-label classification, the Binary Cross-Entropy (BCE) loss with logits can be used.is the logit for the k-th class of the b-th sample.

[0075] In an embodiment, the Distillation Loss 413 (ℒdistill) can be defined as a regularization term encouraging predictions from augmented features (zaug,i,to remain consistent with those from the original features (zorig). The sigmoid probabilities derived from zorigcan be used as soft targets, detaching them from the gradient computation. Let Porig.b=° zorig,b ) detached - The loss is the average BCE between augmented logits and these soft targets:24161PCT Page 20 of 35where G( ) denotes the element-wise sigmoid function.

[0077] In an embodiment, the Weighted Augmented Loss 415 (£BCE-W)canbe defined as the supervised BCE loss applied to predictions from augmented features, modulated by a per-sample weight wi bG [0,1] reflecting the "reasonableness" of the i-th augmentation for the b-th sample. Let ft bbe the average BCE loss for the augmented sample (i,b): iib=.k=iBCEWithLogits zaUgiiib k,porigb i) (6). The total weighted augmented loss is:

[0078] The weights wi bin Eq. 7 dynamically modulate the contribution of each augmented sample based on its predictive divergence from the original sample, relative to other augmentations in the group. The dissimilarity between the i-th augmented prediction and the (detached) original prediction for sample b can be measured using the average BCEwhere

[0079] For each sample b, the mean pd band standard deviation ad bof these distancesacross the N augmentations can be computed: pd b= di b(10) ad b=(11) The weight wi bis calculated using a Gaussian functioncentered at the mean distance \id b, based on the z-score of the distance di bzi b=(12); wi b= exp (— ^-) (13) where s is a hyper-parameter controlling thesensitivity to deviations from the mean distance, and £ is a small constant (e.g., 106) for numerical stability. This strategy assigns higher weights to augmentations whose24161PCT Page 21 of 35predictive divergence is close to the average divergence observed for that sample across all N augmentations.

[0080] In an embodiment, the Entropy Maximization Loss 417 (Lent) can be defined as a loss term encouraging diversity and uncertainty in the predictions derived from augmented features. The negative average binary entropy of the predicted probabilities foraugmented samples can be minimized: Lent=1t=i ' b=i1k=iH.a(.zaug,i,b,k ')') (8) where H(p) = — [plog p + (1 — p)log l — p)] is the binary entropy function.

[0081] In block 623, the training objective can be minimized as a combination of loss functions with regularization terms derived from the augmented samples. In an embodiment, this combination of the loss functions with regularization terms promotes robustness and diversity. The total loss totai isaweighted sum of four components:are scalar hyper-parameters. Let y 6 {0,1}exKbe the ground truth multi-label matrix for the batch.

[0082] In block 630, entity level features can be extracted from the video frames by performing regions of interest alignment on an augmented feature map. In an embodiment, regions of interest (ROI) alignment is used to extract entity level features from the video features from the augmented feature map. ROI alignment extracts small, fixed-size feature maps from arbitrary-sized regions of interest obtained from a larger feature map. The regions of interest can include context of an entity performing an action. The augmented feature map 325 can be generated based on attention masks generated from video frames.

[0083] In block 631, attention masks can be generated by processing video frames obtained by a vision encoder with frozen weights that utilizes the LoRA parameters.24161PCT Page 22 of 35

[0084] Tn an embodiment, after learning the LoRA parameters 323, action detection with few-shot adaptation can be performed. Video frames 303 can be processed by the frozen vision encoder 321 with LoRA parameters 323 and attention mask 311 generated using an attention mask generator 307 which takes as input the entity bounding boxes 301 which includes bounding box coordinates of all the entities in the image / video 102 to output an video feature map.

[0085] In an embodiment, to generate the attention masks 311, the attention mask generator 307 can take as input the top left and top right coordinates of the entity bounding boxes 301 and output the attention mask 311 as a zero-padded mask where the frame patch within the bounding box contains ones and outside is padded with zeros. The vision encoder 321 applies the attention mask 311 during attention computation to emphasize the features around the location of the entity in the frame and guide the LoRA parameters 323 to focus on the entity level details as an attention feature map. This attention feature map contains features per patch across all the input frames.

[0086] In block 633, top-k patches of features from an attention feature map can be selected based on their importance.

[0087] In an embodiment, a patch scoring module is utilized after processing half of the layers of the vision encoder 321 to score the patches based on their importance. To score the patches based on importance, the patch scoring module 503 takes the features of the patch features from previous layers and uses a linear layer plus a sigmoid function to output a score per patch. The sigmoid function outputs a score between 0 and 1 for each of the patches. Since the scoring function is learnable and conditioned on the patch features, it is directly optimized by the downstream task training. The scoring module will learn to score24161PCT Page 23 of 35important patches (like human and objects the human interacts with) higher and background patches lower as they do not help with the action prediction.

[0088] In an embodiment, the patch scoring module 503 is implemented using a simple linear layer without the bias term. The patch scores can be used by the patch selection module 505 to select top-k patches where “k” is a heuristic parameter to control the tradeoff between computation and accuracy.

[0089] To control the tradeoff, the K value in top-K selection is set appropriately. For computation, the memory usage as well as latency of the model (how long it takes to process 1 clip using the model forward pass) can be considered. For accuracy, mean average precision can be utilized which checks how often the model can predict the correct actions while also considering the bounding box prediction of the entity detection.

[0090] In an embodiment, the patch selection module 505 can be any differentiable formulation of top-k operation such as straight through estimator or gumbel softmax approximation which selects the index of the top k maximum values from the patch scores generated by a patch scoring module 503 in a continuous manner. Only the selected patches are further processed by the corresponding layers (e.g., output layers). The output video feature map is padded with zeros for the ignored patches. The top-k patches can be processed by the output layers of the vision encoder 321 to generate the augmented feature maps 325.

[0091] For example, if the total patches in a video clip are 5000, using the scores from the scoring module select top-K patches are selected (e.g., K can be smaller compared to total patches like 32, 64 or 128). This way most of the layers in the model would not need to process all the 5000 patches and only the selected top-K. This drastically reduces the24161PCT Page 24 of 35computation cost but also affects the accuracy as the model only sees few patches to decide on a specific action prediction. Depending on the types of actions, the larger the value of K the better the accuracy but that also means more compute resources can be spent. Because the action detection task can also be affected by background noise, only selecting a few important patches may also help with accuracy compared to processing all the patches as it will help reduce the background noise in the frames.

[0092] In block 640, an alignment loss can be minimized to maximize the similarity between action text embeddings and the ROI aligned embeddings and obtain an optimized ML model for performing downstream tasks.

[0093] In an embodiment, during training, the cosine similarity can be computed and force the model to minimize cross entropy loss using the cosine similarity output as the predicted logits with the ground truth labels. In another embodiment, other similarity methods can be employed such as Euclidean distance, dot product, etc. The cross entropy loss indirectly forces the model to maximize the cosine similarity between the ROI aligned embeddings vectors and the ground truth text embedding vectors.

[0094] During inference, a threshold can be utilized (e.g., a value between 0 and 1 such as 0.7 or 0.8 based on heuristics) to check whether the ROI aligned embeddings matches with the action text embeddings.

[0095] In block 641, the action labels are encoded by the frozen text encoder to produce action text embeddings.

[0096] In an embodiment, the alignment loss can be a cosine similarity between the text and entity level embeddings. An example of the downstream tasks performed with the optimized ML model is shown in FIG. 7.24161PCT Page 25 of 35

[0097] Referring now to FIG. 7, a block diagram showing a practical application for enhanced decision making through few-shot adaptation of artificial intelligence models for action detection, in accordance with an embodiment of the present invention.

[0098] In an embodiment, an autonomous vehicle 710 utilizes the optimized ML model 118. The autonomous vehicle 710 can detect entities and entity actions within traffic scene 700 including: entity 705 running on the sidewalk, entity 740 walking with a stroller on the sidewalk on the left side of autonomous vehicle 710, entity 750 walking on the same side as entity 740, entity 730 stopped at the intersection due to the traffic light 743 being red in the direction of entity 730 and green in the direction of autonomous vehicle 710, entity 720 turning right, and traffic sign 741 showing a no entry sign. These detections assist the autonomous decision making ability of the autonomous vehicle 710.

[0099] Based on the entity and action detections, autonomous vehicle 710 can generate a trajectory to control its movement through traffic scene 700 and avoid undesirable outcomes such as collisions through autonomous decision making. For example, autonomous vehicle 710 can go straight because its path is cleared after entity 720 turned right and changed lanes.

[0100] In another embodiment, the autonomous vehicle 710 can generate optimized trajectories through a traffic scene based on preference such as shorter distance, gas / range saving routes, comfort, etc. In another embodiment, the autonomous vehicle 710 can generate a notification for a decision-making entity (e.g., driver, passenger, owner, etc.) when the detected action (e.g., other entity movement, etc.) satisfies a predetermined trigger condition (e.g., predicted collision, predicted heavy traffic, etc.), wherein the24161PCT Page 26 of 35notification supports the decision making entity’s situational judgment and decision making (e.g., recommendation to reroute the vehicle, etc.).

[0101] Reference in the specification to “one embodiment” or “an embodiment” of the present invention, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment”, as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment. However, it is to be appreciated that features of one or more embodiments can be combined given the teachings of the present invention provided herein.

[0102] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of’, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended for as many items listed.24161PCT Page 27 of 35

[0103] The foregoing is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the present invention and that those skilled in the art may implement various modifications without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention. Having thus described aspects of the invention, with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.24161PCT Page 28 of 35

Claims

WHAT IS CLAIMED IS:

1. A method, comprising:obtaining augmented feature maps by injecting feature augmentations of detected features from video frames into a feature space of a vision encoder of a machine learning model;training the vision encoder to learn low rank adaptation parameters based on the augmented feature maps;extracting entity level features from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters; andmaximizing similarity between the entity level features and action text embeddings obtained from the video frames by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.

2. The method of claim 1, wherein obtaining the augmented feature maps further comprises generating feature augmentations based on modulating parameters predicted by a parameter generator based on a learnable embedding table of the detected features.

3. The method of claim 1, wherein obtaining the augmented feature maps further comprises modulating the detected features from the video frames by performing feature-wise linear modulation.24161PCT Page 29 of 354. The method of claim 1, wherein training the vision encoder further comprises minimizing a training objective as a combination of loss functions with regularization terms derived from augmented samples.

5. The method of claim 4, wherein the loss functions include computing a weighted augmented loss that reflects reasonableness of an augmentation per sample.

6. The method of claim 1, wherein extracting the entity level features further comprises generating attention masks by processing video frames obtained by the vision encoder with frozen weights.

7. The method of claim 1, further comprising generating a notification for a decision-making entity when the detected action satisfies a predetermined trigger condition, wherein the notification supports situational judgment and decision making of the decision-making entity.

8. The method of claim 1, further comprising generating a trajectory to control an autonomous vehicle to avoid collisions responsive to detected entities and their corresponding detected actions within a traffic scene.

9. A system, comprising:a memory device;24161PCT Page 30 of 35one or more processor devices operatively coupled with the memory device to perform operations including:obtaining augmented feature maps by injecting feature augmentations of detected features from video frames into a feature space of a vision encoder of a machine learning model;training the vision encoder to learn low rank adaptation parameters based on the augmented feature maps;extracting entity level features from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters; and maximizing similarity between the entity level features and action text embeddings obtained from the video frames by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.

10. The system of claim 9, wherein obtaining the augmented feature maps further comprises generating feature augmentations based on modulating parameters predicted by a parameter generator based on a learnable embedding table of the detected features.

11. The system of claim 9, wherein obtaining the augmented feature maps further comprises modulating the detected features from the video frames by performing feature-wise linear modulation.24161PCT Page 31 of 3512. The system of claim 9, wherein training the vision encoder further comprises minimizing a training objective as a combination of loss functions with regularization terms derived from augmented samples.

13. The system of claim 12, wherein the loss functions include computing a weighted augmented loss that reflects reasonableness of an augmentation per sample.

14. The system of claim 9, wherein extracting the entity level features further comprises generating attention masks by processing video frames obtained by the vision encoder with frozen weights.

15. The system of claim 9, further comprising generating a notification for a decision-making entity when the detected action satisfies a predetermined trigger condition, wherein the notification supports situational judgment and decision making of the decision-making entity.

16. The system of claim 9, further comprising generating a trajectory to control an autonomous vehicle to avoid collisions responsive to detected entities and their corresponding detected actions within a traffic scene.

17. A non-transitory computer program product comprising a computer-readable storage medium including a program code, wherein the program code when executed on a computer causes the computer to perform operations including:24161PCT Page 32 of 35obtaining augmented feature maps by injecting feature augmentations of detected features from video frames into a feature space of a vision encoder of a machine learning model;training the vision encoder to learn low rank adaptation parameters based on the augmented feature maps;extracting entity level features from the video frames by performing region of interest alignment on the augmented feature maps obtained by the vision encoder injected with low rank adaptation (LoRA) parameters; andmaximizing similarity between the entity level features and action text embeddings obtained from the video frames by minimizing an alignment loss to train the machine learning model and obtain an optimized machine learning model for performing downstream tasks.

18. The non-transitory computer program product of claim 17, wherein obtaining the augmented feature maps further comprises generating feature augmentations based on modulating parameters predicted by a parameter generator based on a learnable embedding table of the detected features by performing feature-wise linear modulation.

19. The non-transitory computer program product of claim 17, wherein training the vision encoder further comprises minimizing a training objective as a combination of loss functions, including a weighted augmented loss that reflects24161PCT Page 33 of 35reasonableness of an augmentation per sample with regularization terms derived from the augmented samples.

20. The non-transitory computer program product of claim 17, further comprising generating a notification for a decision-making entity when the detected action satisfies a predetermined trigger condition, wherein the notification supports situational judgment and decision making of the decision-making entity.24161PCT Page 34 of 35