Visual recognition method and visual recognition system

Abstract

A visual recognition method and a visual recognition system. The method comprises: inputting a video sequence into a 3D feature acquisition model to acquire a plurality of temporal features; extracting at least one 2D image frame from the video sequence, and inputting the at least one 2D image frame into a 2D feature acquisition model to acquire a plurality of spatial features; inputting the plurality of temporal features and the plurality of spatial features into a first neural network architecture, wherein the first neural network architecture comprises a trunk network model for acquiring a first primary feature from the plurality of temporal features and acquiring a second primary feature from the plurality of spatial features, an action head model for executing an action recognition task on the basis of the first primary feature, and a detection head model for executing an object detection task on the basis of the second primary feature; and marking a specific action on the video sequence by means of the action head model, and marking a specific object on the video sequence by means of the detection head model.

Description

Visual recognition methods and visual recognition systems Technical Field

[0001] This application relates to image recognition, and more particularly to vision-based image recognition methods and systems. Background Technology

[0002] Currently, image recognition technology is often used on production lines to assist in monitoring whether products and processing behaviors are appropriate and correct, such as whether the processing behavior is located in the work area and whether the processing sequence is correct.

[0003] Currently, there are various algorithms on the market that can provide visual recognition functions, such as Microsoft's Florence, Apple's 4M, and NVIDIA's Hydra-MDP. However, these algorithms each have their own shortcomings when performing visual recognition, such as achieving multi-vision tasks that simultaneously perform classification and localization, video understanding that classifies behavioral relationships on a timeline, and the need for large amounts of computing resources that seriously affect efficiency. There is still room for improvement.

[0004] Furthermore, some algorithms, when performing visual recognition, first use a Convolutional Neural Network (CNN) to locate people in an image, and then feed the results into a Transformer to identify the people's behavior. However, such algorithms perform a two-stage operation, so their efficiency needs improvement. Moreover, currently available one-stage algorithms can only detect one of the two events: action events or object events; they cannot simultaneously recognize both. Summary of the Invention

[0005] This application discloses a visual recognition method and a visual recognition system that can simultaneously classify object types and behavior types, locate object positions and behavior positions, and take into account speed, GPU memory usage, and recognition accuracy.

[0006] In one embodiment, the visual recognition method of this application includes:

[0007] A video sequence based on time axis information is imported into a 3D feature acquisition model to extract multiple temporal features from the video sequence.

[0008] At least one 2D image frame is extracted from the image sequence, and the at least one 2D image frame is incorporated into a 2D feature acquisition model to obtain multiple spatial features from the at least one 2D image frame.

[0009] The multiple temporal features and multiple spatial features are simultaneously fed into a first neural network architecture for processing. The processing steps of the first neural network architecture include at least a trunk network model for obtaining a first primary feature for the multiple temporal features and a second primary feature for the multiple spatial features; an action head model for performing an action recognition task based on the first primary feature; and a detection head model for performing an object detection task based on the second primary feature.

[0010] The behavior head model marks a behavior information and a behavior position for a specific behavior on the image sequence, and the detection head model marks an object information and an object position for a specific object on at least one image frame of the image sequence.

[0011] In one embodiment, the visual recognition system of this application includes:

[0012] A camera captures images of a work area and generates a sequence of images based on timeline information;

[0013] A memory that stores one or more algorithm instructions; and

[0014] A processor, electrically connected to the camera and the memory, is configured to execute one or more algorithm instructions to perform the following actions and visually recognize the image sequence;

[0015] The image sequence is imported into a 3D feature acquisition model to extract multiple temporal features from the image sequence;

[0016] At least one 2D image frame is extracted from the image sequence, and the at least one 2D image frame is incorporated into a 2D feature acquisition model to obtain multiple spatial features from the at least one 2D image frame;

[0017] The multiple temporal features and multiple spatial features are simultaneously fed into a first neural network architecture for processing. The first neural network architecture includes at least a backbone network model for acquiring a first primary feature for the multiple temporal features and a second primary feature for the multiple spatial features; an action head model for performing an action recognition task based on the first primary feature; and a detection head model for performing an object detection task based on the second primary feature.

[0018] The behavior head model marks a behavior information and a behavior location on the image sequence for a specific behavior, and the detection head model marks an object information and an object location on the image sequence for a specific object.

[0019] Compared to related technologies, this application can simultaneously classify object types and behavior types, as well as locate object and behavior positions, through a one-stage computation. Furthermore, by using an improved, lightweight model, the computational resources required can be reduced. Attached Figure Description

[0020] Figure 1 is a block diagram of a specific embodiment of the visual recognition system of this application;

[0021] Figure 2 is a specific embodiment of the visual recognition architecture of this application;

[0022] Figure 3 is a specific embodiment of the visual recognition method of this application;

[0023] Figure 4 is a specific embodiment of the lightweight 3D ViT architecture of this application;

[0024] Figure 5 is a specific embodiment of the classification and positioning of objects and actions in this application.

[0025] Figure Label Explanation: 1: Visual Recognition System; 11: Processor; 12: Memory; 120: Visual Recognition Algorithm; 13: Camera; 2: Object; 3: Behavior; 4: Image Sequence; 41: Image Frame; 51: 3D Feature Acquisition Model; 52: 2D Feature Acquisition Model; 53: Channel Fusion Model; 54: Hydra-based First Neural Network Architecture; 541: Backbone Network Model; 542: Behavior Head Model; 543: Detection Head Model; 61: Temporal Feature; 62: Spatial Feature; 63: Image Representation; 64: Behavior Recognition Result; 65: Object Detection Result; S31~S40: Recognition Steps; 71: First Bounding Box; 72: Second Bounding Box; 73: Third Bounding Box; 8: Display Screen Detailed Implementation

[0026] This application discloses a visual recognition system, primarily applied on production lines, for visual recognition of images on the production line. Specifically, the visual recognition system of this application identifies information such as what objects (e.g., motherboards) exist in the work area, the location of the objects, what actions production line personnel are performing (e.g., inserting components or tightening screws), and the location of the actions through images. More specifically, the visual recognition system of this application can classify objects and actions in images, and simultaneously locate both objects and actions.

[0027] Please refer to Figure 1, which is a block diagram of a specific embodiment of the visual recognition system of this application. As shown in Figure 1, the visual recognition system 1 of this application mainly includes a processor 11, a memory 12, and at least one camera 13, wherein the processor 11 is electrically connected to the memory 12 and the camera 13. In one embodiment, the processor 11, the memory 12, and the camera 13 may be located in the same location. In another embodiment, the processor 11, the memory 12, and the camera 13 may be located in different locations, wherein the camera 13 is located on the production line and captures images of production line personnel and / or work areas to obtain a time-based video sequence. In other words, the visual recognition system 1 of this application does not perform recognition based on a single image, but processes n real-time images (i.e., a continuous sequence of images).

[0028] In one embodiment, the processor 11 may be implemented by a graphics processing unit (GPU), a central processing unit (CPU), a micro control unit (MCU), a programmable logic controller (PLC), a system on chip (SoC), or a field programmable gate array (FPGA).

[0029] In one embodiment, the memory 12 may be implemented by flash memory (FLASH), read-only memory (ROM), hard disk drive (HDD), solid state disk (SSD) or other storage components with storage capabilities.

[0030] In one embodiment, camera 13 may be implemented by a 2D camera or a 3D camera.

[0031] As shown in Figure 1, the visual recognition system 1 of this application uses a camera 13 to capture images containing all or part of the production line personnel and all the objects being processed. A visual recognition algorithm 120 performs visual recognition on the real-time images, thereby identifying the type of object 2, the position of object 2, the type of behavior 3, and the position of behavior 3 in the image (e.g., the type of object 2 / behavior 3 can be indicated on the display screen connected to the visual recognition system 1, or the position of object 2 / behavior 3 can be marked with a bounding box). Based on the output of the visual recognition system 1, production line managers can quickly determine whether the object 2 used by the production line personnel is correct, whether behavior 3 is correct, whether behavior 3 occurs within the work area, and whether the process follows the standard sequence, thereby monitoring production quality. For example, when a person is tightening a screw in an image, the visual recognition system 1 will generate a bounding box to enclose the screw and another bounding box to enclose the behavior area, simultaneously displaying that this behavior is tightening a screw.

[0032] As shown in Figure 1, the memory 12 of the visual recognition system 1 stores a visual recognition algorithm 120, which is mainly composed of one or more algorithm instructions. In this application, when the processor 11 reads the memory 12 and executes one or more algorithm instructions of the visual recognition algorithm 120, the visual recognition method of this application can be executed to perform visual recognition on the time-axis-based image sequence obtained by the camera 13 (details to follow).

[0033] Please also refer to Figure 2, which is a schematic diagram of a specific embodiment of the visual recognition architecture of this application. Figure 2 discloses the main recognition architecture executed by the visual recognition algorithm 120 of this application. As shown in Figure 2, in this application, the processor 11 of the visual recognition system 1 mainly imports the time-axis-based image sequence 4 (i.e., multiple real-time images) into the 3D feature acquisition model 51 based on 3D to obtain multiple temporal features 61 from the image sequence 4. At the same time, the processor 11 extracts at least one 2D image frame 41 from the image sequence 4 and imports the 2D image frame 41 into the 2D feature acquisition model 52 based on 2D to obtain multiple spatial features 62 from the 2D image frame 41.

[0034] In one embodiment, the 3D feature acquisition model 51 and the 2D feature acquisition model 52 can be implemented using existing state-of-the-art (SOTA) backbone models. In one embodiment, the 2D feature acquisition model 52 can be, for example, a ResNet model, a GooLeNet model, an EfficientNet model, or a MobileNet model, and the 3D feature acquisition model 51 can be, for example, a 3D convolutional neural network (CNN) model or a 3D vision transformer (ViT) model, but is not limited thereto.

[0035] Generally, image feature acquisition can be achieved using either a CNN architecture or a ViT architecture. CNN architecture is stronger at perceiving local regions of an image but weaker at perceiving the global context; conversely, ViT architecture, due to its self-attention mechanism, is stronger at perceiving the global context. As mentioned earlier, this application, in addition to object detection on image sequence 4, also performs action recognition. Action recognition emphasizes global perception, making a direct use of a CNN architecture for feature acquisition less suitable. However, the existing Transformer architecture has a much larger number of parameters than a typical CNN architecture, and the massive computational load makes its deployment cost prohibitively high for production lines. Furthermore, due to the high computational resource consumption, the current Transformer architecture can generally only acquire features from a single image and cannot process real-time images. Therefore, in one embodiment of this application, the visual recognition system 1 employs a lightweight 3D ViT architecture to implement the 3D feature acquisition model 51 for acquiring temporal features 61 (described in detail later).

[0036] In one embodiment, this application improves the architecture of Mobile ViT to establish the lightweight 3D ViT architecture (hereinafter referred to as the 3D ViT architecture). This 3D ViT architecture takes image sequence 4 (i.e., n real-time images) as input and modifies the original 2D convolutional layers in the Mobile ViT architecture to 3D convolutional layers (i.e., adding time (T) to the input tensor). After processing image sequence 4 through the 3D convolutional layers, the 3D ViT architecture can obtain image vectors of the original n images. Then, the n image vectors are simultaneously fed into the Transformer model for processing, thereby realizing the extraction of temporal features 61. It is worth mentioning that since this application converts 2D processing to 3D processing and expands the input tensor (i.e., adds time (T)), using the 3D ViT architecture of this application as the 3D feature acquisition model 51 is not faster than using the traditional ViT architecture as the 2D feature acquisition model 52. However, since the 3D ViT architecture of this application is a lightweight architecture (as shown in Figure 4), the number of parameters used is greatly reduced. Therefore, the visual recognition system 1 only needs to use a general consumer-grade graphics processing unit (GPU), such as Nvidia's A2000, to achieve real-time processing and obtain an accuracy similar to that of the standard ViT architecture.

[0037] The Transformer and Mobile ViT architectures mentioned above are known technologies in this field and will not be described in detail here.

[0038] Returning to the explanation of Figure 2, after extracting multiple temporal features 61 from image sequence 4 and multiple spatial features 62 from at least one image frame 41, processor 11 simultaneously feeds the multiple temporal features 61 and multiple spatial features 62 into a first neural network architecture 54 based on the Hydra model for processing. The Hydra model refers to an efficient multi-task learning framework developed by Tesla, which is one of the known technologies in this field and will not be described in detail here. This application modifies the Hydra model to produce a first neural network architecture 54 (described in detail later) that can process multiple temporal features 61 and multiple spatial features 62 separately, and generate behavior recognition results 64 and object detection results 65 respectively.

[0039] Specifically, as shown in Figure 2, the first neural network architecture 54 has at least a trunk network model 541, an action head model 542, and a detection head model 543. More specifically, the first neural network architecture 54 processes multiple temporal features 61 and multiple spatial features 62 respectively. Therefore, the first neural network architecture 54 may have two trunk network models 541 (e.g., a first trunk network model and a second trunk network model), wherein the first trunk network model is only responsible for obtaining the first primary features for the multiple temporal features 61, and the second trunk network model is only responsible for obtaining the second primary features for the multiple spatial features 62.

[0040] The first backbone network model is connected to the behavior head model 542 and outputs a first primary feature obtained from multiple temporal features 61 as input to the behavior head model 542. The behavior head model 542 performs a behavior recognition task based on the first primary feature, thereby ultimately labeling behavior information and behavior location for a specific behavior on image sequence 4 (i.e., outputting behavior recognition result 64). On the other hand, the second backbone network model is connected to the detection head model 542 and outputs a second primary feature obtained from multiple spatial features 62 as input to the detection head model 543. The detection head model 543 performs an object detection task based on the second primary feature, thereby ultimately labeling object information and object location for a specific object on image sequence 4 (i.e., outputting object detection result 65).

[0041] In this application, the first neural network architecture 54 processes multiple temporal features 61 and multiple spatial features 62 based on two backbone network models 541, one behavior head model 542, and one detection head model 543 to determine what objects are in the image sequence 4 (i.e., object classification), what behavior is being performed in the image sequence 4 (i.e., behavior classification), where the objects in the image sequence 4 are located (i.e., object localization), and where the behavior in the image sequence 4 occurs (i.e., behavior localization). By using the first neural network architecture 54, the visual recognition system 1 of this application can effectively achieve multi-vision tasks of simultaneously performing object and action classification and object and action localization.

[0042] In addition, if the lightweight 3D ViT architecture mentioned above is used when extracting temporal features 61, then in terms of implementation, only a consumer GPU is needed to achieve speed and accuracy comparable to other models, making it easier to realize the purpose of multi-vision tasks on industrial production lines.

[0043] In one embodiment, the architecture of the visual recognition system 1 of this application further includes a channel fusion model 53. In this embodiment, the processor 11 inputs multiple temporal features 61 acquired by the 3D feature acquisition model 51 and the output of the 2D feature acquisition model 52 into the channel fusion model 53 to integrate the features of different channels and generate an image representation 63 of the image sequence 4. In this embodiment, the processor 11 inputs the image representation 63 and multiple spatial features 62 into the first neural network 54 for processing, so as to serve as the inputs of the two backbone network models 541 respectively.

[0044] As described above, the behavior model 542 of the first neural network architecture 54 labels specific behaviors in the image sequence 4 based on a first primary feature, and the detection head model 543 labels specific objects in the image sequence 4 based on a second primary feature. In one embodiment, the behavior head model 542 can generate a first bounding box (as shown in FIG. 5) surrounding the location of the specific behavior based on behavior information and behavior location, and use the first bounding box as part of the behavior recognition result 64. Similarly, the detection head model 543 can generate a second bounding box (as shown in FIG. 5) surrounding the specific object based on object information and object location, and use the second bounding box as part of the object detection result 65.

[0045] Please also refer to Figure 5, which is a specific embodiment of the classification and positioning of objects and actions according to this application. As shown in Figure 5, when recognizing the image sequence 4 acquired by the camera 13, the visual recognition system 1 can simultaneously display the recognition results (including behavior recognition results 64 and object detection results 65) on the display screen 8 connected to the visual recognition system 1. As shown in Figure 5, the image sequence 4 captured by the camera 13 contains images of people and objects. After effective recognition, the visual recognition system 1 generates a first bounding box 71 surrounding the first object (e.g., motherboard), a second bounding box 72 surrounding the second object (e.g., power supply), and a third bounding box 73 containing the location where the first action (e.g., locking) occurs.

[0046] As shown in Figure 5, on one side of the first bounding box 71, the visual recognition system 1 displays the name of the first object (e.g., motherboard) and a confidence value (e.g., 0.97). On one side of the second bounding box 72, the visual recognition system 1 displays the name of the second object (e.g., power circuit board) and a confidence value (e.g., 0.95). On one side of the display screen 8, the visual recognition system 1 displays the content of the first action (e.g., locking). By displaying the recognition results on the display screen 8, the visual recognition system 1 of this application can help production line managers quickly confirm the current status of the production line and detect errors in real time.

[0047] Please refer to Figure 3 for a specific embodiment of the visual recognition method of this application. Figure 3 discloses the specific execution steps of the visual recognition method of this application, and this visual recognition method is mainly applicable to the visual recognition system 1 shown in Figure 1. Specifically, after the visual recognition system 1 obtains the image sequence 4 through the camera 13 and the processor 11 executes the visual recognition algorithm 120, the following visual recognition method can be implemented.

[0048] As shown in Figure 3, firstly, the processor 11 inputs the image sequence 4 acquired by the camera 13 into the 3D feature acquisition model 51 (step S31) to extract multiple temporal features from the image sequence 4 (step S32). As mentioned above, this application simultaneously inputs the image sequence 4 based on a continuous time axis (i.e., n real-time images) into the 3D feature acquisition model 51 to extract multiple temporal features 61. In other words, the 3D feature acquisition model 51 does not perform frame-based feature extraction. More specifically, this application performs feature acquisition on n images at once to obtain n consecutive features on the time axis (i.e., temporal features 61), and then inputs these n consecutive features on the time axis into the first neural network architecture 54 for processing to obtain the final behavior recognition result 64. Therefore, this application differs from related technologies that directly input a single feature from a single image into the model for processing.

[0049] On the other hand, the processor 11 also extracts at least one 2D image frame 41 from the image sequence 4 (step S33) and imports at least one 2D image frame 41 into the 2D feature acquisition model 52 (step S34) to acquire multiple spatial features 62 from the 2D image frame 41 (step S35).

[0050] Next, the processor 11 simultaneously incorporates multiple temporal features 61 and multiple spatial features 62 into a first neural network architecture 54 based on the Hydra model for processing (step S36). In one embodiment, before step S36, the processor 11 first incorporates the multiple temporal features 61 and the output of the 2D feature acquisition model 52 into the channel fusion model 53 to generate the image representation 63 of the image sequence 4. Furthermore, in step S36, the processor 11 simultaneously incorporates the image representation 63 of the image sequence 4 and the multiple spatial features 62 into the first neural network architecture 54, but this is not a limitation.

[0051] As mentioned above, the first neural network architecture 54 based on the Hydra model in this application has at least two backbone network models 541, one behavior head model 542, and one detection head model 543. Since this application needs to simultaneously detect the occurrence positions of behaviors and objects in image sequence 4, if a model that simultaneously detects behaviors and objects is trained, the models may interfere with each other during the inference process, thereby affecting the accuracy. Therefore, this application, referring to the Hydra model, separates the computational paths of behavior recognition (using one of the two backbone network models 541 and the behavior head model 542) from those of object detection (using the other of the two backbone network models 541 and the detection head model 543) to improve the accuracy of the inference results.

[0052] As shown in Figure 3, after step S36, the processor 11 obtains a first primary feature through one of the two backbone network models of the first neural network architecture 54, which consists of multiple temporal features 61 (or image representations 63) (step S37), and then performs a behavior recognition task based on the first primary feature through the behavior head model 542 to label behavior information and behavior location for specific behaviors on the image sequence 4 (step S38). Furthermore, after step S36, the processor 11 simultaneously obtains a second primary feature through the other of the two backbone network models of the first neural network architecture 54, which consists of multiple spatial features 62 (step S39), and then performs an object detection task based on the second primary feature through the detection head model 543 to label object information and object location for specific objects on the image sequence 4 (step S40).

[0053] In one embodiment, in step S38, the processor 11 further generates a first bounding box surrounding the location where a specific behavior occurs based on the behavior information and the behavior location using the behavior head model 542. Furthermore, in step S40, the processor 11 further generates a second bounding box surrounding a specific object based on the object information and the object location using the object detection model 543.

[0054] In this application, the processor 11 uses a first neural network architecture 54 to extract the main features of multiple temporal features 61 and multiple spatial features 62, perform feature convolution, and perform multi-task prediction, thereby simultaneously completing efficient inference for behavior recognition and object detection.

[0055] In one embodiment, the backbone network model 541 employs 1x1 convolutional layers to perform preliminary feature extraction and dimensionality reduction on multiple temporal features 61 (or image representations 63, which will be described below as multiple temporal features 61) and multiple spatial features 62.

[0056] For the backbone network model 541 (such as the first backbone network model) connected to the behavior head model 542, its input tensor is [B, C1, H, W], where B is the batch size (i.e., the number of samples (i.e., images) input into the model at one time), C1 is the number of channels before dimensionality reduction (i.e., the number of input channels of the first neural network architecture 65), H is the height of the image, and W is the width of the image. For example, if the batch size is 1, the number of channels output by the 3D feature acquisition model 51 or the channel fusion model 53 is 256, and the height and width of the n images in the image sequence 4 are 28, then the input tensor of the first backbone network model is [1, 256, 28, 28].

[0057] As mentioned above, the first backbone network model is mainly used to reduce the dimensionality of multiple input temporal features 61, and the degree of dimensionality reduction is: num_anchors*(B+num_ad_classes+4). Here, num_anchors is the number of anchor boxes preset by the first neural network architecture 54, and num_ad_classes is the number of behavior categories to be identified. Taking an example with 3 anchor boxes, 1 batch size, and 14 behavior categories, 3*(1+14+4)=57. Therefore, the first backbone network model will convolve the input tensor [1,256,28,28] into an output tensor [1,57,28,28]. However, the above is only a specific implementation example of this application, and is not limited to it.

[0058] It is worth noting that the aforementioned anchor boxes refer to predefined bounding boxes of fixed size and shape, typically evenly distributed across the image, used as reference boxes to predict the true position / size of objects / behaviors. The more anchor boxes there are, the greater the likelihood that the model will detect objects / behaviors of different sizes and shapes, but this also increases the model's computational load and complexity. The anchor boxes described are known in this technical field and will not be elaborated upon further.

[0059] Similar to the first backbone network model, the input tensor of the backbone network model 541 (such as the second backbone network model) connected to the detection head model 543 is also [B,C1,H,W]. In other words, the output tensors of the 3D feature acquisition model 51, the channel fusion model 53, and the 2D feature acquisition model 52 in this application are the same. In one embodiment, if the batch size is 1, the number of channels output by the 2D feature acquisition model 52 is 256, and the height and width of the image frame 41 are 28, then the input tensor of the second backbone network model is [1,256,28,28].

[0060] Similar to the first backbone network model, the second backbone network model is used to reduce the dimensionality of the multiple spatial features 62 in the input, and its dimensionality reduction is also: num_anchors*(B+num_ad_classes+4). Here, num_anchors is the number of anchor boxes, and num_ad_classes is the number of object categories to be identified. Taking an example with 3 anchor boxes, 1 batch size, and 14 object categories, 3*(1+14+4)=57. Therefore, the second backbone network model will convolve the input tensor [1,256,28,28] into an output tensor [1,57,28,28].

[0061] The behavior head model 542 receives multiple temporal features 61 after dimensionality reduction from the first backbone network model, and performs a behavior recognition task on the multiple temporal features 61 to output the first confidence level, first classification probability, and regression parameters of the first bounding box for a specific behavior in the image sequence 4. Specifically, the input tensor of the behavior head model 542 is the output tensor of the first backbone network model, which is [B,C2,H,W], where C2 is the number of channels after dimensionality reduction (57 in the above embodiment).

[0062] The purpose of the behavior head model 542 in calculating the first confidence score is to predict whether each anchor contains a valid action and output a value between [0,1], activated using a sigmoid function. When calculating the first confidence score, the input tensor of the behavior head model 542 is [B,C3,H,W], and the output shape is [B,M,1]. Here, C3 represents the first K channels of the reduced-dimensionality channel count (e.g., 57), K is the number of anchors, M is the total number of anchors predicted by the model, and M = H x W x K, where 1 corresponds to the confidence value of each anchor.

[0063] For example, if the number of anchors K is 3, the batch size B is 4, the image height H is 28, and the image width W is 28, then the input tensor of the behavior head model 542 when calculating the first confidence level is [4,3,28,28], M = 28x28x3 = 2352, so the output shape is [4,2352,1]. However, the above is only one specific embodiment example of this application, and is not limited thereto.

[0064] The purpose of the behavior head model 542 in calculating the first classification probability is to predict the probability distribution of the action category for each anchor and to activate it using softmax or sigmoid. When calculating the first classification probability, the input tensor of the behavior head model 542 is [B, C4, H, W], and the output shape is [B, M, AC]. Here, C4 represents the K*AC channels after C3, K is the number of anchors, M is the total number of anchors predicted by the model and M = H x W x K, and AC is the number of behavior categories to be identified.

[0065] For example, if the number of anchors K is 3, the batch size B is 4, the image height H is 28, the image width W is 28, and the number of behavior categories AC to be identified is 14, then the input tensor of the behavior head model 542 when calculating the first classification probability is [4, 42, 28, 28], M = 28 x 28 x 3 = 2352, so the output shape is [4, 2352, 14]. However, the above is only one specific embodiment example of this application, and is not limited thereto.

[0066] The purpose of the behavior head model 542 in calculating the regression parameters of the first bounding box is to predict the regression parameters of the bounding box at the location where the wrapping action occurs for each anchor. These regression parameters include four coordinates (tx, ty, tw, th), representing the center coordinates and width and height offsets of the bounding box, respectively. When calculating the regression parameters of the first bounding box, the input tensor of the behavior head model 542 is [B, C5, H, W], and the output shape is [B, M, 4]. Here, C5 represents the last K * four coordinate channels after C4, K is the number of anchors, and M is the total number of anchors predicted by the model, where M = H x W x K.

[0067] For example, if the number of anchors K is 3, the batch size B is 4, the image height H is 28, and the image width W is 28, then the input tensor of the behavior head model 542 when calculating the regression parameters of the first bounding box is [4, 12, 28, 28], M = 28 x 28 x 3 = 2352, so the output shape is [4, 2352, 4]. However, the above is only one specific embodiment example of this application, and is not limited thereto.

[0068] One of the technical features of this application is that the first neural network architecture 54 adopts a one-stage operation method, in which the behavior head model 542 and the detection head model 543 perform object and behavior classification and object and behavior localization simultaneously, instead of using one Transformer for localization and then another Transformer for classification. Therefore, when the behavior head model 542 is activated, the detection head model 543 is also activated simultaneously.

[0069] Specifically, the detection head model 543 receives multiple spatial features 62 after dimensionality reduction from the second backbone network model, and performs object detection on the multiple spatial features 62 to output the second confidence level, second classification probability, and regression parameters of the second bounding box for a specific object in the image sequence 4. Similar to the behavior head model 542 described above, the input tensor of the detection head model 543 is the output tensor of the second backbone network model, which is [B,C2,H,W], where C2 is the number of channels after dimensionality reduction (57 in the above embodiment).

[0070] The purpose of the detection head model 543 in calculating the second confidence score is to predict whether each anchor contains a valid object and output a value between [0,1], activated using a sigmoid function. When calculating the second confidence score, the input tensor of the detection head model 543 is [B,C6,H,W], and the output shape is [B,M,1]. Here, C6 represents the first K channels of the reduced-dimensionality channel count (e.g., 57), K is the number of anchors, M is the total number of anchors predicted by the model, and M = H x W x K, where 1 corresponds to the confidence value of each anchor.

[0071] For example, if the number of anchors K is 3, the number of batches B is 4, the height H of image frame 41 is 28, and the width W is 28, then the input tensor of the head detection model 543 when calculating the second confidence level is [4,3,28,28], M=28x28x3=2352, so the output shape is [4,2352,1].

[0072] The purpose of the detection head model 543 in calculating the second classification probability is to predict the probability distribution of object categories for each anchor and to activate it using softmax or sigmoid. When calculating the second classification probability, the input tensor of the detection head model 543 is [B, C7, H, W], and the output shape is [B, M, OC]. Here, C7 represents the K*OC channels after C6, K is the number of anchors, M is the total number of anchors predicted by the model and M = H x W x K, and OC is the number of object categories to be detected.

[0073] For example, if the number of anchors K is 3, the number of batches B is 4, the height H of image frame 41 is 28, the width W is 28, and the number of object categories OC to be detected is 14, then the input tensor of the head detection model 543 when calculating the second classification probability is [4,42,28,28], M=28x28x3=2352, so the output shape is [4,2352,14].

[0074] The purpose of the detection head model 543 in calculating the regression parameters of the second bounding box is to predict the regression parameters of the bounding box of the object being covered for each anchor. These regression parameters include four coordinates (tx, ty, tw, th), representing the center coordinates and width and height offsets of the bounding box, respectively. When calculating the regression parameters of the second bounding box, the input tensor of the detection head model 543 is [B, C8, H, W], and the output shape is [B, M, 4]. Here, C8 represents the last K * four coordinate channels after C7, K is the number of anchors, and M is the total number of anchors predicted by the model, where M = H x W x K.

[0075] For example, if the number of anchors K is 3, the number of batches B is 4, the height H of image frame 41 is 28, and the width W is 28, then the input tensor of the head detection model 543 when calculating the regression parameters of the second bounding box is [4,12,28,28], M=28x28x3=2352, so the output shape is [4,2352,4].

[0076] The first neural network architecture 54 of this application calculates the first confidence level, first classification probability, and regression parameters of the first bounding box for the identified behavior using the behavior head model 542, and the second confidence level, second classification probability, and regression parameters of the second bounding box for the detected object using the detection head model 543. The architecture can then output behavior recognition result 64 and object detection result 65, respectively. In one embodiment, the behavior recognition result 64 includes marking the location of the behavior in the image sequence 4 using the first bounding box and describing the content of the behavior; the object detection result 65 includes marking the location of the object in the image sequence 4 using the second bounding box and describing the content of the object.

[0077] As mentioned above, the visual recognition system 1 of this case can use an existing SOTA backbone model as the 3D feature acquisition model 51 to obtain multiple temporal features 61 from the image sequence 4. However, this approach suffers from the problem of excessively large parameters of the Transformer, which consumes computational resources. Therefore, in one embodiment, the visual recognition system 1 can use a lightweight 3D Vit architecture to implement the 3D feature acquisition model 51.

[0078] Please also refer to Figure 4, which is a specific embodiment of the lightweight 3D ViT architecture of this application. As mentioned above, the 3D ViT architecture used in this application is derived from a modification of the mobileViT architecture. The main difference is that mobile ViT is extended to process 3D data (i.e., time-axis-based image sequences), increasing the processing capability of the time dimension, making ViT suitable for processing dynamic data. Furthermore, 3D convolutions (including standard 3D convolutions and depthwise separable 3D convolutions) are imported into ViT, and all modules (including inverse residual modules and self-attention modules) are extended to 3D space.

[0079] As shown in Figure 4, the input tensor of the 3D ViT architecture of this application is [B,C,T,H,W], where B is the batch size, C is the number of channels, T is the time axis (i.e., the number of frames), H is the height of the n images contained in image sequence 4, and W is the width of the n images contained in image sequence 4. In the embodiment of Figure 4, the 3D ViT architecture adopts a three-layer structure, which sequentially includes a 3D convolutional layer (3DConv), a first 3D inverted residual layer and a first 3D lightweight vision transformer block (MobileViT) layer (the first layer), a second 3D inverted residual layer and a second 3D MobileViT block layer (the second layer), and a third 3D inverted residual layer and a third 3D MobileViT block layer (the third layer). In other embodiments with sufficient hardware performance, the above-mentioned number of layers of the 3D ViT architecture can be stacked infinitely, without being limited to three layers.

[0080] In step S31 of Figure 3, which involves obtaining multiple temporal features 61 using a 3D ViT architecture, the 3D ViT architecture mainly performs the following actions in sequence: (1) preliminary feature extraction and dimensionality reduction of the input image sequence 4 by a 3D convolutional layer; (2) local feature extraction and dimensionality reduction of the output of the 3D convolutional layer by a first 3D inverse residual layer; (3) local and global modeling by a first 3D MobileViT block layer based on the output of the first 3D inverse residual layer; (4) local feature extraction and dimensionality reduction of the output of the first 3D MobileViT block layer by a second 3D inverse residual layer; (5) local and global modeling by a second 3D MobileViT block layer based on the output of the second 3D inverse residual layer; (6) local feature extraction and dimensionality reduction of the output of the second 3D MobileViT block layer by a third 3D inverse residual layer; and (7) final global modeling by a third 3D MobileViT block layer based on the output of the second 3D inverse residual layer. Finally, the output of the third 3D Mobile ViT block layer is a combination of multiple temporal features 61 based on image sequence 4.

[0081] Specifically, the 3D ViT architecture of this application uses 3D convolution and 3D inverse residual to extract features from image sequence 4 to obtain local features, and performs global feature extraction on the time scale through 3D MobileViT blocks to obtain temporal features.

[0082] In one embodiment, the 3D convolutional layer is a 3x3 convolutional layer.

[0083] In one embodiment, the first, second, and third 3D inverse residual layers each include an expansion model, a depthwise convolution model, a reduction model, and a residual connection model. The expansion model primarily increases the number of channels; for example, it can be a 1x1x1 convolution model. The depthwise convolution model primarily extracts local features; for example, it can be a 3x3x3 depthwise convolution model. The reduction model primarily compresses the number of channels; for example, it can be a 1x1x1 convolution model. The residual connection model adds the input and output to perform residual processing.

[0084] In one embodiment, the first, second, and third 3D MobileViT block layers each include a linear self-attention model for calculating global attention and a linear attention feedforward network (Linear Attn FFN) model for extracting high-level features. The self-attention model and the attention feedforward network model are known technologies in this field and will not be described further here.

[0085] As mentioned above, the 3D ViT architecture used in this case is an improvement on the mobileViT architecture, thus retaining the advantages of combining CNN and Transformer. In one embodiment, the first 3D MobileViT block layer, the second 3D MobileViT block layer, and the third 3D MobileViT block layer each include a local representation model, a global representation model, and a projection model, respectively.

[0086] Local representation models capture local spatial information in an image through convolutional layers, such as a 3x3x3 depthwise convolutional model plus a 1x1x1 convolutional model. Global representation models are implemented using Transformers to capture global relationships in an image through a self-attention mechanism, such as a multi-layer linear attention feedforward network model. Mapping models project the data processed by the Transformer onto the shape of the original image, that is, map the number of channels back to the input dimension, such as a 1x1x1 convolutional model.

[0087] In one embodiment, the linear attention feedforward network model is a 3D linear attention feedforward network (Linear Attn FFN 3D) model, which includes a first pre-normalization model, a self-attention model, a second pre-normalization model, and a feedforward network (FFN) model.

[0088] The first pre-normalization model runs on the attention path, used to normalize the input and model global relationships. For example, it can be implemented using Group Normalization and Linear Self-Attention 3D models. The self-attention model is used to compute global attention, for example, using Q(Query), V(Value), and K(Key) projections combined with an attention mechanism. The second pre-normalization model runs on the feedforward network path, used to normalize the input in preparation for entering the feedforward network. For example, it can be implemented using a Group Normalization model. The feedforward network model is used to extract high-level features, for example, using a 1x1x1 convolution to increase or decrease the dimensionality of the channels. The Group Normalization model, Q, V, K projections, and attention mechanisms are known techniques in this field and will not be elaborated further.

[0089] In summary, the visual recognition system and method of this application can simultaneously perform multi-vision tasks such as object and action classification and object and action localization. Furthermore, if a lightweight 3D ViT architecture is used for temporal feature extraction, it can achieve efficiency and accuracy comparable to other large-scale models using consumer GPUs.

Claims

1. A visual recognition method, comprising: Image sequences based on time axis information are imported into a 3D feature acquisition model to extract multiple temporal features from the image sequences; At least one 2D image frame is extracted from the image sequence, and the at least one 2D image frame is incorporated into a 2D feature acquisition model to obtain multiple spatial features from the at least one 2D image frame; The multiple temporal features and the multiple spatial features are simultaneously fed into a first neural network architecture for processing. The processing steps of the first neural network architecture include at least a backbone network model for obtaining a first primary feature for the multiple temporal features and a second primary feature for the multiple spatial features, a behavior head model for performing a behavior recognition task based on the first primary feature, and a detection head model for performing an object detection task based on the second primary feature. as well as The behavior head model marks behavior information and behavior location on the image sequence for specific behaviors, and the detection head model marks object information and object location on the image sequence for specific objects.

2. The visual recognition method according to claim 1, wherein before the step of simultaneously incorporating the plurality of temporal features and the plurality of spatial features into the first neural network architecture for processing, it further includes: The output of the multiple temporal features and the 2D feature acquisition model is fed into the channel fusion model to generate an image representation of the image sequence. In the step of simultaneously incorporating the multiple temporal features and the multiple spatial features into the first neural network architecture for processing, the first neural network architecture receives the image representation and the multiple spatial features as input.

3. The visual recognition method according to claim 1, further comprising: The behavior head model generates a first bounding box surrounding the location where the specific behavior occurs, based on the behavior information and the behavior location. and The detection head model generates a second bounding box surrounding the specific object based on the object information and the object position.

4. The visual recognition method according to claim 1, wherein the 3D feature acquisition model comprises a 3D convolutional layer, a first 3D inverse residual layer, a first 3D MobileViT block layer, a second 3D inverse residual layer, a second 3D MobileViT block layer, a third 3D inverse residual layer, and a third 3D MobileViT block layer, wherein the step of acquiring the plurality of temporal features from the image sequence includes: The 3D convolutional layer performs preliminary feature extraction and dimensionality reduction on the image sequence; The first 3D inverse residual layer performs local feature extraction and dimensionality reduction on the output of the 3D convolutional layer. The first 3D MobileViT block layer performs local and global modeling based on the output of the first 3D inverse residual layer; The second 3D inverse residual layer performs local feature extraction and dimensionality reduction on the output of the first 3D MobileViT block layer; The second 3D MobileViT block layer performs local and global modeling based on the output of the second 3D inverse residual layer; The third 3D inverse residual layer performs local feature extraction and dimensionality reduction on the output of the second 3D MobileViT block layer; and The third 3D MobileViT block layer performs the final global modeling based on the output of the second 3D inverse residual layer.

5. The visual recognition method according to claim 4, wherein the input tensor of the 3D feature acquisition model is [B,C,T,H,W], where B is the batch number, C is the number of channels, T is time, H is the image height, and W is the image width.

6. The visual recognition method according to claim 4, wherein the 3D convolutional layer is a 3x3 3D ​​convolutional layer; the first 3D inverse residual layer, the second 3D inverse residual layer and the third 3D inverse residual layer respectively include an expansion model for increasing the number of channels, a depth convolutional model for extracting local features, a restoration model for compressing the number of channels, and a residual connection model for adding the input and output to perform residual calculation; the first 3D MobileViT block layer, the second 3D MobileViT block layer and the third 3D MobileViT block layer respectively include a linear self-attention model for calculating global attention and a linear attention feedforward network (Linear Attn FFN) model for extracting high-level features.

7. The visual recognition method according to claim 5, wherein the backbone network model of the first neural network architecture is a 1x1 convolutional layer, which performs preliminary feature extraction and dimensionality reduction on the plurality of temporal features and the plurality of spatial features respectively, wherein the input tensor of the first neural network architecture is [B,C1,H,W], where B is the batch size, C1 is the number of channels before dimensionality reduction, H is the image height, and W is the image width, and the dimensionality reduction degree of the backbone network model is: num_anchors*(B+num_ad_classes+4), where num_anchors is the number of anchor boxes preset by the first neural network architecture, and num_ad_classes is the number of object categories to be recognized or the number of behavior categories to be recognized.

8. The visual recognition method according to claim 7, wherein the output tensor of the backbone network model is [B,C2,H,W], C2 is the number of channels after dimensionality reduction, and the behavior head model performs the behavior recognition task on the plurality of temporal features to output the first confidence level, the first classification probability, and the regression parameters of the first bounding box of the specific behavior, wherein: The input tensor of the behavior head model when calculating the first confidence score is [B,C3,H,W] and the output shape is [B,M,1], where C3 is the first K channels after dimensionality reduction, K is the number of Anchors, M is the total number of predicted Anchors and M = HxWxK, and 1 corresponds to the confidence value of each Anchor. The input tensor of the behavior head model when calculating the first classification probability is [B,C4,H,W] and the output shape is [B,M,AC], where C4 is K*AC channels after C3, and AC is the number of behavior categories to be identified. The input tensor for the behavior head model to calculate the regression parameters of the first bounding box is [B,C5,H,W] and the output shape is [B,M,4], where C5 is the last K*4 coordinate channels after C4, and the four coordinates are (tx,ty,tw,th), where the four coordinates represent the center coordinates and the offset values ​​of the width and height of the first bounding box, respectively.

9. The visual recognition method according to claim 8, wherein the output tensor of the backbone network model is [B, C2, H, W], C2 is the number of channels after dimensionality reduction, and the detection head model performs the object recognition task on the plurality of spatial features to output the second confidence level, the second classification probability, and the regression parameters of the second bounding box of the specific object, wherein: The input tensor of the detection head model when calculating the second confidence value is [B,C6,H,W] and the output shape is [B,M,1], where C6 is the first K channels after dimensionality reduction, K is the number of Anchors, M is the total number of predicted Anchors and M = HxWxK, and 1 corresponds to the confidence value of each Anchor. The input tensor for the detection head model to calculate the second classification probability is [B,C7,H,W] and the output shape is [B,M,OC], where C7 is the K*OC channels after C6, and OC is the number of object categories to be identified; The input tensor for the head detection model to calculate the regression parameters of the second bounding box is [B,C8,H,W] and the output shape is [B,M,4]. C8 is the last K*4-coordinate channels after C7, and the four coordinates are (tx,ty,tw,th), where the four coordinates represent the center coordinates and the offset values ​​of width and height of the second bounding box, respectively.

10. A visual recognition system, comprising: A camera captures images of the work area and generates a sequence of images based on timeline information; Memory, which stores one or more algorithm instructions; and A processor, electrically connected to the camera and the memory, is configured to execute one or more algorithm instructions to perform the following actions and visually recognize the image sequence; The image sequence is imported into a 3D feature acquisition model to obtain multiple temporal features from the image sequence; At least one 2D image frame is extracted from the image sequence, and the at least one 2D image frame is incorporated into a 2D feature acquisition model to obtain multiple spatial features from the at least one 2D image frame; The multiple temporal features and the multiple spatial features are simultaneously fed into a first neural network architecture for processing. The processing steps of the first neural network architecture include at least a backbone network model for obtaining a first primary feature for the multiple temporal features and a second primary feature for the multiple spatial features, a behavior head model for performing a behavior recognition task based on the first primary feature, and a detection head model for performing an object detection task based on the second primary feature. and The behavior head model marks behavior information and behavior location on the image sequence for specific behaviors, and the detection head model marks object information and object location on the image sequence for specific objects.

11. The visual recognition system according to claim 10, wherein before simultaneously incorporating the plurality of temporal features and the plurality of spatial features into the first neural network architecture for processing, the processor further performs the following actions: The output of the multiple temporal features and the 2D feature acquisition model is fed into the channel fusion model to generate an image representation of the image sequence. In the action of simultaneously incorporating the multiple temporal features and the multiple spatial features into the first neural network architecture for processing, the first neural network architecture receives the image representation and the multiple spatial features as input.

12. The visual recognition system according to claim 10, wherein the processor further performs the following actions: The behavior head model generates a first bounding box surrounding the location where the specific behavior occurs, based on the behavior information and the behavior location; and The detection head model generates a second bounding box surrounding the specific object based on the object information and the object position.

13. The visual recognition system according to claim 10, wherein the 3D feature acquisition model comprises a 3D convolutional layer, a first 3D inverse residual layer, a first 3D MobileViT block layer, a second 3D inverse residual layer, a second 3D MobileViT block layer, a third 3D inverse residual layer, and a third 3D MobileViT block layer, wherein the action of acquiring the plurality of temporal features from the image sequence includes: The 3D convolutional layer performs preliminary feature extraction and dimensionality reduction on the image sequence; The first 3D inverse residual layer performs local feature extraction and dimensionality reduction on the output of the 3D convolutional layer. The first 3D MobileViT block layer performs local and global modeling based on the output of the first 3D inverse residual layer; The second 3D inverse residual layer performs local feature extraction and dimensionality reduction on the output of the first 3D MobileViT block layer; The second 3D MobileViT block layer performs local and global modeling based on the output of the second 3D inverse residual layer; The third 3D inverse residual layer performs local feature extraction and dimensionality reduction on the output of the second 3D MobileViT block layer; and The third 3D MobileViT block layer performs the final global modeling based on the output of the second 3D inverse residual layer.

14. The visual recognition system according to claim 13, wherein the input tensor of the 3D feature acquisition model is [B,C,T,H,W], where B is the batch number, C is the number of channels, T is time, H is the image height, and W is the image width.

15. The visual recognition system according to claim 13, wherein the 3D convolutional layer is a 3x3 3D ​​convolutional layer; the first 3D inverse residual layer, the second 3D inverse residual layer, and the third 3D inverse residual layer respectively include an expansion model for increasing the number of channels, a deep convolutional model for extracting local features, a reduction model for compressing the number of channels, and a residual connection model for adding the input and output to perform residual calculation; the first 3D MobileViT block layer, the second 3D MobileViT block layer, and the third 3D MobileViT block layer respectively include a linear self-attention model for calculating global attention and a linear attention feedforward network model for extracting high-level features.

16. The visual recognition system according to claim 14, wherein the backbone network model of the first neural network architecture is a 1x1 convolutional layer, which performs preliminary feature extraction and dimensionality reduction on the plurality of temporal features and the plurality of spatial features respectively, wherein the input tensor of the first neural network architecture is [B,C1,H,W], where B is the batch size, C1 is the number of channels before dimensionality reduction, H is the image height, and W is the image width, and the number of channels of the backbone network model after dimensionality reduction is: num_anchors*(B+num_ad_classes+4), where num_anchors is the number of anchors preset by the first neural network architecture, and num_ad_classes is the number of object categories to be recognized or the number of behavior categories to be recognized.

17. The visual recognition system according to claim 16, wherein the output tensor of the backbone network model is [B, C2, H, W], C2 is the number of channels after dimensionality reduction, and the behavior head model performs the behavior recognition task on the plurality of temporal features to output the first confidence level, the first classification probability, and the regression parameters of the first bounding box of the specific behavior, wherein: The input tensor of the behavior head model when calculating the first confidence score is [B,C3,H,W] and the output shape is [B,M,1], where C3 is the first K channels after dimensionality reduction, K is the number of Anchors, M is the total number of predicted Anchors and M = HxWxK, and 1 corresponds to the confidence value of each Anchor. The input tensor of the behavior head model when calculating the first classification probability is [B,C4,H,W] and the output shape is [B,M,AC], where C4 is K*AC channels after C3, and AC is the number of behavior categories to be identified. The input tensor for the behavior head model to calculate the regression parameters of the first bounding box is [B,C5,H,W] and the output shape is [B,M,4], where C5 is the last K*4 coordinate channels after C4, and the four coordinates are (tx,ty,tw,th), where the four coordinates represent the center coordinates and the offset values ​​of the width and height of the first bounding box, respectively.

18. The visual recognition system according to claim 17, wherein the output tensor of the backbone network model is [B, C2, H, W], C2 is the number of channels after dimensionality reduction, and the detection head model performs the object recognition task on the plurality of spatial features to output the second confidence level, the second classification probability, and the regression parameters of the second bounding box of the specific object, wherein: The input tensor of the detection head model when calculating the second confidence value is [B,C6,H,W] and the output shape is [B,M,1], where C6 is the first K channels after dimensionality reduction, K is the number of Anchors, M is the total number of predicted Anchors and M = HxWxK, and 1 corresponds to the confidence value of each Anchor. The input tensor for the detection head model to calculate the second classification probability is [B,C7,H,W] and the output shape is [B,M,OC], where C7 is the K*OC channels after C6, and OC is the number of object categories to be identified; The input tensor for the head detection model to calculate the regression parameters of the second bounding box is [B,C8,H,W] and the output shape is [B,M,4]. C8 is the last K*4-coordinate channels after C7, and the four coordinates are (tx,ty,tw,th), where the four coordinates represent the center coordinates and the offset values ​​of width and height of the second bounding box, respectively.

Images