Training method and apparatus for railway foreign object phrase localization model, and device and medium

Abstract

The present invention relates to the technical field of object detection. Disclosed are a training method and apparatus for a railway foreign object phrase localization model, and a device and a medium. Cross alignment can be performed on an image feature vector and a text feature vector, so as to obtain an aligned image feature vector and an aligned text feature vector; the pair of the aligned image feature vector and the aligned text feature vector is then used to perform training, such that a model can accurately align a foreign object described in a text with a target region in an image, thereby ensuring accurate phrase localization in a complex scenario, and being applicable to fine-grained scenario description in railway foreign object detection; and a trained railway foreign object phrase localization model can thus effectively execute and complete railway foreign object phrase localization tasks, thereby effectively implementing phrase localization of railway foreign objects.

Description

A training method, apparatus, equipment, and medium for a railway foreign object phrase localization model. Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a training method, apparatus, equipment and medium for a railway foreign object phrase localization model. Background Technology

[0002] Object detection refers to detecting single objects or word-level targets in an image, primarily used to identify and locate individual objects such as stones and tree branches. Phrase localization is an extension of object detection, requiring the location of objects in an image that conform to complex descriptions. For example, when the phrase is described as "stones on the rails," it is necessary to detect the object located on the rails in the image. Phrase localization not only needs to identify the target object but also needs to understand the relationship between the target and its modifiers or environmental information. For instance, "stones on the rails" can be accurately identified and distinguished from "stones outside the rails" through phrase localization.

[0003] Foreign object detection on railway tracks is a crucial component of railway safety management. It aims to monitor the tracks and their surrounding environment in real time, identifying foreign objects that may threaten train safety, such as stones on the tracks and stones outside the tracks. The railway environment is complex and variable; similar foreign objects can have different impacts in different locations. Detection systems that can locate specific foreign objects in images based on phrases are of great significance for railway operational safety. Technical issues

[0004] However, the relevant technologies cannot effectively locate foreign objects on railways in a specific phrase. Technical solutions

[0005] This invention provides a training method, apparatus, equipment, and medium for a railway foreign object phrase localization model, which addresses the shortcomings of related technologies in effectively locating railway foreign objects using phrase localization. The trained railway foreign object phrase localization model can effectively execute and complete the railway foreign object phrase localization task, thus effectively locating railway foreign objects using phrase localization.

[0006] In a first aspect, the present invention provides a training method for a railway foreign object phrase localization model, wherein the railway foreign object phrase localization model includes an identification module, an encoding module, a feature enhancement module, and a detection module; the method includes:

[0007] Obtain the text feature vector of the phrase "railway foreign object" and the image feature vector of the image of the railway foreign object;

[0008] The text feature vector of the railway foreign object phrase text is input into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to multiple preset phrase structure categories, thereby obtaining multiple structure texts in the railway foreign object phrase text and the probability distribution of the structure category corresponding to each structure text;

[0009] Based on each structured text and the corresponding structure category probability distribution, determine the phrase structure category loss;

[0010] Based on the encoding module and the probability distribution of the structure categories, a structural text feature vector corresponding to each preset phrase structure category is generated;

[0011] Based on the feature enhancement module and the structured text feature vector, cross-feature alignment is performed on the image feature vector and the text feature vector to obtain the aligned image feature vector corresponding to the image feature vector and the aligned text feature vector corresponding to the text feature vector;

[0012] The aligned image feature vector and the aligned text feature vector are input into the detection module for target detection to obtain the corresponding detection results, and the detection result loss is determined based on the detection results.

[0013] The railway foreign object phrase localization model is updated based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model.

[0014] Optionally, the plurality of preset phrase structure categories include modifiers, object words, and relation words, and the probability distribution of the structure category corresponding to each structure text includes the probabilities of the phrase structure categories corresponding to the structure text being the modifiers, object words, and relation words, respectively.

[0015] Optionally, determining the phrase structure category loss based on each structured text and the corresponding structure category probability distribution includes:

[0016] For any of the structured texts, calculate the cross-entropy loss corresponding to the structured text based on the cross-entropy loss calculation method and the probability distribution of the structure category corresponding to the structured text;

[0017] The average value is calculated for the cross-entropy loss corresponding to all the structured texts, and this average value is used as the phrase structure category loss.

[0018] Optionally, generating a structural text feature vector corresponding to each preset phrase structural category based on the encoding module and the structural category probability distribution includes:

[0019] For any of the structured texts, determine the maximum probability in the probability distribution of the structure category corresponding to the structured text, and determine the preset phrase structure category corresponding to the maximum probability as the target phrase structure category corresponding to the structured text;

[0020] When the target phrase structure category corresponding to the structured text is the modifier, the structured text is encoded by the encoding module to obtain the corresponding modifier feature vector;

[0021] When the target phrase structure category corresponding to the structured text is the object word, the structured text is encoded by the encoding module to obtain the corresponding object word feature vector;

[0022] When the target phrase structure category corresponding to the structured text is the relation word, the relation word feature vector is extracted from the text feature vector by the encoding module.

[0023] Optionally, the feature enhancement module includes a modifier feature enhancement module, an object word feature enhancement module, and an overall image and text feature enhancement module; the railway foreign object phrase localization model also includes a self-attention mechanism module;

[0024] The step of performing cross-feature alignment on the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector to obtain the aligned image feature vector corresponding to the image feature vector and the aligned text feature vector corresponding to the text feature vector includes:

[0025] The image feature vector and the modifier feature vector are input into the modifier feature enhancement module for cross-feature alignment to obtain the first aligned image feature vector;

[0026] The first aligned image feature vector and the object word feature vector are input into the object word feature enhancement module for cross-feature alignment to obtain the second aligned image feature vector;

[0027] The relation word feature vector is fused with the second aligned image feature vector to obtain a fused feature vector, and the fused feature vector is optimized by the self-attention mechanism module to obtain an optimized feature vector.

[0028] The optimized feature vector and the text feature vector are input into the overall image and text feature enhancement module for cross-feature alignment to obtain the aligned image feature vector and the aligned text feature vector.

[0029] Optionally, the detection result includes the corresponding object category identifier and bounding box information;

[0030] The step of determining the loss of detection results based on the detection results includes:

[0031] The category recognition loss is determined based on the difference between the object category identifier and the true category identifier;

[0032] The bounding box localization loss is determined based on the difference between the bounding box information and the actual bounding box information.

[0033] The category recognition loss and the bounding box localization loss are combined as the detection result loss.

[0034] Optionally, updating the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model includes:

[0035] The total loss is obtained by summing the category recognition loss, the bounding box localization loss, and the phrase structure category loss in the detection result loss.

[0036] If the total loss is determined to be not less than the set loss, then the identification module, the feature enhancement module, and / or the detection module in the railway foreign object phrase localization model are updated to obtain the updated railway foreign object phrase localization model. Based on the text feature vector of the new railway foreign object phrase text and the image feature vector of the new railway foreign object image, the updated railway foreign object phrase localization model is trained again until the trained target phrase localization model is obtained.

[0037] Secondly, the present invention provides a training device for a railway foreign object phrase localization model, wherein the railway foreign object phrase localization model includes an identification module, an encoding module, a feature enhancement module, and a detection module; the device includes:

[0038] The acquisition unit is used to acquire the text feature vector of the phrase "railway foreign object" and the image feature vector of the image of the railway foreign object.

[0039] The first input unit is used to input the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to multiple preset phrase structure categories, and obtains multiple structure texts in the railway foreign object phrase text and the probability distribution of the structure category corresponding to each structure text;

[0040] The first determining unit is configured to determine the phrase structure category loss based on each structured text and the corresponding structure category probability distribution;

[0041] The generation unit is used to generate a structural text feature vector corresponding to each preset phrase structural category based on the encoding module and the structural category probability distribution;

[0042] The alignment unit is used to perform cross-feature alignment on the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector, so as to obtain the aligned image feature vector corresponding to the image feature vector and the aligned text feature vector corresponding to the text feature vector;

[0043] The second input unit is used to input the aligned image feature vector and the aligned text feature vector into the detection module for target detection, and obtain the corresponding detection result.

[0044] The second determining unit is used to determine the detection result loss based on the detection result;

[0045] The update unit is used to update the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss, so as to obtain the trained target railway foreign object phrase localization model.

[0046] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the training method for the railway foreign object phrase localization model of the first aspect or any corresponding embodiment described above.

[0047] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the training method for the railway foreign object phrase localization model of the first aspect or any corresponding embodiment described above. Beneficial effects

[0048] The training method, apparatus, equipment, and medium for the railway foreign object phrase localization model provided by this invention can cross-align image feature vectors and text feature vectors to obtain aligned image feature vectors and aligned text feature vectors. The aligned image feature vectors and aligned text feature vectors are used for training, enabling the model to accurately correspond the foreign object described in the text to the target area in the image. This ensures accurate phrase localization in complex scenes and is suitable for fine-grained scene description in railway foreign object detection. The trained railway foreign object phrase localization model can effectively execute and complete the railway foreign object phrase localization task, effectively realizing the phrase localization of railway foreign objects. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 is a flowchart of a training method for a railway foreign object phrase localization model provided in an embodiment of the present invention;

[0051] Figure 2 is a flowchart of another training method for a railway foreign object phrase localization model provided in an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of the structure of a training device for a railway foreign object phrase localization model provided in an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The best embodiment of the present invention

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] The training method of the railway foreign object phrase localization model of the present invention is described below with reference to Figures 1-2.

[0056] As shown in Figure 1, this embodiment proposes a training method for a first railway foreign object phrase localization model. The railway foreign object phrase localization model includes an identification module, an encoding module, a feature enhancement module, and a detection module. This method may include the following steps:

[0057] S101. Obtain the text feature vector of the phrase "railway foreign object" and the image feature vector of the image of the railway foreign object.

[0058] The railway foreign object phrase localization model can be an open object detection model used to perform phrase localization tasks. It should be noted that the railway foreign object phrase localization model needs to be trained to meet performance requirements. In this embodiment, after obtaining a trained railway foreign object phrase localization model through machine learning, the trained model can be used to perform the railway foreign object phrase localization task.

[0059] Among them, the railway foreign object phrase text is the phrase description text related to railway foreign objects, such as "stones on the rails" and "stones outside the rails".

[0060] Among them, the railway foreign object image is the image to be detected based on the phrase text "railway foreign object".

[0061] In this embodiment, the text of the phrase "railway foreign object" can be encoded to obtain a text feature vector, and the image of the railway foreign object can be encoded to obtain an image feature vector.

[0062] S102. Input the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module can perform named entity recognition on the text feature vector according to multiple preset phrase structure categories, and obtain multiple structure texts in the railway foreign object phrase text and the probability distribution of the structure category corresponding to each structure text.

[0063] Specifically, the recognition module can be a named entity recognition module, used to perform named entity recognition on phrase text.

[0064] Among them, the preset phrase structure category can be a phrase structure category set for railway foreign object phrase text.

[0065] Optionally, multiple preset phrase structure categories include modifiers, object words, and relation words, and the structure category probability distribution corresponding to each structure text includes the probabilities of the phrase structure categories corresponding to the structure text being modifiers, object words, and relation words, respectively.

[0066] It's important to note that modifiers can be categorized into three main types: spatial location modifiers, environmental condition modifiers, and state attribute modifiers. Spatial location modifiers describe the target's relative position to other objects or the environment, such as "on the railway tracks" or "beside the railway tracks." Environmental condition modifiers reflect the target's external environmental factors, such as lighting and weather, for example, "in the shadows" or "target in low light." State attribute modifiers describe the target's own physical state or material changes, such as "loose stones" or "rusty parts." Through these three types of modifiers, the model can more accurately understand the target's attributes and its relationship with the surrounding environment, thereby improving the accuracy and robustness of detection.

[0067] The target word is the object to be detected in the phrase text, such as "stone" in "stones on the rails" or "tree branches" in "tree branches on the rails". Relational words can be defined as the remaining parts of the phrase text excluding the target word and modifiers. For example, "on" and "of" in "stones on the rails".

[0068] Specifically, the recognition module can perform named entity recognition on the text feature vector based on multiple preset phrase structure categories, obtaining multiple structural texts in the railway foreign object phrase text and the probability distribution of the structure category corresponding to each structural text. For example, when the railway foreign object phrase text is "stones on the rails", the recognition module can perform named entity recognition on this railway foreign object phrase text, identifying four structural texts: "railway", "on", "of", and "stone", as well as the probability of each structural text belonging to a preset phrase structure category. Taking "railway" as an example, the probability distribution of the structure category corresponding to "railway" includes the probability that "railway" is a modifier, the probability that "railway" is an object word, and the probability that "railway" is a relation word.

[0069] S103. Determine the phrase structure category loss based on each structured text and the corresponding structure category probability distribution.

[0070] Optionally, step S103 may include:

[0071] For any structured text, calculate the cross-entropy loss corresponding to the structured text based on the cross-entropy loss calculation method and the probability distribution of the structure category corresponding to the structured text;

[0072] The average value of the cross-entropy loss for all structured texts is calculated and used as the phrase structure category loss.

[0073] Specifically, this embodiment can calculate the cross-entropy loss for any structured text based on the cross-entropy loss calculation formula and the probability distribution of the structure category corresponding to that structured text. Then, this embodiment can calculate the average of the cross-entropy losses for all structured texts and use the calculated average as the phrase structure category loss.

[0074] It should be noted that the phrase structure category loss represents the average loss of the model when classifying each word as an object word, modifier, and relation word. In this embodiment, the model's ability to accurately identify phrase structures can be improved by reducing this loss during model training.

[0075] S104. Based on the encoding module and the probability distribution of structural categories, generate the structural text feature vector corresponding to each preset phrase structural category.

[0076] Specifically, in this embodiment, the structural text feature vector corresponding to each preset phrase structural category can be generated based on the encoding module and the probability distribution of structural categories corresponding to each structural text.

[0077] Optionally, for any structured text, determine the maximum probability in the probability distribution of the structure category corresponding to the structured text, and determine the preset phrase structure category corresponding to the maximum probability as the target phrase structure category corresponding to the structured text.

[0078] When the target phrase structure category corresponding to the structured text is a modifier, the structured text is encoded by the encoding module to obtain the corresponding modifier feature vector;

[0079] When the target phrase structure category corresponding to the structured text is an object word, the structured text is encoded by the encoding module to obtain the corresponding object word feature vector;

[0080] When the target phrase structure category corresponding to the structured text is a relation word, the relation word feature vector is extracted from the text feature vector through the encoding module.

[0081] For example, when the phrase text for a foreign object related to railways is "stones on the railway tracks", this embodiment performs named entity recognition on the phrase text to identify four structural texts: "railway", "on", "of", and "stones", as well as the probability of each structural text belonging to a preset phrase structure category. Taking "railway tracks" as an example, if in the probability distribution of the structure category corresponding to "railway", the probability that "railway" belongs to a modifier is 0.7, the probability that "railway" belongs to an object word is 0.2, and the probability that "railway" belongs to a relation word is 0.1, then this embodiment can determine that the modifier is the target phrase structure category corresponding to "railway tracks". After that, this embodiment can encode "railway tracks" through the encoding module to obtain the corresponding modifier feature vector.

[0082] Specifically, in related technologies, positional encoding is used to help models understand the relative or absolute position of each image or sequence element within the overall structure. Drawing on the principles of positional encoding, relational encoding is designed to capture the relationships between the target and its modifiers described in the text and map these relationships to the corresponding image encoding. Specifically, when the target phrase structure category corresponding to the structured text is a relational word, this embodiment can refer to the extraction method of positional encoding in related technologies and extract the relational word feature vector from the text feature vector of the railway foreign object phrase text based on the following transformation: R = FFN(F pharse ).

[0083] Where R is the feature vector of the relation word. FFN represents a feedforward neural network, F pharse The text feature vector representing the phrase "railway foreign object".

[0084] S105. Based on the feature enhancement module and the structured text feature vector, perform cross-feature alignment on the image feature vector and the text feature vector to obtain the aligned image feature vector corresponding to the image feature vector and the aligned text feature vector corresponding to the text feature vector.

[0085] Specifically, this embodiment can perform cross-feature alignment on the image feature vector of the railway foreign object image and the text feature vector of the railway foreign object phrase text according to the feature enhancement module and each structural text feature vector, so as to realize feature alignment and feature enhancement of the image feature vector and the text feature vector, and obtain the aligned image feature vector and the aligned text feature vector.

[0086] Optionally, in the training method of other railway foreign object phrase localization models proposed in this embodiment, the feature enhancement module includes a modifier feature enhancement module, an object word feature enhancement module, and an overall image and text feature enhancement module. The railway foreign object phrase localization model also includes a self-attention mechanism module. In this case, step S105 may include:

[0087] The image feature vector and the modifier feature vector are input into the modifier feature enhancement module for cross-feature alignment to obtain the first aligned image feature vector;

[0088] The first aligned image feature vector and the object word feature vector are input into the object word feature enhancement module for cross-feature alignment to obtain the second aligned image feature vector;

[0089] The feature vector of the relation word is fused with the feature vector of the second aligned image to obtain the fused feature vector, and the fused feature vector is optimized by the self-attention mechanism module to obtain the optimized feature vector.

[0090] The optimized feature vector and text feature vector are input into the overall image and text feature enhancement module for cross-feature alignment to obtain the aligned image feature vector and aligned text feature vector.

[0091] It should be noted that in this embodiment, all feature vectors can have the same dimension. This embodiment can fuse the relation word feature vector and the second aligned image feature vector by adding the relation word feature vector to the second aligned image feature vector. The fused feature vector is then the sum of the relation word feature vector and the second aligned image feature vector.

[0092] Specifically, this embodiment can perform self-attention allocation on the fused feature vector through a self-attention mechanism module, so that the image feature vector can better integrate with the relation word feature vector, and the image feature vector can better understand the relation words and focus on the important features in the image.

[0093] S106. Input the aligned image feature vector and the aligned text feature vector into the detection module to perform target detection and obtain the corresponding detection results.

[0094] Specifically, in this embodiment, target detection can be performed using the aligned image feature vector and the aligned feature vector by the detection module to obtain the corresponding detection results.

[0095] Optionally, the detection results may include the corresponding object category identifier and bounding box information.

[0096] S107. Determine the loss of test results based on the test results.

[0097] Specifically, in this embodiment, the corresponding detection result loss can be calculated based on the detection results output by the detection module.

[0098] Optionally, in the training method of other railway foreign object phrase localization models proposed in this embodiment, when the detection result includes the corresponding object category identifier and bounding box information, step S107 may include:

[0099] The category recognition loss is determined based on the difference between the object category identifier and the true category identifier;

[0100] The bounding box localization loss is determined based on the difference between the bounding box information and the true bounding box information;

[0101] The category recognition loss and bounding box localization loss are combined as the detection result loss.

[0102] Specifically, to ensure the model effectively handles target recognition and modification information understanding during multi-task learning, this embodiment designs a category recognition loss to evaluate the model's prediction accuracy for target categories, employing a binary cross-entropy loss:

[0103] Among them, L BCE The loss is for class identification. N is the number of samples or the number of classes. i The true label (0 or 1) indicates whether the i-th category is the target. The probability of whether the class is the target as predicted by the model is a value obtained after activation with Sigmoid, which ranges from [0,1]. This represents the loss value when the model predicts the existence of the target. This represents the loss value when the model predicts that the target does not exist.

[0104] Specifically, the bounding box localization loss is used to evaluate the difference between the model's predicted target bounding box and the true bounding box, and is calculated using L1 loss. The specific formula is:

[0105] Among them, L loc The loss is used for bounding box localization. N is the number of samples, i.e., the number of bounding boxes. p represents the coordinates of the i-th bounding box predicted by the model (typically including the center coordinates and width and height, or the coordinates of the four vertices). i Let be the coordinates of the actual i-th bounding box. This represents the difference between the model-predicted bounding box and the true bounding box, calculated using the L1 norm as the absolute value of the difference.

[0106] S108. Update the railway foreign object phrase localization model based on the detection result loss and phrase structure category loss to obtain the trained target railway foreign object phrase localization model.

[0107] Specifically, in this embodiment, the model parameters in the railway foreign object phrase localization model can be updated based on the detection result loss and phrase structure category loss.

[0108] Optionally, in the training method of other railway foreign object phrase localization models proposed in this embodiment, step S108 may include:

[0109] The total loss is obtained by summing the category recognition loss, bounding box localization loss, and phrase structure category loss in the detection result loss.

[0110] If the total loss is determined to be no less than the set loss, the recognition module, feature enhancement module, and / or detection module in the railway foreign object phrase localization model are updated to obtain the updated railway foreign object phrase localization model. Based on the text feature vector of the new railway foreign object phrase text and the image feature vector of the new railway foreign object image, the updated railway foreign object phrase localization model is trained again until the trained target phrase localization model is obtained.

[0111] Specifically, this embodiment assigns different weights to each type of loss based on their importance, including category recognition loss, bounding box localization loss, and phrase structure category loss. The total loss is obtained by weighted summation of these losses based on their respective weights. It should be noted that this embodiment can initially initialize the weights equally and then automatically adjust the weights between the object detection loss and the modifier understanding loss by analyzing the data distribution in real time. Selecting appropriate weight coefficients requires adjustment based on the model's performance on the validation set, which involves considering the impact on different tasks and the trade-offs between them. Adjusting the weights is a process of trial and error, requiring multiple experiments to find the optimal settings.

[0112] Specifically, this embodiment can update the model parameters of the identification module, feature enhancement module, and / or detection module in the railway foreign object phrase localization model to update the railway foreign object phrase localization model and improve the detection accuracy of the railway foreign object phrase localization model.

[0113] It should be noted that this embodiment improves the accuracy and flexibility of phrase localization. By introducing a multi-task learning framework including name recognition, image-text alignment, and object detection, this embodiment can simultaneously handle object detection and modifier understanding tasks. The railway foreign object phrase localization model can effectively handle complex phrase structures, especially when describing the relationship between the object and its environment (such as "stones on the rails" and "stones outside the rails"). This flexibility significantly improves the model's accuracy in detecting foreign objects on railways and can address foreign object localization problems under various linguistic descriptions.

[0114] This embodiment enhances the model's adaptability to environmental changes. By designing an environmental feature loss function, it enables the model to adapt to common changes in lighting, weather, and viewing angles in railway scenarios. Especially under extreme conditions such as heavy rain, snow accumulation, strong winds, or low-light nighttime environments, the model maintains high stability and detection accuracy. The model's performance under different environmental conditions ensures the reliability of railway safety.

[0115] This embodiment achieves efficient image-text multimodal alignment. By optimizing the multimodal fusion strategy, this embodiment achieves efficient alignment between image and text features. The model can accurately match the foreign object described in the text with the target region in the image, ensuring accurate phrase localization in complex scenes, and is suitable for fine-grained scene description in railway foreign object detection.

[0116] The training method for the railway foreign object phrase localization model proposed in this embodiment can cross-align image feature vectors and text feature vectors to obtain aligned image feature vectors and aligned text feature vectors. The aligned image feature vectors and aligned text feature vectors are then used for training, enabling the model to accurately match the foreign object described in the text with the target area in the image. This ensures accurate phrase localization in complex scenarios and is suitable for fine-grained scene descriptions in railway foreign object detection. The trained railway foreign object phrase localization model can effectively perform and complete the railway foreign object phrase localization task, effectively achieving phrase localization of railway foreign objects. Embodiments of the present invention

[0117] In related technologies, open object detection has become an important research direction in the field of computer vision, aiming to handle object categories not seen during training. Grounding DINO is an object detection model based on multimodal information, which aims to improve the accuracy and flexibility of object localization by combining image and text features. This model can effectively understand the semantics of input text and locate the corresponding objects in images, thus performing excellently in complex scenes.

[0118] This embodiment focuses on the application of open target detection technology based on the Grounding DINO model in railway foreign object detection. By fine-tuning the model, it explores how to integrate new data and categories while retaining the original detection capabilities, thereby improving the accuracy and generalization ability of phrase localization tasks. The aim is to provide innovative solutions for railway safety monitoring and promote the practical application of open target detection technology.

[0119] Grounding DINO includes a dual encoder-single decoder architecture, mainly comprising the following components:

[0120] Image and text backbone networks: used to extract multi-scale features from images and semantic features from text. This process ensures that the model can fully understand the modal diversity of the input data.

[0121] Feature enhancer: Responsible for cross-modal feature fusion, improving the alignment between image and text features, thereby enhancing the model's ability to understand the target.

[0122] Language-guided query selection module: By analyzing the input text, it selects relevant queries from image features for decoder initialization, ensuring that the model can accurately focus on the target region.

[0123] Cross-modal decoder: The selected query is input into the self-attention layer and the cross-attention layer to refine the target bounding box and extract phrase information, so as to achieve efficient target localization and recognition.

[0124] The phrase localization task aims to accurately locate relevant objects in an image based on input text. Grounding DINO effectively handles this task through multimodal learning, specifically including:

[0125] Multimodal input processing: The model can process images and text simultaneously, establish effective connections, and ensure the comprehensiveness and accuracy of object detection.

[0126] Semantic matching capability: Through advanced feature fusion technology, the model can handle different language structures and expressions, ensuring semantic consistency between the detected object and the input phrase.

[0127] Fine-grained positioning: Grounding DINO can accurately locate the region corresponding to a specific phrase and supports complex phrase combinations, thus providing accurate information in special scenarios such as railway monitoring.

[0128] Despite its good performance in phrase localization, the emergence of new categories in open-set scenarios still poses challenges to the model's adaptability and accuracy. Grounding DINO performs well at the word level in open object detection tasks. By fine-tuning common word object categories, the pre-trained model can significantly improve accuracy when introducing new data and categories. The model incorporates the original data during this fine-tuning process, essentially performing a new pre-training, thus maintaining good generalization ability.

[0129] However, in phrase localization tasks, due to the complexity of phrases, effectively integrating new and existing data to ensure the model's accuracy and generalization ability at the phrase level remains a pressing issue. This indicates that in specific application scenarios (such as railway monitoring), further optimizing multimodal data processing schemes, especially improving phrase-level generalization ability, is crucial.

[0130] The training method for the railway foreign object phrase localization model proposed in this embodiment can be used to solve the following specific technical problems:

[0131] 1. Automatic Distinction Between Target and Modifier: In railway scenarios, phrase localization tasks require models to accurately distinguish between targets and modifiers, especially when describing the relationship between the target and the environment (e.g., "stones on the rails" and "stones outside the rails"). Existing models often struggle to distinguish between target and modifier information when dealing with complex phrase structures, resulting in inaccurate target localization.

[0132] 2. Recognition of New Categories and Model Generalization Ability: Railways contain a wide variety of foreign objects, and existing models perform poorly in recognizing new, unseen target categories, especially in open target detection scenarios where their generalization ability is limited. Furthermore, when adapting to new data, models are prone to "catastrophic forgetting," meaning that while learning new categories, their ability to recognize older targets declines.

[0133] 3. Adaptability to Environmental Changes: Railway scenarios are complex and ever-changing; factors such as lighting, weather, and viewing angle can significantly affect the appearance of targets. Existing models struggle to effectively adapt to these environmental changes, especially when understanding the relationship between targets and their environment is required, resulting in poor performance and reduced stability and reliability of the detection system.

[0134] 4. Fusion and Processing of Multimodal Data: The task of railway foreign object detection requires the model to effectively combine image and text information. However, existing multimodal models struggle to achieve accurate image-text alignment when processing complex phrases, especially those containing modifiers (such as "stones beside the railway tracks"), resulting in unsatisfactory detection performance.

[0135] The above-mentioned technical problems can be solved in the following way in this embodiment:

[0136] 1. Enhanced diversity of data.

[0137] This embodiment can introduce diversity data augmentation techniques to improve the model's generalization ability in railway track foreign object detection tasks. The railway environment is complex and variable; different weather conditions, lighting conditions, seasonal changes, and track locations can all affect the appearance of foreign objects. Furthermore, the types of foreign objects are diverse (such as stones, branches, and debris), requiring the model to have strong adaptability. Word-based data may lead to insufficient recognition performance in practical applications, especially when encountering new scenes or unseen foreign objects, where the model may lose accuracy.

[0138] To address this issue, this embodiment employs diverse data augmentation strategies to significantly improve the model's adaptability to different scenarios and types of foreign objects, thereby achieving higher detection accuracy and robustness.

[0139] Basic Labeling Rules: This embodiment first establishes basic labeling rules for foreign objects on railway tracks. These rules not only cover the basic physical characteristics of the target, such as shape, color, and size, but also include railway-specific terminology and definitions of specific events. In railway foreign object detection tasks, target modifiers can be divided into three main categories: spatial location, environmental conditions, and state attributes. Spatial location modifiers describe the relative position of the target to other objects or the environment, such as "on the tracks" or "beside the tracks." Environmental condition modifiers reflect the external environmental factors of the target, such as lighting and weather, for example, "in the shadows" or "target under low light." State attribute modifiers describe the target's own physical state or material changes, such as "loose stones" or "rusted fittings." Through these three types of modifiers, the model can more accurately understand the attributes of the target and its relationship with the surrounding environment, thereby improving the accuracy and robustness of detection.

[0140] Railway-Specific Scene Labeling: In railway foreign object detection tasks, the scene is not limited to the railway tracks themselves, but also includes the surrounding environment, such as the embankments on both sides of the tracks, tunnel entrances, or the space under bridges. These locations may contain different types of foreign objects that can affect the safe operation of trains. Therefore, special attention must be paid to the details of these environments during the labeling process. For example, snow may accumulate at tunnel entrances, potentially sliding onto the tracks when a train passes; fallen rocks on the embankments may roll onto the tracks during the rainy season, directly threatening traffic safety. Labeling rules incorporate this environmental information, providing the model with a richer contextual understanding.

[0141] Artificial Intelligence Generates Expressions: To further expand the training data, this embodiment can employ artificial intelligence to generate rich phrase expressions. This process combines annotation rules with natural language processing techniques, automatically creating diverse descriptions using generative models (such as generative pre-trained transformers). These descriptions are not only based on existing annotation rules but also generate phrase expressions containing more details or different grammatical structures, thus enhancing data diversity.

[0142] For example, for the same foreign object, artificial intelligence can generate multiple expressions, such as: "A loose stone has been found on the railway track, which may affect the operation of the train." or "A stone is located on the railway track, and it is recommended to remove it in time." Such diverse expressions not only enhance the richness of the training data, but also help the model better understand different descriptions of the same scene, thereby improving its ability to understand complex scenes.

[0143] Rule-based annotation and intelligent correction: To ensure data accuracy and diversity, this embodiment employs a human-machine collaborative annotation approach. First, humans manually annotate the images according to basic annotation rules, ensuring professionalism and consistency. Then, an AI model generates corresponding phrase descriptions for these images. When the phrases generated by the AI ​​model are significantly inconsistent with the semantics of the manually annotated images, human experts can be automatically requested to review and correct them. This approach not only guarantees data quality but also greatly enriches the diversity of the dataset, enabling the model to be trained more efficiently.

[0144] This embodiment can construct diverse railway foreign object phrase texts through the above methods, achieving diverse data enhancement for railway foreign object detection tasks, thereby improving the model's generalization ability and detection accuracy. The implementation of this technical solution will provide a more effective solution for railway safety monitoring.

[0145] 2. Network structure modification and multi-task learning design.

[0146] To address the complex phrase localization problem in railway track foreign object detection, this embodiment employs a multi-task learning training strategy. In this architecture, the model is responsible not only for target identification but also for processing target-related modifiers (such as spatial location and environmental influences). To achieve this, the model's network structure has been modified by introducing a recognition module, an encoding module, and a feature enhancement module.

[0147] In the image annotation, the precise location of the main targets, which may be objects or specific scene elements, is marked. The bounding box around each target is designed to accurately capture its spatial location in the image. We also annotate modifier information related to the targets, such as color, shape, or spatial relationships, to help the model understand how these attributes affect the interpretation of the targets. For phrases describing the entire scene, we annotate a bounding box containing all relevant elements, which helps the model learn how to holistically understand and process complex image content.

[0148] Text annotation involves classifying text that describes targets, modifiers, and relationships. We categorize the text into three classes: text describing the main target is labeled "I," text describing modifier information is labeled "II," and text describing the relationship between the target and modifiers is labeled "III." This classification method helps the model more accurately identify and classify different elements in the text when performing named entity recognition tasks.

[0149] Data processing begins with input of images and related descriptive text. A Vision Transformer (ViT) is used to process the image, dividing it into multiple equal-sized image patches. Each patch is converted into a vector of fixed dimensions. The processed image features have dimensions N×d, where N is the number of patches and d is the depth of the feature vector, typically 768. These image feature vectors can then be used collectively as the image feature vector for the railway foreign object image.

[0150] The text data is transformed into vectors of dimension L×d through an embedding layer, where L is the length of the text sequence. These text vectors are then processed by a self-attention mechanism, and the output text feature vectors also have a dimension of L×d. These text vectors can be used as a whole as the text feature vector for the phrase "railway foreign object".

[0151] Phase 1: Phrase composition identification.

[0152] A feedforward neural network and a sigmoid module are used to process the textual feature vectors of railway foreign object phrases to perform a named entity recognition task, identifying target (i.e., object words), modifiers, and relation words in the railway foreign object phrase text. Each type of entity (target, modifier, relation; relation is defined as the remaining part of the phrase excluding target and modifier words) is identified separately, and the phrase text is combined to generate corresponding codes for target and modifier words respectively. These codes have their own length and dimensions, forming a representation similar to the phrase encoding structure. This process is optimized using a classification loss.

[0153] As shown in Figure 2, in this embodiment, the text feature vector of the railway foreign object phrase text can be input into the recognition module. The recognition module then performs named entity recognition on the text feature vector based on multiple preset phrase structure categories, obtaining multiple structural texts in the railway foreign object phrase text and the probability distribution of the structure category corresponding to each structural text. The probability distribution of the structure category corresponding to each structural text includes the probability that the phrase structure category of the structural text is a modifier, object word, or relation word. Then, in this embodiment, for any structural text, the maximum probability is determined in the probability distribution of the structure category corresponding to the structural text, and the preset phrase structure category corresponding to the maximum probability is determined as the target phrase structure category of the structural text. When the target phrase structure category of the structural text is a modifier, the structural text is encoded by the encoding module to obtain the corresponding modifier feature vector; when the target phrase structure category of the structural text is an object word, the structural text is encoded by the encoding module to obtain the corresponding object word feature vector; when the target phrase structure category of the structural text is a relation word, the relation word feature vector is extracted from the text feature vector by the encoding module.

[0154] It should be noted that the original image encoding in Figure 2 is the image feature vector of the railway foreign object image, and the original phrase encoding is the text feature vector of the railway foreign object phrase text. The original phrase text includes multiple structured texts obtained by performing named entity recognition on the text feature vector of the railway foreign object phrase text through the recognition module in this embodiment. The modifier encoding is the modifier feature vector obtained by encoding the structured text corresponding to the modifier in this embodiment. The target encoding is the object word feature vector obtained by encoding the structured text corresponding to the object words in multiple structured texts in this embodiment. The relation encoding is the relation word feature vector obtained by feature extraction from the original phrase encoding in this embodiment. The modifier feature enhancer and the target feature enhancer are the modifier feature enhancement module and the object word feature enhancement module, respectively, and the overall image and text feature enhancer is the overall image and text feature enhancement module. The new image encoding and the new phrase encoding are the aligned image feature vector and the aligned text feature vector, respectively.

[0155] Second stage: Multi-round feature alignment.

[0156] After identifying the target, modifiers, and relationships in the text, multiple rounds of feature alignment are performed:

[0157] The first layer is alignment (modifiers). Modifier encodings are first used in the alignment process and then fed into the cross-attention module. This step allows the model to focus on scene or background regions in the image. Through the cross-attention mechanism, parts of the image features related to modifiers receive higher attention weights. This helps the model better understand and interpret the environment and context in the image.

[0158] As shown in Figure 2, in this embodiment, the image feature vector and the modifier feature vector can be input into the modifier feature enhancement module for cross-feature alignment to obtain the first aligned image feature vector.

[0159] The second alignment layer (target word). Subsequently, the target word encoding is input into the model. At this point, the model has obtained the alignment information of the modifiers with the background region, allowing it to further focus on specific target regions in the image without ignoring the influence of the modifiers. This step-by-step alignment method ensures that the model can accurately identify and locate target entities while integrating scene information. The loss functions used in the alignment process include recognition and localization losses.

[0160] As shown in the figure, in this embodiment, the first aligned image feature vector and the object word feature vector can be input into the object word feature enhancement module for cross-feature alignment to obtain the second aligned image feature vector.

[0161] Third stage: Relation encoding generation.

[0162] In visual models of related technologies, positional encoding is used to help the model understand the relative or absolute position of each image or sequence element within the overall structure. Drawing inspiration from positional encoding, relational encoding is designed to capture the relationships between the target and its embellishments described in the text and map these relationships to corresponding image codes.

[0163] As shown in Figure 2, this embodiment fuses the relation word feature vector with the second aligned image feature vector to obtain a fused feature vector. Further processing is performed using a self-attention mechanism module to optimize the fused feature vector, resulting in an optimized feature vector. The processed encoding is aligned with the overall phrase encoding to ensure that the image features fully reflect the semantic relationships described in the text.

[0164] 3. Combining continuous learning with open target detection.

[0165] To enhance the model's adaptability to open target detection scenarios, this embodiment can also incorporate continuous learning. When introducing new categories or new data, the model is fine-tuned through progressive learning techniques to ensure that the introduction of new data does not impair the model's ability to recognize existing targets and avoids "catastrophic forgetting" of the model.

[0166] Specifically, this embodiment allows for the gradual introduction of new categories. Five to ten new categories are introduced at a time, combined with a small amount of labeled data (100-500 labeled images), ensuring the model quickly adapts to the new categories while retaining its ability to recognize the old ones. Data is introduced in batches to progressively adjust the model's recognition capabilities and avoid overfitting to the new categories.

[0167] Specifically, this embodiment can perform feature-based forgetting prevention. Through knowledge distillation, feature information of old categories is preserved when introducing new categories, preventing the model from forgetting old categories. Temperature softening is used to smooth the category distribution, ensuring a balance between learning new and old categories. Feature alignment minimizes the feature differences between the new and old models, ensuring that the introduction of new categories does not affect the recognition of old targets.

[0168] Specifically, this embodiment allows for adaptive learning adjustments. As the model introduces a new category, it adaptively adjusts the weight distribution between the new and old data, initially giving higher weights to the new category and gradually balancing them out. This mechanism ensures that the model can effectively identify both new and old categories simultaneously.

[0169] Understandably, this embodiment can improve the efficiency and diversity of data augmentation. By combining intelligently generated diverse phrase descriptions with basic annotation rules, this embodiment significantly enhances the diversity of data. This augmentation method ensures that the model has stronger generalization ability when dealing with complex and variable scenarios, and can greatly improve the accuracy and efficiency of foreign object detection, especially when facing various foreign objects on railway tracks (such as stones, branches, tools, etc.).

[0170] This embodiment enhances the generalization ability of open object detection. By combining continuous learning with an adaptive loss function, this embodiment significantly improves the model's performance in open object detection. When faced with unfamiliar new categories (such as foreign objects of different materials and shapes), the model maintains high detection accuracy while avoiding "catastrophic forgetting" caused by the introduction of new data. This makes the model more applicable to railway track foreign object detection, effectively handling the emergence of various new foreign objects.

[0171] This embodiment provides a scalable open target detection solution. Through a continuous learning mechanism, it enables incremental updates when new data is introduced. The model can quickly adapt to new categories in railway foreign object detection tasks while maintaining high accuracy in identifying existing categories. This solution is highly scalable and adaptable to long-term dynamic changes in the railway safety monitoring environment, providing stable and reliable technical support for railway safety.

[0172] This embodiment of the railway foreign object phrase localization model employs a dedicated multi-task learning architecture for railway foreign object detection. This embodiment designs a dedicated multi-task learning architecture for railway foreign object detection tasks. The model can simultaneously detect foreign objects and resolve the spatial relationship between the target and the railway tracks (e.g., "a stone on the tracks"), thereby improving the accuracy of phrase localization tasks. This architecture is particularly suitable for handling tasks involving complex phrases and multiple scenarios (such as various environmental changes in railway foreign object detection tasks), significantly improving the model's generalization ability and accuracy. The multi-task learning architecture, specifically designed for the target and modifier parsing task in railway foreign object detection, ensures the model's efficiency and accuracy in complex railway scenarios. This architecture and its optimization algorithm ensure the model's real-time response and efficiency in complex scenarios by dynamically adjusting the task processing flow.

[0173] This embodiment implements an adaptive loss function design for railway scenarios. Based on the Grounding DINO model, this embodiment improves the loss function to achieve an adaptive weight adjustment mechanism, enabling the model to dynamically adapt to different phrase structures (such as the relationship between the target and modifiers) and complex railway environments (such as lighting and weather). This mechanism, through weight adjustment, can respond to environmental changes in real time, such as extreme weather and different lighting conditions, allowing the model to maintain stable performance in varied railway scenarios. The adaptive loss function can dynamically adjust task weights to adapt the model to different phrase structures and railway environmental conditions (such as lighting and viewing angle). This mechanism performs particularly well in extreme environments (such as insufficient lighting and heavy rain), ensuring the model maintains high accuracy in target detection and modifier parsing.

[0174] This embodiment features a continuous learning mechanism for foreign object detection in railway systems. This mechanism allows the model to maintain its detection capabilities for existing target categories when new foreign object categories are introduced, avoiding "catastrophic forgetting." Through batch-based progressive learning, a smooth transition during the introduction of new categories is ensured, preventing the impact of new categories on the detection performance of older categories. By continuously optimizing the training strategy, the model can quickly adapt to new categories in the dynamic changes of the railway scenario, ensuring the effectiveness of open object detection. Through continuous learning, the model can adapt to new foreign object detection in railway scenarios, ensuring that the introduction of new categories in open object detection tasks does not affect the detection performance of older categories. This continuous learning mechanism enables the model to maintain high performance in the long-term dynamic changes of railway safety monitoring, adapting to the constantly changing environment and newly added foreign object categories.

[0175] As shown in Figure 3, this embodiment proposes a training device for a railway foreign object phrase localization model. The railway foreign object phrase localization model includes an identification module, an encoding module, a feature enhancement module, and a detection module. The device may include:

[0176] The acquisition unit 301 is used to acquire the text feature vector of the phrase "railway foreign object" and the image feature vector of the image of the railway foreign object;

[0177] The first input unit 302 is used to input the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module can perform named entity recognition on the text feature vector according to multiple preset phrase structure categories, and obtain multiple structure texts in the railway foreign object phrase text and the probability distribution of the structure category corresponding to each structure text.

[0178] The first determining unit 303 is used to determine the phrase structure category loss based on each structure text and the corresponding structure category probability distribution;

[0179] The generation unit 304 is used to generate structural text feature vectors corresponding to each preset phrase structural category based on the encoding module and the structural category probability distribution;

[0180] Alignment unit 305 is used to perform cross-feature alignment on image feature vector and text feature vector based on feature enhancement module and structured text feature vector, so as to obtain aligned image feature vector corresponding to image feature vector and aligned text feature vector corresponding to text feature vector;

[0181] The second input unit 306 is used to input the aligned image feature vector and the aligned text feature vector into the detection module for target detection and to obtain the corresponding detection result.

[0182] The second determining unit 307 is used to determine the detection result loss based on the detection result;

[0183] Update unit 308 is used to update the railway foreign object phrase localization model based on detection result loss and phrase structure category loss, so as to obtain a trained target railway foreign object phrase localization model.

[0184] It should be noted that the processing procedures and beneficial effects of the acquisition unit 301, the first input unit 302, the first determination unit 303, the generation unit 304, the alignment unit 305, the second input unit 306, the second determination unit 307, and the update unit 308 can be referred to steps S101 to S108 in Figure 1, and will not be repeated here.

[0185] Optionally, multiple preset phrase structure categories include modifiers, object words, and relation words, and the structure category probability distribution corresponding to each structure text includes the probabilities of the phrase structure categories corresponding to the structure text being modifiers, object words, and relation words, respectively.

[0186] Optionally, the first determining unit 303 is also used for:

[0187] For any structured text, calculate the cross-entropy loss corresponding to the structured text based on the cross-entropy loss calculation method and the probability distribution of the structure category corresponding to the structured text;

[0188] The average value of the cross-entropy loss for all structured texts is calculated and used as the phrase structure category loss.

[0189] Optionally, generation unit 304 is also used for:

[0190] For any structured text, determine the maximum probability in the probability distribution of the structure category corresponding to the structured text, and determine the preset phrase structure category corresponding to the maximum probability as the target phrase structure category corresponding to the structured text.

[0191] When the target phrase structure category corresponding to the structured text is a modifier, the structured text is encoded by the encoding module to obtain the corresponding modifier feature vector;

[0192] When the target phrase structure category corresponding to the structured text is an object word, the structured text is encoded by the encoding module to obtain the corresponding object word feature vector;

[0193] When the target phrase structure category corresponding to the structured text is a relation word, the relation word feature vector is extracted from the text feature vector through the encoding module.

[0194] Optionally, the feature enhancement module includes a modifier feature enhancement module, an object word feature enhancement module, and an overall image and text feature enhancement module; the railway foreign object phrase localization model also includes a self-attention mechanism module;

[0195] Alignment unit 305 is also used for:

[0196] The image feature vector and the modifier feature vector are input into the modifier feature enhancement module for cross-feature alignment to obtain the first aligned image feature vector;

[0197] The first aligned image feature vector and the object word feature vector are input into the object word feature enhancement module for cross-feature alignment to obtain the second aligned image feature vector;

[0198] The feature vector of the relation word is fused with the feature vector of the second aligned image to obtain the fused feature vector, and the fused feature vector is optimized by the self-attention mechanism module to obtain the optimized feature vector.

[0199] The optimized feature vector and text feature vector are input into the overall image and text feature enhancement module for cross-feature alignment to obtain the aligned image feature vector and aligned text feature vector.

[0200] Optionally, the detection results may include the corresponding object category identifier and bounding box information;

[0201] The second determining unit 307 is also used for:

[0202] The category recognition loss is determined based on the difference between the object category identifier and the true category identifier;

[0203] The bounding box localization loss is determined based on the difference between the bounding box information and the true bounding box information;

[0204] The category recognition loss and bounding box localization loss are combined as the detection result loss.

[0205] Optionally, update unit 308 is also used for:

[0206] The total loss is obtained by summing the category recognition loss, bounding box localization loss, and phrase structure category loss in the detection result loss.

[0207] If the total loss is determined to be no less than the set loss, the recognition module, feature enhancement module, and / or detection module in the railway foreign object phrase localization model are updated to obtain the updated railway foreign object phrase localization model. Based on the text feature vector of the new railway foreign object phrase text and the image feature vector of the new railway foreign object image, the updated railway foreign object phrase localization model is trained again until the trained target phrase localization model is obtained.

[0208] The training device for the railway foreign object phrase localization model proposed in this embodiment can cross-align image feature vectors and text feature vectors to obtain aligned image feature vectors and aligned text feature vectors. The aligned image feature vectors and aligned text feature vectors are used for training, enabling the model to accurately match the foreign object described in the text with the target area in the image. This ensures accurate phrase localization in complex scenes and is suitable for fine-grained scene description in railway foreign object detection. The trained railway foreign object phrase localization model can effectively execute and complete the railway foreign object phrase localization task, effectively realizing the phrase localization of railway foreign objects.

[0209] In this embodiment, the training device for the railway foreign object phrase localization model is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0210] This invention also provides a computer device having a training apparatus for the railway foreign object phrase localization model shown in FIG3.

[0211] Please refer to Figure 4, a schematic diagram of a computer device according to an optional embodiment of the present invention. The computer device includes one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 shows an example of a single processor 10.

[0212] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0213] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0214] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0215] Memory 20 may include volatile memory, such as random access memory. Memory may also include non-volatile memory, such as flash memory, hard disk, or solid-state drive. Memory 20 may also include combinations of the above types of memory.

[0216] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0217] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0219] Cross-reference to related applications

[0220] This application claims priority to Chinese patent application No. 202411628972.5, filed on November 15, 2024, the entire contents of which are incorporated herein by reference. Industrial applicability

[0221] This invention can be widely applied to EEG data management, primarily in clinical hospitals, to help doctors manage and access EEG data more efficiently, improving the accuracy and efficiency of diagnosis and treatment. Simultaneously, it can also be applied to scientific research, enabling researchers to access and analyze data in batches, thus promoting the development of EEG-related research.

[0222] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a railway foreign object short phrase positioning model, characterized in that, The railway foreign object phrase localization model includes an identification module, an encoding module, a feature enhancement module, and a detection module; the method includes: Obtain the text feature vector of the phrase "railway foreign object" and the image feature vector of the image of the railway foreign object; The text feature vector of the railway foreign object phrase text is input into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to multiple preset phrase structure categories, thereby obtaining multiple structure texts in the railway foreign object phrase text and the probability distribution of the structure category corresponding to each structure text; Based on each structured text and the corresponding structure category probability distribution, determine the phrase structure category loss; Based on the encoding module and the probability distribution of the structure categories, a structural text feature vector corresponding to each preset phrase structure category is generated; Based on the feature enhancement module and the structured text feature vector, cross-feature alignment is performed on the image feature vector and the text feature vector to obtain the aligned image feature vector corresponding to the image feature vector and the aligned text feature vector corresponding to the text feature vector; The aligned image feature vector and the aligned text feature vector are input into the detection module for target detection to obtain the corresponding detection results, and the detection result loss is determined based on the detection results. The railway foreign object phrase localization model is updated based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model.

2. The method of claim 1, wherein, The plurality of preset phrase structure categories include modifiers, object words, and relation words. The probability distribution of the structure category corresponding to each structure text includes the probability that the phrase structure category corresponding to the structure text is the modifier, the object word, and the relation word, respectively.

3. The method of claim 2, wherein, The step of determining the phrase structure category loss based on each structured text and the corresponding structure category probability distribution includes: For any of the structured texts, calculate the cross-entropy loss corresponding to the structured text based on the cross-entropy loss calculation method and the probability distribution of the structure category corresponding to the structured text; The average value is calculated for the cross-entropy loss corresponding to all the structured texts, and this average value is used as the phrase structure category loss.

4. The method of claim 2, wherein, The step of generating a structural text feature vector corresponding to each preset phrase structural category based on the encoding module and the structural category probability distribution includes: For any of the structured texts, determine the maximum probability in the probability distribution of the structure category corresponding to the structured text, and determine the preset phrase structure category corresponding to the maximum probability as the target phrase structure category corresponding to the structured text; When the target phrase structure category corresponding to the structured text is the modifier, the structured text is encoded by the encoding module to obtain the corresponding modifier feature vector; When the target phrase structure category corresponding to the structured text is the object word, the structured text is encoded by the encoding module to obtain the corresponding object word feature vector; When the target phrase structure category corresponding to the structured text is the relation word, the relation word feature vector is extracted from the text feature vector by the encoding module.

5. The method of claim 4, wherein, The feature enhancement module includes a modifier feature enhancement module, an object word feature enhancement module, and an overall image and text feature enhancement module; The railway foreign object phrase localization model also includes a self-attention mechanism module; The step of performing cross-feature alignment on the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector to obtain the aligned image feature vector corresponding to the image feature vector and the aligned text feature vector corresponding to the text feature vector includes: The image feature vector and the modifier feature vector are input into the modifier feature enhancement module for cross-feature alignment to obtain the first aligned image feature vector; The first aligned image feature vector and the object word feature vector are input into the object word feature enhancement module for cross-feature alignment to obtain the second aligned image feature vector; The relation word feature vector is fused with the second aligned image feature vector to obtain a fused feature vector, and the fused feature vector is optimized by the self-attention mechanism module to obtain an optimized feature vector. The optimized feature vector and the text feature vector are input into the overall image and text feature enhancement module for cross-feature alignment to obtain the aligned image feature vector and the aligned text feature vector.

6. The method of claim 1, wherein, The detection results include the corresponding object category identifier and bounding box information; The step of determining the loss of detection results based on the detection results includes: The category recognition loss is determined based on the difference between the object category identifier and the true category identifier; The bounding box localization loss is determined based on the difference between the bounding box information and the actual bounding box information. The category recognition loss and the bounding box localization loss are combined as the detection result loss.

7. The method of claim 6, wherein, The step of updating the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss to obtain a trained target railway foreign object phrase localization model includes: The total loss is obtained by summing the category recognition loss, the bounding box localization loss, and the phrase structure category loss in the detection result loss. If the total loss is determined to be not less than the set loss, then the identification module, the feature enhancement module, and / or the detection module in the railway foreign object phrase localization model are updated to obtain the updated railway foreign object phrase localization model. Based on the text feature vector of the new railway foreign object phrase text and the image feature vector of the new railway foreign object image, the updated railway foreign object phrase localization model is trained again until the trained target phrase localization model is obtained.

8. A device for training a railway foreign object short phrase positioning model, characterized by, The railway foreign object phrase localization model includes an identification module, an encoding module, a feature enhancement module, and a detection module; the device includes: The acquisition unit is used to acquire the text feature vector of the phrase "railway foreign object" and the image feature vector of the image of the railway foreign object. The first input unit is used to input the text feature vector of the railway foreign object phrase text into the recognition module, so that the recognition module performs named entity recognition on the text feature vector according to multiple preset phrase structure categories, and obtains multiple structure texts in the railway foreign object phrase text and the probability distribution of the structure category corresponding to each structure text; The first determining unit is configured to determine the phrase structure category loss based on each structured text and the corresponding structure category probability distribution; The generation unit is used to generate a structural text feature vector corresponding to each preset phrase structural category based on the encoding module and the structural category probability distribution; The alignment unit is used to perform cross-feature alignment on the image feature vector and the text feature vector based on the feature enhancement module and the structured text feature vector, so as to obtain the aligned image feature vector corresponding to the image feature vector and the aligned text feature vector corresponding to the text feature vector; The second input unit is used to input the aligned image feature vector and the aligned text feature vector into the detection module for target detection, and obtain the corresponding detection result. The second determining unit is used to determine the detection result loss based on the detection result; The update unit is used to update the railway foreign object phrase localization model based on the detection result loss and the phrase structure category loss, so as to obtain the trained target railway foreign object phrase localization model.

9. A computer device, comprising: include: A memory and a processor are interconnected, the memory storing computer instructions, and the processor executing the computer instructions to perform the training method of the railway foreign object phrase localization model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the training method of the railway foreign object phrase localization model according to any one of claims 1 to 7.

Images