Accident detection method and apparatus, and storage medium

Abstract

Disclosed are an accident detection method and apparatus, and a storage medium. The method comprises: receiving an input video; identifying a plurality of objects in the video; when a specific object from the plurality of objects complies with a predetermined rule of transitioning from a moving state to a stopped state and finally remaining in the stopped state, determining that the occurrence of a potential accident is experienced by the specific object; determining a frame image in the video where the occurrence of a potential accident is experienced by the specific object; and instructing a trained image processing model to analyze and process the frame image, so as to determine whether the occurrence of an accident is experienced by the specific object.

Description

Methods and apparatus for detecting accidents, and storage media Technical Field

[0001] This disclosure generally relates to information processing technology, and more specifically, to technology for automatically detecting traffic accidents. Background Technology

[0002] Automatic traffic accident detection technology is a key component of Intelligent Transportation Systems (ITS). This technology identifies and responds to traffic accidents by monitoring and analyzing road conditions in real time.

[0003] Automated traffic accident detection can significantly improve the efficiency and effectiveness of traffic safety management. First, it can quickly and accurately identify traffic accidents and promptly notify rescue departments, thereby reducing personal injury and property damage. Second, through real-time monitoring and data analysis, it can provide decision support for traffic management departments, optimize traffic flow management, and reduce traffic congestion. Furthermore, by collecting and analyzing accident data, researchers and policymakers can understand the causes of accidents, thus developing more effective traffic strategies. In addition, automated detection technology can reduce the workload of law enforcement officers, allowing them to focus more on other important aspects of accident handling.

[0004] Therefore, the research and application of automatic traffic accident detection technology is of great significance for improving the overall traffic safety level of society. Reducing traffic accidents can improve road efficiency, reduce economic losses, and enhance public confidence in traffic safety. In short, the development and application of automatic traffic accident detection technology has a profound impact on improving road safety, reducing accidents, and protecting people's lives and property.

[0005] Many technologies have been proposed for the automatic detection of traffic accidents. However, existing technologies often have the following drawbacks:

[0006] - High false alarm rate: Due to severe weather conditions, obstructions, or abnormal situations, it is impossible to accurately detect the occurrence of accidents;

[0007] - Computational complexity: It typically requires a large amount of computing resources, which may affect real-time detection performance, especially in high-traffic areas;

[0008] -Limited generalization: Most existing methods are tailored to specific environments or road types, making it difficult to generalize to different traffic scenarios or areas with different driving behaviors;

[0009] - Lack of multimodal data integration: Many existing methods rely on single data sets, which limits their ability to detect accurately.

[0010] Summary of the Invention

[0011] This disclosure presents an innovative automatic traffic accident detection technology that can effectively detect traffic accidents in traffic videos and provide specific information related to the accidents, with higher accuracy and robustness compared to traditional methods.

[0012] According to one aspect of this disclosure, a method for detecting an accident is provided, comprising: receiving an input video; identifying a plurality of objects in the video; determining that a particular object among the plurality of objects has potentially been involved in an accident if it conforms to a predetermined rule that transitions from a moving state to a stopped state and ultimately remains in a stopped state; determining frame images in the video in which the particular object has potentially been involved in an accident; and instructing a trained image processing model to analyze and process the frame images to determine whether the particular object has been involved in an accident.

[0013] The predetermined rules include: the position of the specific object does not change in multiple frames of the video, wherein the multiple frames correspond to a time window; the distance between the positions of the specific object in the following two frames is greater than a first threshold: the first frame in which the specific object first appears in the video, and the second frame corresponding to the start of the time window; and the distance between the positions of the specific object in the following two frames is less than a second threshold: the third frame corresponding to the end of the time window, and the fourth frame in which the specific object last appears in the video.

[0014] The position of the specific object in multiple frames of the video is determined to be unchanged by the following method: dividing any two frames corresponding to the time window into a group to obtain multiple groups; determining the distance between the positions of the specific object in the two frames contained in each group; selecting groups whose determined distance is less than a third threshold; and determining that the position of the specific object in the multiple frames is unchanged if the ratio of the number of selected groups to the number of multiple groups is greater than a fourth threshold.

[0015] The process of determining the frame image of the potential accident involving the specific object further includes: detecting whether each other object appearing in one or more frames between the first frame and the second frame, besides the specific object, conforms to a predetermined rule for transitioning from a moving state to a stopped state and ultimately remaining in a stopped state; if no other object is detected conforming to the predetermined rule, determining that the second frame is the frame of the potential accident involving the specific object; if one or more other objects are detected conforming to the predetermined rule, determining the distance between each detected other object and the specific object, and if the minimum distance among the determined distances is less than a fifth threshold, determining that the frame in which the minimum distance occurs is the frame of the potential accident involving the specific object.

[0016] The above method further includes: instructing the trained image processing model to analyze and process the frame image to output information related to the accident, wherein the information related to the accident includes one or more of the following: the time of the accident, the duration of the accident, the identifier or image of the frame in which the accident occurred, a description of the accident scene, and the possible causes of the accident.

[0017] The trained image processing model includes a multimodal large language model.

[0018] The above method further includes: detecting multiple objects in the frame image; clustering the detected multiple objects so that objects that are close to each other in the frame image are included in an image patch; and inputting one or more image patches obtained by clustering and the frame image into the trained image processing model for analysis and processing.

[0019] Clustering the detected objects further includes: a) determining the bounding box of each of the detected objects in the frame image, and determining the largest bounding box; b) determining one or more adjacent bounding boxes whose distance from the largest bounding box is less than a sixth threshold; c) including the largest bounding box and the one or more adjacent bounding boxes in an image patch; d) determining the largest bounding box among the bounding boxes that have not yet been included in any image patch; e) iteratively performing steps b)-d) until there are no bounding boxes that have not yet been included in any image patch.

[0020] The method further includes: receiving the video from the user via a large language model; and providing the user with information related to the accident generated by the trained image processing model via the large language model.

[0021] According to another aspect of this disclosure, an apparatus for detecting accidents is provided, comprising: a memory storing a program, and one or more processors. The processors are configured to perform the following operations by executing the program: receiving input video; identifying a plurality of objects in the video; determining that a particular object among the plurality of objects has potentially experienced an accident if it conforms to a predetermined rule of transitioning from a moving state to a stopped state and ultimately remaining in a stopped state; determining frame images in the video in which the particular object has potentially experienced an accident; and instructing a trained image processing model to analyze and process the frame images to determine whether the particular object has experienced an accident.

[0022] According to another aspect of this disclosure, a non-transitory computer-readable medium storing a program that, when executed by a computer, causes the computer to perform the methods described above. Attached Figure Description

[0023] Figure 1 schematically illustrates a block diagram of a system for automatically detecting traffic accidents according to the present disclosure.

[0024] Figure 2 shows an overall flowchart of a method for automatically detecting traffic accidents according to this disclosure.

[0025] Figure 3 shows a flowchart of a method for clustering objects in an image according to the present disclosure.

[0026] Figure 4 schematically illustrates the results of clustering processing according to this disclosure.

[0027] Figure 5 shows an exemplary configuration block diagram of computer hardware that implements the present disclosure. Detailed Implementation

[0028] Figure 1 schematically illustrates a block diagram of a system for automatically detecting traffic accidents according to the present disclosure. Figure 2 shows an overall flowchart of a method for automatically detecting traffic accidents according to the present disclosure.

[0029] Referring to Figures 1 and 2, firstly, in step S210, the input video to be processed is received. As an example, the video can be traffic video captured by a surveillance camera, or it can be a video containing traffic-related content captured using a mobile terminal.

[0030] In step S220, objects appearing in the received video are identified. For example, objects may include vehicles such as cars and ships, and pedestrians. Steps S210 and S220 can be performed by the object tracking module 110.

[0031] In step S230, for each object identified in the video, it is determined whether the object is potentially involved in a traffic accident based on predetermined rules. Specifically, if an object meets predetermined rules that it transitions from a moving state to a stopped state and ultimately remains in a stopped state, it is determined that the object has potentially been involved in an accident.

[0032] In step S240, frames in the video where the object is potentially involved in an accident are identified. Steps S230 and S240 can be performed by the potential accident detection module 120.

[0033] Then, in step S250, the image of the frame in which the object may have been involved in an accident is input to the trained image processing model 130, and the image processing model 130 analyzes and processes the frame image to further determine whether the object has been involved in an accident.

[0034] As shown in Figure 1, the object tracking module 110, the potential accident detection module 120, and the image processing model 130 constitute the core modules of the system according to this disclosure. Furthermore, the system may also include an interaction module 140. The interaction module 140 can act as an intermediary between the user and the core modules, for inputting user-provided traffic video into the core modules and providing the user with the accident detection results generated by the core modules.

[0035] The interaction module 140 may include a Large Language Model (LLM), tools, and datasets. The LLM is responsible for interacting with the user, using natural language processing techniques to understand user input requests and provide processing results. Tools represent auxiliary tools used for LLM interaction, such as algorithm execution and data analysis tools. Datasets may include, for example, traffic videos and accident data.

[0036] It should be noted that the LLM-based interaction module 140 provided in this disclosure is to facilitate user interaction. However, the interaction module 140 is not necessary for the implementation of the technology of this disclosure. Those skilled in the art can use other suitable human-computer interaction units.

[0037] The following will describe each module shown in Figure 1 in detail.

[0038] Object tracking module 110

[0039] Regarding the object tracking module 110, those skilled in the art can implement this module using any known object detection and tracking techniques, and this disclosure does not impose any limitations on it. As an example, the ByteTrack algorithm is used in this disclosure. ByteTrack can detect abnormal behavior of objects, such as sudden stops, collisions, or deviations, by tracking them.

[0040] First, ByteTrack uses a detection model (e.g., YOLO) to detect all objects (e.g., vehicles, pedestrians) appearing in the video. It can produce detection results with both high and low confidence levels.

[0041] After detection, ByteTrack identifies objects appearing in consecutive frames by assigning each object a unique ID. It uses the object's motion information (such as velocity and direction of motion) and appearance (such as shape and size) to achieve matching and tracking in consecutive frames.

[0042] A key feature of ByteTrack is its ability to track not only high-confidence detected objects but also low-confidence detected objects. This allows ByteTrack to perform tracking even when objects are partially occluded or less obvious, making it more robust in complex traffic conditions.

[0043] In addition, if an object disappears within a short period of time (e.g., is completely obscured or removed from view), ByteTrack will attempt to re-identify it when it reappears, for example by matching it with previously tracked objects.

[0044] Potential accident detection module 120

[0045] The potential accident detection module 120 determines whether an object identified from the video is potentially involved in a traffic accident in the video, based on whether the object conforms to predetermined rules that allow it to transition from a moving state to a stopped state and ultimately remain in a stopped state. Specifically, in this disclosure, the predetermined rules may include the following three aspects.

[0046] Firstly, the object's position remains unchanged across multiple frames corresponding to a time window (e.g., 5 seconds). That is, the object remains stationary within that time window. It's important to note that "position not changing" does not necessarily require the object's displacement to be zero. In this disclosure, the determination that the object's position remains unchanged across multiple frames corresponding to the time window is made in the following manner.

[0047] First, any two frames from a set of frames corresponding to a time window are grouped together, resulting in multiple groups. The distance between the positions of the object in the two frames within each group is determined, and groups with distances less than a threshold (e.g., 5 pixels) are selected. Then, the ratio of the number of selected groups to the total number of these multiple groups is determined, and if the ratio is greater than a threshold (e.g., 90%), it is determined that the object's position within these multiple frames has not changed.

[0048] For each identified object, the potential accident detection module 120 searches for a time window in the video. As an example, a window of predetermined length (e.g., 5 seconds) can be set, and this window can be slid along the video's timeline in units of one or more frames, sequentially determining whether the object's position has changed for each window. If a time window in the video is found where the object's position has not changed, the potential accident detection module 120 can determine that the object meets a first aspect of a predetermined rule. At this time, the frames corresponding to the start and end of the found time window can also be determined.

[0049] As a second aspect of the predetermined rule, the distance between the positions of an object in the following two frames is greater than a threshold (e.g., 20 pixels): the first frame in which the object first appears in the video, and the second frame corresponding to the start of the time window. That is, from the initial appearance of the object to the start of the time window, the object has a large displacement and is therefore in a moving state.

[0050] If an object meets the first aspect of a predetermined rule, the potential accident detection module 120 can determine whether the object meets the second aspect based on a determined time window.

[0051] As a third aspect of the predetermined rule, the distance between the positions of an object in the following two frames is less than a threshold (e.g., 5 pixels): the third frame corresponding to the end of the time window, and the fourth frame in which the object last appears in the video. That is, from the end of the time window until the object's last appearance, the object's displacement is very small, and therefore it is in a stationary state.

[0052] If an object meets the first aspect of a predetermined rule, the potential accident detection module 120 can determine whether the object meets the third aspect based on a determined time window.

[0053] Combining the first to third aspects, it can be concluded that the object is moving before the start of the time window, then the object is stationary within the time window, and remains stationary until the object appears last in the video. If the potential accident detection module 120 determines that the object conforms to the first to third aspects of the predetermined rules, it can be determined that the object has potentially been involved in an accident, because if an object such as a vehicle suddenly stops while moving, it is very likely that an accident has occurred.

[0054] Furthermore, based on common sense, collisions occur in the vast majority of traffic accidents, and there is usually a certain amount of time between the moment of collision and when the object comes to a stop. Therefore, the potential accident detection module 120 can also determine the frame at which an object is potentially involved in an accident.

[0055] Specifically, since the collision occurs before the object stops (i.e., before the time window), the potential accident detection module 120 identifies other objects besides the object that appear in one or more frames between the first frame (the frame in which the object first appears in the video) and the second frame (the frame corresponding to the start of the time window), and determines whether each other object conforms to the predetermined rules mentioned above during the time period from the first frame to the end of the video.

[0056] If no other object matches the predetermined rules, this usually corresponds to a situation where the object has a single-vehicle accident (e.g., the object hits a building, tree, etc.), in which case the second frame can be identified as the frame where the object is potentially involved in an accident.

[0057] If one or more other objects are detected that conform to a predetermined rule (i.e., change from motion to rest), this typically corresponds to a collision between that object and another object (e.g., a pedestrian or vehicle). In this case, the distance between each detected other object and the current object is determined, and if the minimum distance among the determined distances is less than a certain threshold, the frame in which the minimum distance occurs is determined to be the frame in which a potential accident has occurred for that object. Furthermore, it can be determined that a collision has occurred between the other object that generated this minimum distance and the current object.

[0058] The potential accident detection module 120 performs the above-described processing on each object identified from the video to determine whether the object has potentially been involved in an accident and the frame in which the accident may have occurred, or it may also determine information such as other objects involved in an accident with the object.

[0059] Image processing model 130

[0060] The accidents detected by the potential accident detection module 120 according to predetermined rules may not be accurate enough. For example, it may fail to detect an accident in a video containing a traffic accident (missed detection) or detect an accident in a video without an accident (false detection). Therefore, in this disclosure, an image processing model 130 is used to further optimize the detection results to improve the detection accuracy. Specifically, the images of frames in which the object is potentially involved in an accident are input into the image processing model 130 to further determine whether the object has been involved in an accident in the video.

[0061] As an example, this disclosure utilizes a multimodal large language model (MLLM) to implement an image processing model 130. Unlike traditional large language models (LLMs) that only process text, MLLMs can combine and process information from different modalities (e.g., text, images, audio, and video) to enhance understanding and improve interactivity. For example, an MLLM can answer questions about images, describe what it sees, and even generate images based on given prompts. Therefore, MLLMs are particularly useful for applications such as visual question answering, content generation, and multimodal interaction agents.

[0062] The main advantage of MLLM lies in its ability to provide richer, more context-aware responses. By incorporating various types of information, MLLM can perform complex tasks and provide a more intuitive and general user experience. Therefore, it is beneficial to leverage the powerful understanding and analysis capabilities of MLLM to optimize incident detection results. The techniques of this disclosure will be described below using MLLM as an example of an image processing model 130 (referred to as MLLM 130). However, it should be noted that this disclosure is not limited thereto, and those skilled in the art can implement image processing model 130 using other trained models.

[0063] The MLLM 130 performs image understanding, analysis, and processing based on received frame images of objects potentially involved in accidents and image patches obtained by clustering objects in those frame images (described below) to determine whether an accident has actually occurred. The MLLM 130 can produce detection results that are the same as or opposite to those of the potential accident detection module 120. Furthermore, if the MLLM 130 determines that an accident has occurred, it can output more information related to the accident, such as the time of the accident, the duration of the accident, the identifier or image of the frame in which the accident occurred, a description of the accident scene, possible causes of the accident, etc.

[0064] In traditional techniques using MLLMs, the image to be processed is typically divided into multiple blocks, and these blocks are then fed into the MLLM for processing. However, since the content of interest often exists only in certain regions of the image rather than throughout the entire image, some of the divided blocks may contain irrelevant information that is not of interest, and traditional techniques also feed such blocks into the MLLM.

[0065] To reduce the number of image patches to be processed and improve efficiency, this disclosure detects multiple objects in a frame image where an accident may have occurred (e.g., using YOLO), and clusters the detected objects so that objects that are close to each other in the frame image are grouped into one image patch. The frame image, along with one or more image patches obtained through clustering, is then input into an MLLM 130 for analysis and processing.

[0066] The following description, in conjunction with Figure 3, will illustrate the process of clustering objects.

[0067] As shown in Figure 3, in step S310, the bounding box of each of the detected multiple objects is determined, and the largest bounding box is determined. As an example, the bounding box of each object can be initially marked as "unused", while the determined largest bounding box can be marked as "used".

[0068] In step S320, one or more adjacent bounding boxes are determined whose distance from the maximum bounding box is less than a certain threshold. For example, the distance between the center of the determined adjacent bounding box and the center of the maximum bounding box is less than a predetermined percentage (e.g., 15%) of the image length. For example, the determined adjacent bounding boxes can then be marked as "used".

[0069] In step S330, the largest bounding box and the determined adjacent bounding boxes are included in an image patch. As an example, the boundary of the image patch can be determined by the smallest bounding rectangle of the individual bounding boxes contained within it.

[0070] In step S340, it is determined whether there are any bounding boxes in the image that have not yet been included in any image patch. As an example, in step S340, it is determined whether there are any bounding boxes in the image that are still marked as "unused".

[0071] When there are bounding boxes that are not yet included in an image patch, the largest bounding box is determined among these bounding boxes, as shown in step S350. Then, steps S320 and S330 are performed again based on the currently determined largest bounding box. That is, among these bounding boxes that are not yet included in an image patch (marked as "unused"), the largest bounding box and its nearest neighboring bounding boxes are determined, and they are included in an image patch. As an example, they can be marked as "used" at this time.

[0072] Furthermore, if the result of step S340 is that there are no bounding boxes that have not yet been included in the image patch, then the process ends.

[0073] Steps S320-S350 are performed iteratively as described above until there are no bounding boxes in the image that have not yet been included in any image block (i.e., there are no bounding boxes marked as "unused").

[0074] Figure 4 schematically illustrates the results of clustering objects according to this disclosure. Figure 4 shows an image in which three image patches were obtained by clustering the objects (e.g., cars, pedestrians) appearing in the image. Each image patch includes one or more objects. In Figure 4, the small rectangle surrounding each object represents the bounding box of that object, and the large rectangle surrounding multiple objects represents the boundary of the image patch.

[0075] In conventional techniques, all blocks obtained after dividing an image are typically input into an MLLM. Taking the image shown in Figure 4 as an example, conventional techniques might divide the image into 3*4 blocks. Blocks at the bottom of the image, which do not contain the object of interest, are still input into the MLLM, thus reducing its processing efficiency. In contrast, according to this disclosure, three image blocks containing the object of interest are input into the MLLM 130. This not only reduces the processing load on the MLLM 130 but also provides it with more targeted material, thereby facilitating accurate understanding and analysis by the MLLM 130.

[0076] Interactive Module 140

[0077] Referring to Figure 1, the interaction module 140, implemented based on the Large Language Model (LLM), provides an interface for user interaction. Through this interface, users can upload traffic videos to be analyzed and obtain accident detection results generated by the core module, such as whether an accident occurred and specific accident information. By providing the interaction module 140, even users without any programming skills can interact with the system using natural language, greatly enhancing the user experience.

[0078] Tables 1-3 below illustrate the performance of the technology according to this disclosure. This performance was obtained by testing on a dataset comprising 19 long videos, including 12 videos with incidents and 7 videos without incidents. In Tables 1-3, the accuracy A / B indicates that A out of B videos correctly detected the presence or absence of incidents, and the redundancy C / D indicates that D incidents existed in the test dataset, but C incidents were detected.

[0079] [Table 1]

[0080] [Table 2]

[0081] [Table 3]

[0082] As shown in Tables 1-3, by further optimizing the detection results using MLLM, the accuracy and redundancy of accident detection were generally improved. In particular, the redundancy was significantly reduced, indicating that a large number of detection results that were incorrectly detected as traffic accidents could be removed by using MLLM.

[0083] It should be noted that while exemplary values ​​for various parameters (such as thresholds and time window lengths) have been provided in the above description, this disclosure is not limited to these values. Those skilled in the art may use other values ​​based on actual requirements or experience. Furthermore, although the technology of this disclosure has been described above using vehicles and pedestrians as examples of traffic objects, this disclosure is not limited thereto. For example, the technology according to this disclosure can also be applied when processing traffic videos containing ships, trains, etc.

[0084] The technology of this disclosure has been described above in conjunction with specific embodiments. This disclosure proposes an innovative automatic traffic accident detection technology that combines object tracking technology and trained image processing models (such as MLLM) to automatically detect traffic accidents in videos with higher accuracy and robustness, and also provides specific information related to the accident.

[0085] The methods described in the above embodiments can be implemented by software, hardware, or a combination of software and hardware. Programs included in the software can be stored beforehand in a storage medium located internally or externally to the device. As an example, during execution, these programs are written to random access memory (RAM) and executed by a processor (e.g., a CPU) to implement the various methods and processes described herein.

[0086] This disclosure provides computer program code and computer program products for implementing the methods described above, as well as a computer-readable storage medium on which the computer program code is recorded.

[0087] Figure 5 shows an example configuration block diagram of computer hardware for performing the methods of this disclosure according to a program. This computer hardware is an example of an apparatus for detecting incidents according to this disclosure.

[0088] As shown in Figure 5, in computer 500, central processing unit (CPU) 501, read-only memory (ROM) 502 and random access memory (RAM) 503 are connected to each other via bus 504.

[0089] The input / output interface 505 is further connected to the bus 504. The input / output interface 505 is connected to the following components: an input device 506 formed by a keyboard, mouse, microphone, etc.; an output device 507 formed by a display, speaker, etc.; a storage device 508 formed by a hard disk, non-volatile memory, etc.; a communication device 509 formed by a network interface card (such as a local area network (LAN) card, modem, etc.); and a driver 510 for driving a removable medium 511, such as a disk, optical disk, magneto-optical disk, or semiconductor memory.

[0090] In a computer with the above structure, the CPU 501 loads the program stored in the storage device 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program to perform the method described above.

[0091] The program to be executed by the computer (CPU 501) can be recorded on a removable medium 511, which is formed as a packaging medium, such as a disk (including a floppy disk), an optical disk (including a compact optical disk-read-only memory (CD-ROM)), a digital multifunction optical disk (DVD), etc.), a magneto-optical disk, or a semiconductor memory. Furthermore, the program to be executed by the computer (CPU 501) can also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0092] When the removable medium 511 is installed in the drive 510, the program can be installed in the storage device 508 via the input / output interface 505. Alternatively, the program can be received by the communication device 509 via a wired or wireless transmission medium and installed in the storage device 508. Alternatively, the program can be pre-installed in the ROM 502 or the storage device 508.

[0093] A program executed by a computer may be a program that performs processing in the order described in this disclosure, or it may be a program that performs processing in parallel or when needed (such as when invoked).

[0094] The modules or devices described in this disclosure are for logical purposes only and do not strictly correspond to physical devices or entities. For example, the function of each module described in this disclosure may be implemented by multiple physical entities, or the function of multiple modules described in this disclosure may be implemented by a single physical entity. Furthermore, features, components, elements, steps, etc., described in one embodiment are not limited to that embodiment, but can also be applied to other embodiments, such as replacing specific features, components, elements, steps, etc., in other embodiments, or in combination with them.

[0095] The scope of this disclosure is not limited to the specific embodiments described herein. Those skilled in the art will understand that various modifications or variations can be made to the embodiments described herein, depending on design requirements and other factors, without departing from the principles of this disclosure. The scope of this disclosure is defined by the appended claims and their equivalents.

Claims

1. A method for detecting accidents, comprising: Receive input video; Multiple objects were identified in the video; If a particular object among the plurality of objects conforms to a predetermined rule that transitions from a moving state to a stopped state and ultimately remains in a stopped state, it is determined that the particular object has potentially experienced an accident. Identify the frame images in the video in which a potential accident may occur involving the specific object; The instruction is to use a trained image processing model to analyze and process the frame image to determine whether an accident has occurred to the specific object.

2. The method according to claim 1, wherein, The predetermined rules include: The position of the specific object remains unchanged across multiple frames of the video, wherein the multiple frames correspond to a time window; The distance between the positions of the specific object in the following two frames is greater than a first threshold: the first frame in which the specific object first appears in the video, and the second frame corresponding to the start of the time window; and The distance between the positions of the specific object in the following two frames is less than a second threshold: the third frame corresponding to the end of the time window, and the fourth frame in which the specific object last appears in the video.

3. The method according to claim 2, wherein, The position of the specific object remained unchanged across multiple frames of the video by the following method: Any two frames among the multiple frames corresponding to the time window are grouped together to obtain multiple groups; Determine the distance between the positions of the specific object in the two frames contained in each group; Select groups whose determined distance is less than the third threshold; If the ratio of the number of selected groups to the number of said multiple groups is greater than a fourth threshold, it is determined that the position of the specific object in said multiple frames has not changed.

4. The method according to claim 2, wherein, The frame images used to determine the potential occurrence of an accident on the specific object also include: Detect whether every other object appearing in one or more frames between the first frame and the second frame, except for the specific object, conforms to a predetermined rule that transitions from a moving state to a stopped state and ultimately remains in a stopped state; If no other object is detected that conforms to the predetermined rule, the second frame is determined to be the frame in which the specific object is potentially involved in an accident; If one or more other objects are detected that match the predetermined rules. Determine the distance between each other detected object and the specific object; If the minimum distance among the determined distances is less than the fifth threshold, the frame in which the minimum distance occurs is determined to be the frame in which an accident may occur for the specific object.

5. The method according to claim 1, further comprising: The trained image processing model described in the instruction analyzes and processes the frame image to output information related to the accident. The information related to the accident includes one or more of the following: the time of the accident, the duration of the accident, the identifier or image of the frame in which the accident occurred, a description of the accident scene, and the possible causes of the accident.

6. The method according to claim 5, wherein, The trained image processing model includes a multimodal large language model.

7. The method according to claim 5, further comprising: Detect multiple objects in the frame image; Cluster the detected objects so that objects that are close to each other in the frame image are included in one image patch; One or more image patches obtained through clustering, along with the frame image, are input into the trained image processing model for analysis and processing.

8. The method according to claim 7, wherein, Clustering multiple detected objects also includes: a) Determine the bounding box of each of the multiple objects detected in the frame image, and determine the largest bounding box; b) Identify one or more neighboring objects whose distance from the largest bounding box is less than the sixth threshold. The bounding box; c) Include the largest bounding box and the one or more adjacent bounding boxes in an image patch; d) Determine the largest bounding box among those not yet included in any image patch; e) Iteratively execute steps b)-d) until there are no bounding boxes that have not yet been included in any image patch.

9. The method according to claim 5, further comprising: The video is received from the user via a large language model; Information related to the accident, generated by the trained image processing model, is provided to the user via the large language model.

10. An apparatus for detecting accidents, comprising: A memory that stores programs; as well as One or more processors, the processors being configured to perform the following operations by executing the program: Receive input video; Multiple objects were identified in the video; If a particular object among the plurality of objects conforms to a predetermined rule that transitions from a moving state to a stopped state and ultimately remains in a stopped state, it is determined that the particular object has potentially experienced an accident. Identify the frame images in the video in which a potential accident may occur involving the specific object; The instruction is to use a trained image processing model to analyze and process the frame image to determine whether an accident has occurred to the specific object.

11. The apparatus according to claim 10, wherein, The predetermined rules include: The position of the specific object remains unchanged across multiple frames of the video, wherein the multiple frames correspond to a time window; The distance between the positions of the specific object in the following two frames is greater than a first threshold: the first frame in which the specific object first appears in the video, and the second frame corresponding to the start of the time window; and The distance between the positions of the specific object in the following two frames is less than a second threshold: the third frame corresponding to the end of the time window, and the fourth frame in which the specific object last appears in the video.

12. The apparatus according to claim 11, wherein, The processor is configured to determine that the position of the specific object remains unchanged across multiple frames of the video by: Any two frames among the multiple frames corresponding to the time window are grouped together to obtain multiple groups; Determine the distance between the positions of the specific object in the two frames contained in each group; Select groups whose determined distance is less than the third threshold; If the ratio of the number of selected groups to the number of said multiple groups is greater than a fourth threshold, it is determined that the position of the specific object in said multiple frames has not changed.

13. The apparatus according to claim 11, wherein, The processor is also configured to perform the following operations: Detect whether every other object appearing in one or more frames between the first frame and the second frame, except for the specific object, conforms to a predetermined rule that transitions from a moving state to a stopped state and ultimately remains in a stopped state; If no other object is detected that conforms to the predetermined rule, the second frame is determined to be the frame in which the specific object is potentially involved in an accident; If one or more other objects are detected that match the predetermined rules. Determine the distance between each other detected object and the specific object; If the minimum distance among the determined distances is less than the fifth threshold, the frame in which the minimum distance occurs is determined to be the frame in which an accident may occur for the specific object.

14. The apparatus according to claim 10, wherein, The processor is also configured to perform the following operations: The trained image processing model described in the instruction analyzes and processes the frame image to output information related to the accident. The information related to the accident includes one or more of the following: the time of the accident, the duration of the accident, the identifier or image of the frame in which the accident occurred, a description of the accident scene, and the possible causes of the accident.

15. The apparatus according to claim 14, wherein, The trained image processing model includes a multimodal large language model.

16. The apparatus according to claim 14, wherein, The processor is also configured to perform the following operations: Detect multiple objects in the frame image; Cluster the detected objects so that objects that are close to each other in the frame image are included in one image patch; One or more image patches obtained through clustering, along with the frame image, are input into the trained image processing model for analysis and processing.

17. The apparatus according to claim 16, wherein, The processor is also configured to perform the following operations: a) Determine the bounding box of each of the multiple objects detected in the frame image, and determine the largest bounding box; b) Identify one or more adjacent bounding boxes whose distance from the largest bounding box is less than the sixth threshold; c) Include the largest bounding box and the one or more adjacent bounding boxes in an image patch; d) Determine the largest bounding box among those not yet included in any image patch; e) Iteratively execute steps b)-d) until there are no bounding boxes that have not yet been included in any image patch.

18. The apparatus according to claim 14, wherein, The processor is also configured to perform the following operations: The video is received from the user via a large language model; Information related to the accident, generated by the trained image processing model, is provided to the user via the large language model.

19. A non-transitory computer-readable medium storing a program, which, when executed by a computer, causes the computer to perform a method for detecting an incident according to any one of claims 1-9.

Images