Method and apparatus for recognizing abnormal object in elevator on basis of vision-based neural network
By installing cameras at different locations inside the elevator car to collect video and fusing feature recognition, combined with weighing sensor correction scores, the problem of low accuracy in electric vehicle identification was solved, thus improving elevator safety.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- GUANGDONG XINTUO NETWORK TECHNOLOGY CO LTD
- Filing Date
- 2024-12-17
- Publication Date
- 2026-06-04
AI Technical Summary
In existing technologies, the visual neural network inside elevators is easily obstructed by passengers when identifying electric vehicles, which reduces the accuracy of identification and often misidentifies electric vehicles as ordinary bicycles, posing a safety hazard.
A dual-input electric vehicle recognition neural network is adopted. Video is collected by cameras at different locations inside the elevator car, and feature extraction and recognition are fused. The weight inside the car is obtained by weighing sensors, and the recognition score is corrected to ensure accuracy.
This improves the accuracy and reliability of electric vehicle identification, eliminates the safety hazards of electric vehicles entering elevators, and ensures the safety performance of elevators.
Smart Images
Figure CN2024139917_04062026_PF_FP_ABST
Abstract
Description
Method and apparatus for identifying abnormal objects in elevators based on visual neural networks
[0001] This application claims priority to Chinese Patent Application No. 202411737502.2, filed with the Chinese Patent Office on November 29, 2024, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of elevator safety technology, and for example to a method and apparatus for identifying abnormal objects inside an elevator based on a visual neural network. Background Technology
[0003] Elevators are usually installed in high-rise residential buildings to facilitate residents going up and down floors and improve travel efficiency. However, with the convenience of electric vehicles, residents often push them into elevators, posing safety hazards such as battery fires and explosions.
[0004] In related technologies, after the camera inside the elevator captures images of the car, a visual neural network is used to identify whether there is an electric vehicle in the car. If so, a voice alarm is triggered and the car door and hall door are opened to prompt the resident to push the electric vehicle out of the car.
[0005] However, electric vehicles inside the elevator car are often obscured by passengers, which reduces the accuracy of the visual neural network in recognizing electric vehicles and often leads to misidentification of electric vehicles as ordinary bicycles. Summary of the Invention
[0006] This application provides a method for identifying abnormal objects inside elevators based on visual neural networks, used to identify electric vehicles inside elevators, including:
[0007] When the elevator car door opens, the system controls the first camera inside the elevator car to capture the first video and controls the second camera to capture the second video. The first camera and the second camera are cameras located at different positions inside the elevator car.
[0008] The first video and the second video are input into the electric vehicle recognition neural network to obtain the first score of whether an electric vehicle exists in the elevator car.
[0009] Determine whether the first score is greater than a preset first score threshold;
[0010] If so, confirm that an electric vehicle is present in the elevator car and generate an alarm message;
[0011] If not, when the first score is greater than the preset second score threshold, the weighing sensor is activated to obtain the first weight of all objects in the elevator car, and the second score threshold is less than the first score threshold.
[0012] The second weight of the passenger inside the elevator car is obtained by inputting at least one of the first video and the second video into the weight prediction model.
[0013] The first score is corrected using the first weight and the second weight to obtain the second score;
[0014] Determine whether the second score is greater than the first score threshold;
[0015] If so, proceed with the step of determining that an electric vehicle exists inside the elevator car and generating an alarm message;
[0016] If not, it is determined that there is no electric vehicle inside the elevator car.
[0017] Optionally, the electric vehicle recognition neural network includes a first feature extraction network, a second feature extraction network, a feature fusion network, a first detection head, and a second detection head. The first video and the second video are input into the electric vehicle recognition neural network to obtain a first score indicating the presence of an electric vehicle in the elevator car, including:
[0018] The first video is input into the first feature extraction network, and the first feature map is extracted from the first video in the first feature extraction network.
[0019] The second video is input into the second feature extraction network, and the second feature map is extracted from the second video in the second feature extraction network.
[0020] The first feature map and the second feature map are fused in the feature fusion network to obtain fused features;
[0021] The first detection head and the second detection head respectively identify the fused features to obtain a first initial score for the presence of electric vehicles in the first video and a second initial score for the presence of electric vehicles in the second video;
[0022] The average of the first initial score and the second initial score is calculated as the first score of the electric vehicle present in the elevator car.
[0023] Optionally, the electric vehicle recognition neural network is trained in the following manner:
[0024] First and second images of the elevator car taken from different angles are used as training images. The first image is labeled with a third score indicating the presence of an electric vehicle and a first occlusion rate of the electric vehicle. The second image is labeled with a fourth score indicating the presence of an electric vehicle and a second occlusion rate of the electric vehicle.
[0025] The first image and the second image are input into the electric vehicle recognition neural network to obtain the fifth score of the presence of an electric vehicle in the first image and the sixth score of the presence of an electric vehicle in the second image.
[0026] The loss rate is calculated using the third score, fourth score, fifth score, sixth score, first occlusion rate, and second occlusion rate.
[0027] Determine whether the preset training conditions are met;
[0028] If so, confirm that the electric vehicle recognition neural network has completed training;
[0029] If not, adjust the network parameters of the electric vehicle recognition neural network according to the loss rate, and return to the step of inputting the first image and the second image into the electric vehicle recognition neural network.
[0030] Optionally, the loss rate is calculated using the third score, fourth score, fifth score, sixth score, first occlusion rate, and second occlusion rate, including:
[0031] The first loss rate is calculated using the third and fifth scores, and the second loss rate is calculated using the fourth and sixth scores;
[0032] Calculate the first product of the first loss rate and the first occlusion rate, and calculate the second product of the second loss rate and the second occlusion rate;
[0033] The loss rate is obtained by calculating the sum of the first product and the second product.
[0034] Optionally, the weight prediction model is trained in the following manner:
[0035] Obtain a sample image set, wherein the sample images in the sample image set are images of the elevator car with passengers inside, and each sample image is labeled with a third weight;
[0036] The sample image is input into the weight prediction model to obtain the passenger's fourth weight;
[0037] The loss rate is calculated using the third weight and the fourth weight.
[0038] Determine whether the preset training conditions are met;
[0039] If so, confirm that the weight prediction model has completed training;
[0040] If not, adjust the model parameters of the weight prediction model according to the loss rate, and return to the step of inputting the sample image into the weight prediction model to obtain the passenger's second weight.
[0041] Optionally, a second score is obtained by correcting the first score using the first weight and the second weight, including:
[0042] Calculate the correction coefficient based on the first weight and the second weight;
[0043] The second score is obtained by multiplying the correction coefficient by the first score.
[0044] Optionally, a correction coefficient is calculated based on the first weight and the second weight, including:
[0045] Calculate the difference between the first weight and the second weight;
[0046] Determine whether the difference is greater than the value 0;
[0047] If not, proceed to the step of determining that there is no electric vehicle in the elevator car;
[0048] If so, when the difference is greater than a preset weight threshold, calculate the ratio of the difference to the preset weight;
[0049] The correction coefficient is determined using the ratio in a pre-configured ratio-coefficient table.
[0050] This application provides a device for identifying abnormal objects inside an elevator based on a visual neural network, used to identify electric vehicles inside the elevator, including:
[0051] The video acquisition module is configured to control a first camera inside the elevator car to acquire a first video and control a second camera to acquire a second video when the elevator car door is opened. The first camera and the second camera are cameras located at different positions inside the elevator car.
[0052] The first score prediction module is configured to input the first video and the second video into the electric vehicle recognition neural network to obtain a first score indicating the presence of an electric vehicle in the elevator car.
[0053] The first score judgment module is configured to determine whether the first score is greater than a preset first score threshold. If yes, the first determination module is executed; otherwise, the first weight acquisition module is executed.
[0054] The first determining module is configured to determine that an electric vehicle exists inside the elevator car and generate an alarm message.
[0055] The first weight acquisition module is configured to activate the weighing sensor to acquire the first weight of all objects in the elevator car when the first score is greater than a preset second score threshold, wherein the second score threshold is less than the first score threshold.
[0056] The second weight prediction module is configured to obtain the second weight of the passenger in the elevator car from at least one of the first video and the second video input weight prediction models.
[0057] The second fraction correction module is configured to correct the first fraction using the first weight and the second weight to obtain a second fraction.
[0058] The second score determination module is configured to determine whether the second score is greater than the first score threshold. If yes, the first determination module is executed; otherwise, the second determination module is executed.
[0059] The second determining module is configured to determine that there is no electric vehicle inside the elevator car.
[0060] This application provides an electronic device, the electronic device comprising:
[0061] At least one processor; and
[0062] A memory communicatively connected to the at least one processor; wherein,
[0063] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for identifying abnormal objects in an elevator based on a visual neural network as described in any of the first aspects of this application.
[0064] This application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method for identifying abnormal objects in an elevator based on a visual neural network, as described in any of the first aspects of this application. Attached Figure Description
[0065] Figure 1 is a flowchart of a method for identifying abnormal objects in an elevator based on a visual neural network according to an embodiment of this application;
[0066] Figure 2 is a schematic diagram of the camera installation inside the elevator car;
[0067] Figure 3 is a flowchart of another method for identifying abnormal objects in an elevator based on a visual neural network according to an embodiment of this application;
[0068] Figure 4 is a schematic diagram of the structure of the electric vehicle recognition neural network in an embodiment of this application;
[0069] Figure 5 is a schematic diagram of the structure of a device for identifying abnormal objects in an elevator based on a visual neural network according to an embodiment of this application;
[0070] Figure 6 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. Detailed Implementation
[0071] Figure 1 is a flowchart of a method for identifying abnormal objects in an elevator based on a visual neural network according to an embodiment of this application. This embodiment is applicable to identifying the presence of an electric vehicle in an elevator. This method can be executed by a device for identifying abnormal objects in an elevator based on a visual neural network. This device can be implemented in hardware and / or software and can be configured in an electronic device. As shown in Figure 1, the method for identifying abnormal objects in an elevator based on a visual neural network includes:
[0072] S101. When the elevator car door opens, control the first camera inside the elevator car to collect the first video and control the second camera to collect the second video. The first camera and the second camera are cameras in different positions inside the elevator car.
[0073] This application embodiment is used to detect whether there is an electric vehicle in the elevator car. The electric vehicle can refer to an electric bicycle or an electric motorcycle. Since passengers usually get on and off the elevator when the elevator door is open, the camera can be triggered to collect video of the elevator car when the elevator door is open.
[0074] In one embodiment, a first camera and a second camera can be installed inside the elevator car. The first camera and the second camera are installed at different locations inside the elevator car. For example, as shown in the top view of the elevator car in Figure 2, the first camera and the second camera can be set at 90°, that is, the first camera and the second camera are installed on two adjacent walls of the elevator car, so that the electric bicycle can be captured at the maximum angle when the electric bicycle enters the elevator car in any posture.
[0075] When the car door is opened, the first camera can capture the first video and the second camera can be controlled to capture the second video.
[0076] S102. Input the first video and the second video into the electric vehicle recognition neural network to obtain the first score of whether an electric vehicle exists in the elevator car.
[0077] This embodiment can pre-train an electric vehicle recognition neural network. This electric vehicle recognition neural network is a dual-stream input. After inputting the first video and the second video of the elevator car collected from different angles, the electric vehicle recognition neural network can identify whether there is an electric vehicle in the elevator car from two angles. This can avoid the electric vehicle being identified from the video collected from the other direction when the electric vehicle is blocked by passengers in one direction. It realizes the fusion of video features from two directions to obtain features of the electric vehicle from multiple angles, thereby improving the accuracy of electric vehicle recognition when the electric vehicle is blocked. The higher the first score, the higher the reliability of electric vehicle recognition.
[0078] S103. Determine whether the first score is greater than the preset first score threshold.
[0079] The first score threshold can be a score that indicates a high reliability of identifying electric vehicles. For example, if 100 is the maximum score, the first score threshold can be 90. The first score can be compared with the first score threshold. If the first score is greater than the first score threshold, S104 can be executed. If the first score is less than the first score threshold, S105 can be executed.
[0080] S104. Confirm the presence of an electric vehicle inside the elevator car and generate an alarm message.
[0081] If the first score is greater than the first score threshold, it is determined that there is an electric vehicle in the elevator car. For example, when a passenger pushes an electric vehicle into the elevator car, and only a few passengers in the elevator car do not obstruct the electric vehicle, the electric vehicle recognition neural network can accurately and reliably identify the electric vehicle through the first video and the second video. If the first score is greater than the first score threshold, the reliability of identifying the electric vehicle is high. It can be directly determined that there is an electric vehicle in the elevator car, generate alarm information, trigger the voice device in the elevator car to play a prompt tone, and control the elevator car door to open and control the elevator car to stop running, so as to prompt the passenger to push the electric vehicle out of the elevator car.
[0082] S105. When the first score is greater than the preset second score threshold, the weighing sensor is activated to obtain the first weight of all objects in the elevator car. The second score threshold is less than the first score threshold.
[0083] The second score threshold can be a score lower than the first score threshold, indicating a high degree of reliability that there are no electric vehicles in the elevator car. For example, if the first score threshold is 90 points, the second score threshold can be 60 points. If the first score is lower than the first score threshold but higher than the second score threshold, it indicates that there is likely an electric vehicle in the elevator car, requiring further confirmation. The total weight of all objects in the elevator car can be obtained using a weighing sensor on the elevator as the first weight. For example, the total weight can be collected by the elevator car's weight sensor, and the car's own weight can be subtracted to obtain the first weight. If the first score is lower than the second score threshold, it indicates a high degree of reliability that there are no electric vehicles in the elevator car, and thus S109 is executed to determine that there are no electric vehicles in the elevator car.
[0084] S106. Obtain the second weight of the passenger in the elevator car from at least one of the first video and the second video in the input weight prediction model.
[0085] This embodiment can pre-train a weight prediction model, which predicts a second weight based on video inside the elevator car. This second weight can be the weight of all objects (passengers and items) inside the elevator car, excluding electric vehicles or suspected electric vehicles.
[0086] S107. The first score is corrected by the first weight and the second weight to obtain the second score.
[0087] In one embodiment, the difference between a first weight and a second weight can be calculated, which represents the weight of a suspected electric vehicle object inside the elevator car. Given that electric vehicles are often misidentified as ordinary bicycles, and the weight of an electric vehicle is usually 3-5 times that of an ordinary bicycle, if the difference is less than 0, S109 can be executed to determine that there is no electric vehicle inside the elevator car. If the difference is greater than 0, the ratio of the difference to a preset weight (the weight of most electric vehicles) can be calculated, and then a correction coefficient can be determined in a pre-configured ratio-coefficient table. The product of the correction coefficient and the first score is calculated to obtain the second score.
[0088] S108. Determine whether the second score is greater than the threshold of the first score.
[0089] For example, after calculating the corrected second score, it can be determined whether the second score is greater than the first score threshold. If so, S104 is executed to determine that there is an electric vehicle in the elevator car and generate an alarm message. If the second score is less than the first score threshold, S109 can be executed.
[0090] S109. Confirm that there are no electric vehicles inside the elevator car.
[0091] If the corrected second score is less than the first score threshold, it is determined that there are no electric vehicles in the elevator car, and the elevator car door can be closed and the elevator car can be allowed to run.
[0092] This application embodiment uses a first camera and a second camera at different locations inside the elevator car to collect a first video and a second video, respectively. The first video and the second video are input into an electric vehicle recognition neural network to obtain a first score indicating the presence of an electric vehicle in the elevator car. It is then determined whether the first score is greater than a preset first score threshold. If so, it is determined that an electric vehicle is present in the elevator car and an alarm message is generated. If not, when the first score is greater than a preset second score threshold, a weighing sensor is activated to obtain the first weight of all objects in the elevator car, where the second score threshold is less than the first score threshold. At least one of the first video and the second video is input into a weight prediction model to obtain the second weight of the passengers in the elevator car. The first score is corrected using the first weight and the second weight to obtain a second score. It is then determined whether the second score is greater than the first score threshold. If so, it is determined that an electric vehicle is present in the elevator car and an alarm message is generated. If not, it is determined that no electric vehicle is present in the elevator car. This system enables the identification of electric vehicles by inputting video footage captured by cameras at different angles into an electric vehicle recognition neural network. This avoids the problem of electric vehicles being unrecognizable or misidentified when they are obstructed, thus improving the accuracy of electric vehicle recognition. Furthermore, when the score is low, weight can be used to correct the score. For example, if an electric vehicle is identified as a regular bicycle, resulting in a low score, weight can be used to correct the score. The corrected score is then used to determine whether an electric vehicle is present. This results in higher accuracy and reliability in identifying electric vehicles, enabling accurate and reliable alerts when electric vehicles enter elevators, eliminating the safety hazards of electric vehicles entering elevators, and improving the safety performance of elevators.
[0093] Figure 3 is a flowchart of a method for identifying abnormal objects in an elevator based on a visual neural network according to an embodiment of this application. This embodiment is an optimization based on the above embodiment. As shown in Figure 3, the method for identifying abnormal objects in an elevator based on a visual neural network includes:
[0094] S301. When the elevator car door opens, control the first camera inside the elevator car to collect the first video and control the second camera to collect the second video. The first camera and the second camera are cameras in different positions inside the elevator car.
[0095] As shown in Figure 2, which is a top view inside the elevator car, the first camera and the second camera can be set at 90° to each other. That is, the first camera and the second camera are installed on two adjacent walls of the elevator car. When the elevator door is open, the first camera can be controlled to capture the first video and the second camera can be controlled to capture the second video.
[0096] S302. Input the first video into the first feature extraction network, extract the first feature map from the first video in the first feature extraction network, input the second video into the second feature extraction network, and extract the second feature map from the second video in the second feature extraction network.
[0097] Figure 4 shows a schematic diagram of the electric vehicle recognition neural network according to an embodiment of this application. The electric vehicle recognition neural network includes a first feature extraction network, a second feature extraction network, a feature fusion network, a first detection head, and a second detection head. The electric vehicle recognition neural network can be trained through the following steps:
[0098] S1. Obtain the first and second images of the elevator car taken from different angles as training images. The first image is labeled with the third score of the presence of electric vehicles and the first occlusion rate of electric vehicles. The second image is labeled with the fourth score of the presence of electric vehicles and the second occlusion rate of electric vehicles.
[0099] S2. Input the first image and the second image into the electric vehicle recognition neural network to obtain the fifth score of the presence of an electric vehicle in the first image and the sixth score of the presence of an electric vehicle in the second image.
[0100] S3. Calculate the loss rate using the third score, fourth score, fifth score, sixth score, first occlusion rate, and second occlusion rate;
[0101] S4. Determine if the preset training conditions are met. If yes, proceed to S5; otherwise, proceed to S6.
[0102] S5. Determine if the electric vehicle recognition neural network has completed training;
[0103] S6. Adjust the network parameters of the electric vehicle recognition neural network according to the loss rate, and return to S2.
[0104] For example, a first image is input into a first feature extraction network to extract a first feature map, a second image is input into a second feature extraction network to extract a second feature map, the first feature map and the second feature map are input into a feature fusion network to perform feature fusion to obtain fused features, and the fused features are input into a first detection head and a second detection head to obtain a fifth score and a sixth score, respectively.
[0105] When calculating the loss rate, a first loss rate (e.g., mean squared error loss) can be calculated using the third and fifth scores, and a second loss rate (e.g., mean squared error loss) can be calculated using the fourth and sixth scores. The first product of the first loss rate and the first occlusion rate is then calculated, as is the second product of the second loss rate and the second occlusion rate. The sum of the first and second products yields the final loss rate. By using the occlusion rate as a weight to calculate the loss rate, and then adjusting the network parameters based on this loss rate, the electric vehicle recognition neural network can learn to recognize occluded electric vehicles.
[0106] After the electric vehicle recognition neural network has been trained and deployed, a first video can be input into a first feature extraction network to extract a first feature map from the first video, and a second video can be input into a second feature extraction network to extract a second feature map from the second video.
[0107] S303. The first feature map and the second feature map are fused in the feature fusion network to obtain the fused features.
[0108] For example, the first occlusion rate can be used as the weight of the first feature map, and the second occlusion rate can be used as the weight of the second feature map. The first feature map and the second feature map can be fused together using the weights to obtain the fused feature.
[0109] S304. The first detection head and the second detection head respectively identify the fused features to obtain a first initial score for the presence of electric vehicles in the first video and a second initial score for the presence of electric vehicles in the second video.
[0110] For example, the fused features can be input into the first detection head to obtain a first initial score indicating the presence of an electric vehicle in the first video, and the fused features can be input into the second detection head to obtain a second initial score indicating the presence of an electric vehicle in the second video.
[0111] S305. Calculate the average of the first initial score and the second initial score as the first score of the electric vehicle inside the elevator car.
[0112] For example, after obtaining the first initial score and the second initial score, the arithmetic mean of the first initial score and the second initial score can be calculated as the first score.
[0113] In another embodiment, the weight of the first camera can be configured as the weight of the first initial score, and the weight of the second camera can be configured as the weight of the second initial score. The weighted average of the weights, the first initial score, and the second initial score is calculated as the first score. For example, if the first camera is set on one of the left and right side walls of the elevator car, the first camera can capture the side of the electric vehicle after it enters the car (the electric vehicle is usually located in the elevator car in the posture when it enters, so a larger weight can be set). The weight of the first camera can be 0.6. The second camera is set on one of the front and rear side walls of the elevator car, and the weight of the second camera can be 0.4.
[0114] S306. Determine whether the first score is greater than the preset first score threshold.
[0115] For example, the first score threshold can be a score that indicates a high reliability of identifying electric vehicles. The first score can be compared with the first score threshold. If the first score is greater than the first score threshold, S307 can be executed. If the first score is less than the first score threshold, S308 can be executed.
[0116] S307. Confirm the presence of an electric vehicle inside the elevator car and generate an alarm message.
[0117] If the first score is greater than the first score threshold, it can be determined that there is an electric vehicle inside the elevator car. An alarm message can be generated, triggering the voice device inside the elevator car to play a prompt tone. The elevator car door can be opened and the elevator car can be stopped to prompt passengers to push the electric vehicle out of the elevator car.
[0118] S308. When the first score is greater than the preset second score threshold, the weighing sensor is activated to obtain the first weight of all objects in the elevator car. The second score threshold is less than the first score threshold.
[0119] The second score threshold can be a score lower than the first score threshold, indicating a high degree of reliability that there are no electric vehicles in the elevator car. If the first score is lower than the first score threshold but higher than the second score threshold, it suggests that an electric vehicle may be present in the elevator car, requiring further confirmation. This can be achieved by using a weighing sensor on the elevator to obtain the total weight of all objects in the elevator car as the first weight, along with the actual weight.
[0120] S309. Obtain the second weight of the passenger in the elevator car from at least one of the first video and the second video in the input weight prediction model.
[0121] In this embodiment, the second weight can be the estimated weight of an object inside the elevator car other than a suspected electric vehicle. For example, it can be predicted using a weight prediction model, which can be trained through the following steps:
[0122] S10. Obtain a sample image set. The sample images in the sample image set are images of the elevator car when there are passengers inside. Each sample image is labeled with a third weight.
[0123] S11. Input the sample image into the weight prediction model to obtain the passenger's fourth weight;
[0124] S12. Calculate the loss rate using the third and fourth weights;
[0125] S13. Determine whether the preset training conditions are met;
[0126] S14. Determine if the weight prediction model has been trained.
[0127] S15. Adjust the model parameters of the weight prediction model according to the loss rate, and return to S11.
[0128] The specific training process can refer to the supervised training methods for neural network models in the existing technology. This embodiment does not limit the training process.
[0129] After the weight prediction model is trained and deployed, at least one of the first video and the second video can be input into the weight prediction model. A frame image is extracted from the input video and input into the weight prediction model to predict the estimated weight of objects (such as passengers) in the car other than the suspected electric vehicle as the second weight.
[0130] S310. Calculate the correction coefficient based on the first weight and the second weight.
[0131] In one embodiment, the first weight is the actual weight, and the second weight is the estimated weight of objects other than the suspected electric vehicle. The difference between the first weight and the second weight can be calculated, and this difference represents the gravity of the suspected electric vehicle. It can be determined whether the difference is greater than 0. If not, S313 can be executed to determine that there is no electric vehicle in the elevator car. If so, when the difference is greater than a preset weight threshold, the ratio of the difference to the preset weight is calculated. The correction coefficient is determined by the ratio in a pre-configured ratio-coefficient table. The preset weight can be the weight of most electric vehicles.
[0132] For example, if the first weight is 640KG and the second weight is 600KG, the difference is 40KG. This difference is greater than the weight threshold of 20KG. The ratio of the difference of 40KG to the preset weight (50KG) can be calculated as 0.8. Then, the correction coefficient corresponding to 0.8 is determined to be 1.2 in the ratio-coefficient table.
[0133] If the first weight is 640KG and the second weight is 630KG, the difference is 10KG. This difference is less than the weight threshold of 20KG. In other words, the difference between the actual first weight of 60KG and the second weight of 630KG (excluding suspected electric vehicles, such as ordinary bicycles) is small. It is possible that it is indeed an ordinary bicycle, and it can be determined that there are no electric vehicles in the elevator car.
[0134] S311. Calculate the product of the correction coefficient and the first score to obtain the second score.
[0135] After obtaining the calibration coefficient, the product of the calibration coefficient and the first score can be calculated to obtain the second score.
[0136] S312. Determine whether the second score is greater than the threshold of the first score.
[0137] For example, after calculating the corrected second score, it can be determined whether the second score is greater than the first score threshold. If so, S307 is executed to determine that there is an electric vehicle in the elevator car and generate an alarm message. If the second score is less than the first score threshold, S313 can be executed.
[0138] S313. Confirm that there are no electric vehicles inside the elevator car.
[0139] If the corrected second score is less than the first score threshold, it is determined that there are no electric vehicles in the elevator car, and the elevator car door can be closed and the elevator car can be allowed to run.
[0140] In this embodiment, the videos captured by the first and second cameras inside the elevator car are respectively input into the first feature extraction network and the second feature map extraction network of the electric vehicle recognition neural network to obtain the first feature map and the second feature map. After being fused by the feature fusion network, the first initial score is output through the first detection head and the second initial score is output through the second detection head. The average of the first initial score and the second initial score is calculated as the first score. It is determined whether the first score is greater than a preset first score threshold. If it is, it is determined that there is an electric vehicle in the elevator car and an alarm message is generated. If not, when the first score is greater than the preset second score threshold, the weighing sensor is activated to obtain the first weight of all objects in the elevator car, where the second score threshold is less than the first score threshold. The second weight of the passenger in the elevator car is obtained by inputting at least one of the first and second videos into the weight prediction model. The first score is corrected using the first weight and the second weight to obtain the second score. It is determined whether the second score is greater than the first score threshold. If it is, it is determined that there is an electric vehicle in the elevator car and an alarm message is generated. If not, it is determined that there is no electric vehicle in the elevator car. This system enables the identification of electric vehicles by inputting video footage captured by cameras at different angles into an electric vehicle recognition neural network. This avoids the problem of electric vehicles being unrecognizable or misidentified when they are obstructed, thus improving the accuracy of electric vehicle recognition. Furthermore, when the score is low, weight can be used to correct the score. For example, if an electric vehicle is identified as a regular bicycle, resulting in a low score, weight can be used to correct the score. The corrected score is then used to determine whether an electric vehicle is present. This results in higher accuracy and reliability in identifying electric vehicles, enabling accurate and reliable alerts when electric vehicles enter elevators, eliminating the safety hazards of electric vehicles entering elevators, and improving the safety performance of elevators.
[0141] Figure 5 is a schematic diagram of a device for identifying abnormal objects in an elevator based on a visual neural network, according to an embodiment of this application. As shown in Figure 5, the device for identifying abnormal objects in an elevator based on a visual neural network includes:
[0142] The video acquisition module 501 is configured to control a first camera inside the elevator car to acquire a first video and control a second camera to acquire a second video when the elevator car door is opened. The first camera and the second camera are cameras located at different positions inside the elevator car.
[0143] The first score prediction module 502 is configured to input the first video and the second video into the electric vehicle recognition neural network to obtain a first score of whether an electric vehicle exists in the elevator car.
[0144] The first score judgment module 503 is configured to determine whether the first score is greater than a preset first score threshold. If yes, the first determination module 504 is executed; otherwise, the first weight acquisition module 505 is executed.
[0145] The first determining module 504 is configured to determine that an electric vehicle exists inside the elevator car and generate alarm information.
[0146] The first weight acquisition module 505 is configured to activate the weighing sensor to acquire the first weight of all objects in the elevator car when the first score is greater than a preset second score threshold, wherein the second score threshold is less than the first score threshold.
[0147] The second weight prediction module 506 is configured to obtain the second weight of the passenger in the elevator car from at least one of the first video and the second video input weight prediction models.
[0148] The second score correction module 507 is configured to correct the first score using the first weight and the second weight to obtain a second score;
[0149] The second score judgment module 508 is configured to determine whether the second score is greater than the first score threshold. If yes, the first determination module 504 is executed; if no, the second determination module 509 is executed.
[0150] The second determining module 509 is configured to determine that there is no electric vehicle inside the elevator car.
[0151] Optionally, the electric vehicle recognition neural network includes a first feature extraction network, a second feature extraction network, a feature fusion network, a first detection head, and a second detection head. The first score prediction module 502 includes:
[0152] The first feature map extraction unit is configured to input the first video into the first feature extraction network and extract the first feature map from the first video in the first feature extraction network;
[0153] The second feature map extraction unit is configured to input the second video into the second feature extraction network and extract the second feature map from the second video in the second feature extraction network.
[0154] The feature fusion unit is configured to fuse the first feature map and the second feature map in the feature fusion network to obtain fused features;
[0155] The score prediction unit is configured to identify the fused features by the first detection head and the second detection head respectively, and obtain a first initial score for the presence of an electric vehicle in the first video and a second initial score for the presence of an electric vehicle in the second video.
[0156] The score calculation unit is configured to calculate the average of the first initial score and the second initial score as the first score of the electric vehicle present in the elevator car.
[0157] Optionally, the electric vehicle recognition neural network is trained using the following modules:
[0158] The training image acquisition module is configured to acquire a first image and a second image of the elevator car taken from different angles as training images. The first image is labeled with a third score indicating the presence of an electric vehicle and a first occlusion rate of the electric vehicle. The second image is labeled with a fourth score indicating the presence of an electric vehicle and a second occlusion rate of the electric vehicle.
[0159] The score prediction module is configured to input the first image and the second image into the electric vehicle recognition neural network to obtain a fifth score indicating the presence of an electric vehicle in the first image and a sixth score indicating the presence of an electric vehicle in the second image.
[0160] The electric vehicle recognition loss rate calculation module is configured to calculate the loss rate using the third score, fourth score, fifth score, sixth score, first occlusion rate, and second occlusion rate.
[0161] The electric vehicle recognition neural network training condition judgment module is set to determine whether the preset training conditions are met. If yes, the electric vehicle recognition neural network training completion determination module is executed; otherwise, the electric vehicle recognition neural network adjustment module is executed.
[0162] The electric vehicle recognition neural network training completion determination module is configured to determine whether the electric vehicle recognition neural network has completed training.
[0163] The electric vehicle recognition neural network adjustment module is configured to adjust the network parameters of the electric vehicle recognition neural network according to the loss rate and return the result to the score prediction module.
[0164] Optionally, the electric vehicle recognition loss rate calculation module includes:
[0165] The first loss rate calculation unit is configured to calculate a first loss rate using the third score and the fifth score, and to calculate a second loss rate using the fourth score and the sixth score.
[0166] The second loss rate calculation unit is configured to calculate a first product of the first loss rate and the first occlusion rate, and to calculate a second product of the second loss rate and the second occlusion rate.
[0167] The total loss rate calculation unit is configured to calculate the loss rate by summing the first product and the second product.
[0168] Optionally, the weight prediction model is trained using the following modules:
[0169] The sample set acquisition module is configured to acquire a sample image set, wherein the sample images in the sample image set are images of the elevator car with passengers present, and each sample image is labeled with a third weight;
[0170] The sample input module is configured to input the sample image into the weight prediction model to obtain the passenger's fourth weight;
[0171] The weight loss calculation module is configured to calculate the loss rate using the third weight and the fourth weight.
[0172] The weight prediction model training condition judgment module is set to determine whether the preset training conditions are met; if yes, the weight prediction model training completion determination module is executed; if no, the weight prediction model adjustment module is executed.
[0173] The weight prediction model training completion determination module is configured to determine whether the weight prediction model has completed training.
[0174] The weight prediction model adjustment module is configured to adjust the model parameters of the weight prediction model according to the loss rate and return to the sample input module.
[0175] Optionally, the second fraction correction module 507 includes:
[0176] The correction coefficient determination unit is configured to calculate the correction coefficient based on the first weight and the second weight;
[0177] The second score calculation unit is configured to calculate the product of the correction coefficient and the first score to obtain the second score.
[0178] Optionally, the correction coefficient determination unit includes:
[0179] The weight difference calculation subunit is configured to calculate the difference between the first weight and the second weight;
[0180] The difference judgment subunit is configured to determine whether the difference is greater than the value 0. If not, the second determination module 509 is executed; if yes, the ratio calculation subunit is executed.
[0181] The ratio calculation subunit is configured to calculate the ratio of the difference to the preset weight when the difference is greater than a preset weight threshold.
[0182] The correction coefficient determination subunit is configured to determine the correction coefficient in a pre-configured ratio-coefficient table using the ratio.
[0183] The apparatus for identifying abnormal objects in elevators based on visual neural networks provided in this application can execute the method for identifying abnormal objects in elevators based on visual neural networks provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.
[0184] Figure 6 illustrates a schematic diagram of an electronic device 40 that can be used to implement embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.
[0185] As shown in Figure 6, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 can also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.
[0186] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0187] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as the method of identifying abnormal objects in an elevator based on a visual neural network.
[0188] In some embodiments, the method for identifying abnormal objects in an elevator based on a visual neural network can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the method for identifying abnormal objects in an elevator based on a visual neural network described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the method for identifying abnormal objects in an elevator based on a visual neural network by any other suitable means (e.g., by means of firmware).
[0189] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0190] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0191] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0192] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0193] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0194] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) in terms of management difficulty and weak business scalability.
Claims
1. A method for identifying abnormal objects inside an elevator based on a visual neural network, used to identify electric vehicles inside the elevator, including: When the elevator car door opens, the system controls the first camera inside the elevator car to capture the first video and controls the second camera to capture the second video. The first camera and the second camera are cameras located at different positions inside the elevator car. The first video and the second video are input into the electric vehicle recognition neural network to obtain the first score of whether an electric vehicle exists in the elevator car. Determine whether the first score is greater than a preset first score threshold; If so, confirm that an electric vehicle is present in the elevator car and generate an alarm message; If not, when the first score is greater than the preset second score threshold, the weighing sensor is activated to obtain the first weight of all objects in the elevator car, and the second score threshold is less than the first score threshold. The second weight of the passenger inside the elevator car is obtained by inputting at least one of the first video and the second video into the weight prediction model. The first score is corrected using the first weight and the second weight to obtain the second score; Determine whether the second score is greater than the first score threshold; If so, proceed with the step of determining that an electric vehicle exists inside the elevator car and generating an alarm message; If not, it is determined that there is no electric vehicle inside the elevator car.
2. The method according to claim 1, wherein, The electric vehicle recognition neural network includes a first feature extraction network, a second feature extraction network, a feature fusion network, a first detection head, and a second detection head. The first video and the second video are input into the electric vehicle recognition neural network to obtain a first score indicating the presence of an electric vehicle in the elevator car, including: The first video is input into the first feature extraction network, and the first feature map is extracted from the first video in the first feature extraction network. The second video is input into the second feature extraction network, and the second feature map is extracted from the second video in the second feature extraction network. The first feature map and the second feature map are fused in the feature fusion network to obtain fused features; The first detection head and the second detection head respectively identify the fused features to obtain a first initial score for the presence of electric vehicles in the first video and a second initial score for the presence of electric vehicles in the second video; The average of the first initial score and the second initial score is calculated as the first score of the electric vehicle present in the elevator car.
3. The method according to claim 2, wherein, The electric vehicle recognition neural network is trained in the following way: First and second images of the elevator car taken from different angles are used as training images. The first image is labeled with a third score indicating the presence of an electric vehicle and a first occlusion rate of the electric vehicle. The second image is labeled with a fourth score indicating the presence of an electric vehicle and a second occlusion rate of the electric vehicle. The first image and the second image are input into the electric vehicle recognition neural network to obtain the fifth score of the presence of an electric vehicle in the first image and the sixth score of the presence of an electric vehicle in the second image. The loss rate is calculated using the third score, fourth score, fifth score, sixth score, first occlusion rate, and second occlusion rate. Determine whether the preset training conditions are met; If so, confirm that the electric vehicle recognition neural network has completed training; If not, adjust the network parameters of the electric vehicle recognition neural network according to the loss rate, and return to the step of inputting the first image and the second image into the electric vehicle recognition neural network.
4. The method according to claim 3, wherein, The loss rate is calculated using the third score, fourth score, fifth score, sixth score, first occlusion rate, and second occlusion rate, including: The first loss rate is calculated using the third and fifth scores, and the second loss rate is calculated using the fourth and sixth scores; Calculate the first product of the first loss rate and the first occlusion rate, and calculate the second product of the second loss rate and the second occlusion rate; The loss rate is obtained by calculating the sum of the first product and the second product.
5. The method according to any one of claims 1-4, wherein, The weight prediction model is trained in the following way: Obtain a sample image set, wherein the sample images in the sample image set are images of the elevator car with passengers inside, and each sample image is labeled with a third weight; The sample image is input into the weight prediction model to obtain the passenger's fourth weight; The loss rate is calculated using the third weight and the fourth weight. Determine whether the preset training conditions are met; If so, confirm that the weight prediction model has completed training; If not, adjust the model parameters of the weight prediction model according to the loss rate, and return to the step of inputting the sample image into the weight prediction model to obtain the passenger's second weight.
6. The method according to any one of claims 1-4, wherein, The second score is obtained by correcting the first score using the first weight and the second weight, including: Calculate the correction coefficient based on the first weight and the second weight; The second score is obtained by multiplying the correction coefficient by the first score.
7. The method according to claim 6, wherein, The correction coefficient is calculated based on the first weight and the second weight, including: Calculate the difference between the first weight and the second weight; Determine whether the difference is greater than the value 0; If not, proceed to the step of determining that there is no electric vehicle in the elevator car; If so, when the difference is greater than a preset weight threshold, calculate the ratio of the difference to the preset weight; The correction coefficient is determined using the ratio in a pre-configured ratio-coefficient table.
8. A device for identifying abnormal objects inside an elevator based on a visual neural network, used to identify electric vehicles inside the elevator, comprising: The video acquisition module is configured to control a first camera inside the elevator car to acquire a first video and control a second camera to acquire a second video when the elevator car door is opened. The first camera and the second camera are cameras located at different positions inside the elevator car. The first score prediction module is configured to input the first video and the second video into the electric vehicle recognition neural network to obtain a first score indicating the presence of an electric vehicle in the elevator car. The first score judgment module is configured to determine whether the first score is greater than a preset first score threshold. If yes, the first determination module is executed; otherwise, the first weight acquisition module is executed. The first determining module is configured to determine that an electric vehicle exists inside the elevator car and generate an alarm message. The first weight acquisition module is configured to activate the weighing sensor to acquire the first weight of all objects in the elevator car when the first score is greater than a preset second score threshold, wherein the second score threshold is less than the first score threshold. The second weight prediction module is configured to obtain the second weight of the passenger in the elevator car from at least one of the first video and the second video input weight prediction models. The second fraction correction module is configured to correct the first fraction using the first weight and the second weight to obtain a second fraction. The second score determination module is configured to determine whether the second score is greater than the first score threshold. If yes, the first determination module is executed; otherwise, the second determination module is executed. The second determining module is configured to determine that there is no electric vehicle inside the elevator car.
9. An electronic device, the electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for identifying abnormal objects in an elevator based on a visual neural network, as described in any one of claims 1-7.
10. A computer-readable storage medium storing computer instructions for causing a processor to execute the method for identifying abnormal objects in an elevator based on a visual neural network, as described in any one of claims 1-7.