Boarding reminder processing method and apparatus, and medium

Abstract

A boarding reminder processing method and apparatus, and a medium. The method comprises: once a vehicle has been started, identifying whether a rear door of the vehicle undergoes an opening and closing process; and if it is identified that the rear door of the vehicle undergoes the opening and closing process, acquiring action data of a person outside the vehicle within a preset range from the vehicle, and when the action data is determined to be anomalous, sending a passenger left-behind reminder to the interior of the vehicle. The present disclosure prevents the vehicle from departing when a passenger outside the vehicle has not boarded, thereby avoiding the passenger left-behind problem.

Description

Boarding reminder handling methods, devices and media Cross-references to related applications

[0001] This application claims priority to Chinese Patent Application No. 202411807901.1, filed on December 10, 2024, entitled “Method, Apparatus and Medium for Handling Boarding Reminders”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of vehicle control technology, and in particular to a method, device and medium for processing boarding reminders. Background Technology

[0003] In daily travel, drivers often judge whether passengers have boarded the vehicle by the sound of the doors opening and closing. After hearing the door close, the driver assumes that the passengers have boarded and then drives the vehicle away.

[0004] However, in many real-world scenarios, even if the sound of the car door opening and closing is heard, it is possible that the passenger has not yet boarded the vehicle. For example, a passenger outside the vehicle may open the rear door and find that there are items on their side of the seat, making it impossible to sit down. They may then close the door and intend to sit down from the other side of the vehicle. In this driving situation, the driver hears the sound of the door opening and closing and assumes that the passenger has already sat down in the back seat, leading to the driver driving away directly without the passenger actually sitting down. Summary of the Invention

[0005] This disclosure provides a method, device, and medium for processing boarding reminders.

[0006] This disclosure provides a method for handling boarding reminders. The method includes: after the vehicle is started, identifying whether the rear door of the vehicle is in the process of opening and closing; if the rear door of the vehicle is identified to be in the process of opening and closing, acquiring the action data of people outside the vehicle within a preset range of the vehicle, and when the action data is determined to be abnormal, sending a passenger-left-behind reminder to the vehicle.

[0007] This disclosure also provides a vehicle entry reminder processing device, the device comprising: an identification module configured to identify whether the rear door of the vehicle is in an opening and closing process after the vehicle is started; an acquisition module configured to acquire action data of persons outside the vehicle within a preset range from the vehicle when the opening and closing process of the rear door of the vehicle is detected; and a reminder processing module configured to send a passenger left-behind reminder to the vehicle when the action data is determined to be abnormal.

[0008] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the boarding reminder processing method provided in this disclosure.

[0009] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, causes the processor to perform the boarding reminder processing method provided in this disclosure.

[0010] The above is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. The drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0012] Figure 1 is a flowchart illustrating a boarding reminder processing method provided in an embodiment of this disclosure.

[0013] Figure 2 is a schematic diagram of a set of joint locations provided in an embodiment of this disclosure.

[0014] Figure 3 is a schematic diagram of a boarding reminder processing device provided in an embodiment of this disclosure. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0017] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0019] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0020] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0021] To address the aforementioned issues, this disclosure provides a method for handling boarding reminders, which will be described below with reference to specific embodiments.

[0022] Figure 1 is a flowchart illustrating a boarding reminder processing method according to an embodiment of this disclosure. This method can be executed by a boarding reminder processing device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device in a vehicle. As shown in Figure 1, the method includes:

[0023] Step 101: After the vehicle is started, identify whether the rear doors of the vehicle are in the process of opening and closing.

[0024] In one embodiment of this disclosure, after the vehicle is started, it is identified whether the rear doors of the vehicle are in the process of opening and closing, that is, whether a user intends to get into the vehicle. Specifically, after the vehicle is started, the identification of the rear door opening and closing process is initiated.

[0025] It should be noted that the methods for identifying whether the rear doors of a vehicle are opening or closing vary depending on the application scenario. Specifically, the implementation can be a combination of one or more of the following examples:

[0026] In some possible embodiments, image information of the rear doors is acquired, for example, by communicating with a preset monitoring system to obtain image information of the rear doors captured by a monitoring camera, or by capturing image information of the rear doors through a camera installed in the vehicle and extracting image features. In this embodiment, the presence of an opening or closing process of the rear doors is identified based on the image features. The camera installed in the vehicle can be located on the B-pillar, the rearview mirror, or other similar locations.

[0027] For example, multiple consecutive image features corresponding to consecutive frames of images are input into a pre-trained artificial intelligence model. The AI ​​model determines whether a car door is open or closed based on these image features, because the image features for opening and closing a car door are different. In this embodiment, since the opening and closing process involves two actions, after first identifying that the car door is open based on the image features of the rear door image information, the model continues to acquire car door image information. If, within a preset time period, the model identifies that the rear door has been closed based on the image features of the acquired rear door image information, then it is determined that the rear door has undergone an opening and closing process.

[0028] In some possible embodiments, sound information from the rear doors is collected using a preset microphone. This microphone can be pre-positioned within the vehicle, such as on the B-pillar or inside the rearview mirror, to extract sound features. These features are then used to identify whether the rear doors are opening or closing. In this embodiment, the sound features can be input into a pre-trained artificial intelligence model. The model identifies the direction of sound propagation and intensity differences based on these features. Based on pre-learned information about the direction of sound propagation and intensity differences during door opening and closing, it determines whether the door is open or closed. Since the opening and closing process involves two separate actions, after initially identifying a door as open based on its sound features, the system continues to acquire sound information from the rear doors. If, within a preset time period, the sound features of the acquired rear door sound information indicate that the rear door has closed, then it is determined that a door opening and closing process has occurred.

[0029] In some possible embodiments, the angle between the rear door and the vehicle body is identified using a preset angle sensor. This preset angle sensor can be located at the door hinge of the rear door. In this embodiment, the angle between the rear door and the vehicle body is identified using the preset angle sensor, and the presence of an opening / closing process for the rear door is determined based on this angle. Specifically, if the angle is less than a preset angle threshold, the rear door is determined to be closed; if the angle is greater than the preset angle threshold, the rear door is determined to be open.

[0030] Step 102: If the opening and closing process of the rear door of the vehicle is detected, the motion data of the people outside the vehicle within a preset range of the vehicle is obtained, and when the motion data is judged to be abnormal, a passenger left-behind reminder is sent to the inside of the vehicle.

[0031] After the rear doors open and close (i.e., after the rear doors open and then close), motion data of people outside the vehicle within a preset range are acquired. This motion data includes at least one of the following: limb movement data, vocal movement data, and lip movement data. The preset range can be set according to the scenario requirements. The purpose of setting a preset range is to avoid excessive computational power consumption when recognizing the motion data of all people outside the vehicle.

[0032] After acquiring the motion data, it is determined whether the motion data of the person outside the vehicle is abnormal. In this embodiment, it is determined whether the person outside the vehicle is a passenger who wanted to board the vehicle but failed to do so. If the motion data of the person outside the vehicle is abnormal, the person outside the vehicle is considered to be a passenger who was left behind.

[0033] Furthermore, when the movement data of people outside the vehicle is determined to be abnormal, a passenger-left-behind alert is sent to the inside of the vehicle.

[0034] In one embodiment of this disclosure, when the motion data of people outside the vehicle is determined to be abnormal, a passenger abandonment reminder is sent to the vehicle to remind the driver that there are still passengers who have not boarded the vehicle, so as to prevent the driver from driving the vehicle away.

[0035] In some possible embodiments, when the motion data of a person outside the vehicle is determined to be abnormal, emergency braking can be applied to the vehicle to ensure the safety of the person outside. Before applying emergency braking, a braking reminder message can be sent to the driver. If the driver does not refuse the braking reminder message within a preset time period, emergency braking is applied. In this embodiment, to further ensure that emergency braking does not cause safety issues, when the motion data of a person outside the vehicle is determined to be abnormal, it is also necessary to determine that the current distance between the person outside the vehicle and the vehicle is less than a preset distance threshold before applying emergency braking. In other possible embodiments, when the motion data of a person outside the vehicle is determined to be abnormal, but the current distance between the person outside the vehicle and the vehicle is greater than the preset distance threshold, to avoid safety hazards, emergency braking is not applied; only a message prompting the driver to allow the person outside to board the vehicle is sent.

[0036] It should be noted that the methods for handling boarding reminders in vehicles differ in different application scenarios.

[0037] In one embodiment of this disclosure, a message corresponding to a passenger leaving a behind can be generated, and then the passenger leaving a behind can be announced via voice in the vehicle.

[0038] Based on the above embodiments, if the door is detected to be open after the vehicle is started, the vehicle will be subjected to emergency braking, such as by pulling up the vehicle's handbrake.

[0039] The following describes, with reference to specific embodiments, whether the motion data of the person outside the vehicle is abnormal in step 102 above. The different embodiments below can be implemented separately or in combination.

[0040] In one embodiment of this disclosure, video stream information of people outside the vehicle within a preset range of the vehicle is acquired. The video stream information includes multiple consecutive video frames. For example, the video stream information of people outside the vehicle can be acquired through an external camera, and then motion data can be acquired based on the multiple consecutive video frames.

[0041] For example, it can identify the set of joint points of people outside the vehicle in each video frame. Referring to Figure 2, the set of joint points of people outside the vehicle includes at least knee joints, ankle joints, elbow joints, shoulder key nodes, and hand joints.

[0042] In this embodiment, the set of multiple consecutive key point positions corresponding to multiple consecutive video frames is obtained as motion data. Therefore, the motion type of the person outside the vehicle can be determined based on the set of multiple consecutive key point positions. For example, the set of multiple consecutive key point positions can be input into a pre-trained deep learning model, and the motion type of the person outside the vehicle can be identified based on the deep learning model. In this embodiment, it is determined whether the motion type belongs to a preset motion type. When the motion type belongs to the preset motion type, the motion data of the person outside the vehicle is judged as abnormal. The preset motion type may include: reaching out to open the car door, chasing the vehicle, etc.

[0043] In this embodiment, if the preset action type is identified as stationary, then multiple consecutive video frames are acquired for a preset duration. Based on the acquired multiple consecutive video frames, it is determined whether the person outside the vehicle continues to remain stationary. If the person continues to remain stationary, the action data of the person outside the vehicle is determined to be abnormal.

[0044] In one embodiment of this disclosure, when the motion data includes sound data, sound data can be collected through a preset microphone. Based on the sound data, it can be identified whether preset keywords such as "I haven't gotten in the car yet" or "Don't drive" are included. If preset keywords are included, the motion data of the person outside the car is determined to be abnormal.

[0045] In this embodiment, multiple consecutive video frames can be collected to obtain lip-sync data, and the lip-sync data and audio data can be combined to determine whether the preset keywords are included.

[0046] In summary, the boarding reminder processing method of this disclosure, after the vehicle starts, identifies whether the rear doors of the vehicle are opening or closing. Then, upon detecting this opening or closing process, it acquires the motion data of persons outside the vehicle within a preset range. Based on the motion data, it determines whether the persons outside the vehicle are acting abnormally. If the motion data of the persons outside the vehicle is determined to be abnormal, a passenger abandonment reminder is sent to the vehicle. This technical solution prevents the vehicle from leaving before passengers outside the vehicle board, thus avoiding the problem of passengers being left behind.

[0047] To implement the above embodiments, this disclosure also proposes a vehicle boarding reminder processing device.

[0048] Figure 3 is a schematic diagram of a vehicle boarding reminder processing device provided in an embodiment of this disclosure. This device can be implemented by software and / or hardware, and is generally integrated into a vehicle processor. As shown in Figure 3, the device includes: an identification module 310, an acquisition module 320, and a reminder processing module 330.

[0049] The identification module 310 is configured to identify whether the rear doors of the vehicle are opening or closing after the vehicle is started.

[0050] The acquisition module 320 is configured to acquire motion data of people outside the vehicle within a preset range when the opening and closing process of the rear door of the vehicle is detected.

[0051] The reminder processing module 330 is configured to send a passenger-left-behind reminder to the vehicle when the motion data is determined to be abnormal.

[0052] In one embodiment of this disclosure, the identification module 310 is configured to: acquire image information of the rear door, extract image features of the image information, and identify whether the rear door is in an opening and closing process based on the image features; and / or, acquire sound information of the rear door through a preset microphone, extract sound features of the sound information, and identify whether the rear door is in an opening and closing process based on the sound features; and / or, identify the angle between the rear door and the vehicle body based on a preset angle sensor, and identify whether the rear door is in an opening and closing process based on the angle.

[0053] In one embodiment of this disclosure, the acquisition module 320 is configured to: acquire video stream information of people outside the vehicle within a preset range of the vehicle, wherein the video stream information includes multiple consecutive video frames; and acquire motion data based on the multiple consecutive video frames.

[0054] In one embodiment of this disclosure, the acquisition module 320 is used to: identify the set of key point positions of people outside the vehicle in each video frame, and acquire the set of multiple key point positions corresponding to multiple consecutive video frames as motion data.

[0055] In one embodiment of this disclosure, the device further includes: a determination module, which, when the motion data includes a set of multiple consecutive joint point positions, is configured to: determine the motion type of the person outside the vehicle based on the set of multiple consecutive joint point positions, and determine whether the motion type belongs to a preset motion type, wherein, when the motion type belongs to the preset motion type, the motion data is determined to be abnormal.

[0056] In one embodiment of this disclosure, the reminder processing module 330 is used to: broadcast a passenger's forgotten item reminder via voice in the vehicle.

[0057] The boarding reminder processing device provided in this disclosure can execute the boarding reminder processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0058] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, causes the processor to perform the boarding reminder processing method in the above embodiments.

[0059] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the above-described boarding reminder processing method.

[0060] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program that, when executed by a processor, causes the processor to perform the above-described boarding reminder processing method.

[0061] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from memory. When the computer program is executed by a processor, it performs the functions defined in the boarding reminder processing method of embodiments of this disclosure.

[0062] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk-ROM (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0063] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0064] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0065] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), Application-Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0066] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0067] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0068] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0069] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for handling boarding reminders, comprising: After the vehicle is started, it is determined whether the rear doors of the vehicle are in the process of opening and closing. If the opening and closing process of the rear door of the vehicle is detected, then the action data of the people outside the vehicle within a preset range of the vehicle are acquired, and When the motion data is determined to be abnormal, a passenger-left-behind alert is sent to the vehicle.

2. The method as described in claim 1, wherein, The identification of whether the rear door of the vehicle is in the process of opening and closing includes at least one of the following: Based on the image features of the image information of the rear door, identify whether the opening and closing process of the rear door exists; Based on the sound characteristics of the sound information of the rear door collected through a preset microphone, it is determined whether the opening and closing process of the rear door exists; or, Based on the angle between the rear door and the vehicle body collected by the preset angle sensor, it is determined whether the rear door is in the opening and closing process.

3. The method as described in claim 1 or 2, wherein, The acquisition of motion data of people outside the vehicle within a preset range from the vehicle includes: Acquire video stream information of the person outside the vehicle within a preset range of the vehicle, wherein the video stream information includes multiple consecutive video frames; The motion data is obtained from the consecutive multiple video frames.

4. The method of claim 3, wherein, The step of obtaining the motion data based on the consecutive multiple video frames includes: Identify the set of keypoint locations of the people outside the vehicle in each of the video frames. The motion data is obtained by acquiring the set of multiple consecutive key point positions corresponding to the consecutive multiple video frames.

5. The method of claim 4, wherein, The following methods are used to determine whether the action data is abnormal: The action type of the person outside the vehicle is determined based on the set of consecutive key point positions, and If the action type is determined to be a preset action type, the action data is deemed abnormal.

6. The method as described in any one of claims 1-5, wherein, Sending a passenger-left-behind alert to the vehicle includes: The system will broadcast a reminder via voice prompts inside the vehicle that a passenger may have left something behind.

7. A boarding reminder processing device, comprising: The identification module is configured to identify whether the rear doors of the vehicle are opening or closing after the vehicle is started. The acquisition module is configured to acquire motion data of people outside the vehicle within a preset range from the vehicle when the opening and closing process of the rear door of the vehicle is detected. as well as The reminder processing module is configured to send a passenger-left-behind reminder to the vehicle when the action data is determined to be abnormal.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the boarding reminder processing method according to any one of claims 1-6.

9. A computer program product comprising a computer program, wherein when executed by a processor, the computer program causes the processor to perform the boarding reminder processing method according to any one of claims 1-6.

Images