Method and apparatus for controlling autonomous vehicle, device, and storage medium

Abstract

Embodiments of the present invention provide a method and apparatus for controlling an autonomous vehicle, a device, and a storage medium. The method comprises: in response to an autonomous vehicle being in a trapped state, sending an assistance request associated with the trapped state to a remote device, the assistance request indicating an obstacle identified by a perception system of the autonomous vehicle; receiving a target message from the remote device; in response to the target message indicating that the obstacle is associated with false perception, updating perception information of the autonomous vehicle to remove the obstacle; and on the basis of the updated perception information, planning an escape route of the autonomous vehicle.

Description

Methods, apparatus, devices, and storage media for controlling autonomous vehicles

[0001] This application claims priority to Chinese Patent Application No. 202411746621.4, filed on November 29, 2024, entitled "Method, Apparatus, Device and Storage Medium for Controlling Autonomous Vehicles", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for controlling autonomous vehicles. Background Technology

[0003] Autonomous driving is a technology that uses computers to replace or assist human drivers in perceiving the vehicle's surroundings, planning the vehicle's trajectory, and controlling the vehicle to reach a designated destination.

[0004] During operation, autonomous vehicles may become stuck due to unforeseen circumstances and may be unable to extricate themselves from this situation independently. Therefore, efficiently assisting autonomous vehicles in overcoming these challenges is crucial for improving driving safety and reliability. Summary of the Invention

[0005] In a first aspect of this disclosure, a method for controlling an autonomous vehicle is provided. The method includes: in response to the autonomous vehicle being in a trapped state, sending an assistance request associated with the trapped state to a remote device, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system; receiving a target message from the remote device; in response to the target message indicating that the obstacle is associated with a false perception, updating the autonomous vehicle's perception information to remove the obstacle; and planning an escape route for the autonomous vehicle based on the updated perception information.

[0006] In a second aspect of this disclosure, a method for remote assistance is provided. The method includes: receiving an assistance request from an autonomous vehicle, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system; receiving, via an assistance interface, an interactive operation regarding the obstacle's association with a false perception; and, based on the interactive operation, sending a target message to the autonomous vehicle indicating that the obstacle is associated with the false perception, causing the autonomous vehicle to update its perception information to remove the obstacle.

[0007] In a third aspect of this disclosure, an apparatus for controlling an autonomous vehicle is provided. The apparatus includes: a first transmitting module configured to, in response to the autonomous vehicle being in a trapped state, transmit an assistance request associated with the trapped state to a remote device, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system; a first receiving module configured to receive a target message from the remote device; an updating module configured to, in response to the target message indicating that the obstacle is associated with a false perception, update the autonomous vehicle's perception information to remove the obstacle; and a planning module configured to, based on the updated perception information, plan an escape route for the autonomous vehicle.

[0008] In a fourth aspect of this disclosure, an apparatus for remote assistance is provided. The apparatus includes: a second receiving module configured to receive an assistance request from an autonomous vehicle, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system; an interaction module configured to receive, via an assistance interface, an interactive operation regarding the obstacle's association with a false perception; and a second sending module configured to, based on the interactive operation, send a target message to the autonomous vehicle indicating that the obstacle is associated with a false perception, causing the autonomous vehicle to update its perception information to remove the obstacle.

[0009] In a fifth aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the methods of the first or second aspect.

[0010] In a sixth aspect of this disclosure, a computer-readable storage medium is provided. A computer program is stored on the medium, which, when executed by a processor, implements the method of the first or second aspect.

[0011] In a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, it implements the method of the first or second aspect.

[0012] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0014] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0015] Figure 2 illustrates a flowchart of the signaling flow for remote assistance according to some embodiments of the present disclosure;

[0016] Figures 3A to 3F show schematic diagrams of examples of assistance interfaces according to some embodiments of the present disclosure;

[0017] Figure 4 shows a flowchart of a process for controlling an autonomous vehicle according to some embodiments of the present disclosure;

[0018] Figure 5 illustrates a flowchart of a remote assistance process according to some embodiments of the present disclosure;

[0019] Figure 6 illustrates an exemplary architecture block diagram of a device for controlling an autonomous vehicle according to some embodiments of the present disclosure;

[0020] Figure 7 shows a schematic structural block diagram of a device for remote assistance according to some embodiments of the present disclosure; and

[0021] Figure 8 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. Detailed Implementation

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0024] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0026] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0027] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably.

[0028] As briefly mentioned earlier, one mode of failure for autonomous vehicles is called "stuck," which means that the autonomous vehicle is unable to continue driving on its current path and stops, potentially preventing it from completing its subsequent journey and causing adverse traffic impacts. In some cases, due to errors in the autonomous vehicle's environmental perception system, false perception (FP) may occur, where the identified obstacle may not exist, leading to the autonomous vehicle being stuck.

[0029] In view of this, embodiments of the present disclosure provide an improved scheme for controlling an autonomous vehicle. In this scheme, if the autonomous vehicle is in a trapped state, it sends an assistance request associated with the trapped state to a remote device. The assistance request indicates an obstacle identified by the autonomous vehicle's perception system. A target message is received from the remote device. If the target message indicates that the obstacle is associated with a false perception, the autonomous vehicle's perception information is updated to remove the obstacle. Subsequently, based on the updated perception information, an escape route for the autonomous vehicle is planned.

[0030] Therefore, when an autonomous vehicle is trapped, embodiments of this disclosure assist in determining whether a perceived obstacle is associated with a false perception through remote assistance. If the obstacle is determined to be associated with a false perception, the obstacle is removed by updating the perception information, enabling the autonomous vehicle to quickly escape the trapped state. Embodiments of this disclosure can improve the safety and reliability of autonomous vehicles.

[0031] Example Environment

[0032] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown, environment 100 may include an autonomous vehicle 101. The autonomous vehicle 110 may be a vehicle with autonomous driving capability (or driverless capability), also known as an unmanned vehicle, autonomous driving vehicle, etc.

[0033] In some embodiments, the autonomous vehicle 101 can be any type of vehicle capable of carrying people and / or goods and moving via a power system such as an engine, including but not limited to cars, trucks, buses, electric vehicles, motorhomes, etc. The autonomous vehicle 101 can be an automated driving vehicle (also known as an autonomous vehicle) that integrates functions such as environmental perception, planning and decision-making, and multi-level assisted driving.

[0034] In environment 100, the autonomous vehicle 101 is equipped with electronic equipment 150, which can communicate with remote equipment 120. For example, electronic equipment 150 can communicate with remote equipment 120 via appropriate wireless communication methods. Electronic equipment 150 can be any device with computing capabilities, which can generate a route to get out of trouble or execute corresponding commands based on the path planning information issued by remote equipment 120.

[0035] In some embodiments, the remote device 120 may include, for example, a cloud device or an edge computing device. Such a remote device 120 may be configured to provide the autonomous vehicle 101 with supplemental perception information about the traffic environment and / or provide guidance for the stranded autonomous vehicle 101 to extricate itself from trouble.

[0036] In environment 100, assistant 110 can interact directly with remote device 120, or via an attached device connected to remote device 120. Remote device 120 can present user interface 140 to assistant 110, allowing assistant 110 to view vehicle information or perform route planning, etc.

[0037] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0038] Example Interaction

[0039] Some exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0040] Figure 2 illustrates a flowchart of a signaling flow 200 for remote assistance according to some embodiments of the present disclosure. The signaling flow 200 relates to an electronic device 150 and a remote device 120. For ease of discussion, some embodiments of the present disclosure will be described below in conjunction with the environment 100 in Figure 1, but these are merely exemplary.

[0041] In embodiments of this disclosure, if it is determined (205) that the autonomous vehicle 101 is in a trapped state, the electronic device 150 sends (210) an assistance request associated with the trapped state to the remote device 120. In some embodiments, when it is determined that the autonomous vehicle 101 is in a trapped state, the electronic device 150 can receive perception information about obstacles from the perception system of the autonomous vehicle 101. Based on the perception information, the type of obstacle causing the autonomous vehicle 101 to be trapped is determined. If it is determined that the obstacle causing the trapped state is a predetermined type, the electronic device 150 can send an assistance request associated with the trapped state to the remote device 120. The predetermined type may include a type associated with error perception, i.e., a type that is prone to error perception. In some examples, the predetermined type may include, but is not limited to, static small objects, unknown obstacles, roadblocks, construction zones (CZ), etc. By determining whether it belongs to a predetermined type that is prone to error perception, not only can the triggering of remote assistance be reduced, but the remote device 120 can also be prompted to assist in determining whether it is an error perception.

[0042] In some embodiments, an assistance request may include at least one of the following: an obstacle identifier, obstacle location information, obstacle type information, and image information associated with the obstacle. The obstacle identifier is configured to uniquely identify the obstacle recognized by the perception system, and may include, but is not limited to, a name, code, serial number, etc. The location information may include, but is not limited to, the relative distance and relative orientation (e.g., heading angle, pitch angle, roll angle, etc.) between the obstacle and the autonomous vehicle. The type information indicates the type of obstacle, such as a small static object, an unknown obstacle, a roadblock, a construction zone, a vehicle, a person, an animal, etc. The image information may include static or dynamic images (i.e., videos) associated with the obstacle. In some examples, the assistance request may also include obstacle size information, such as the obstacle's three-dimensional dimensions.

[0043] Referring again to Figure 2, if an assistance request is received from electronic device 150, remote device 120 receives (215) an interactive operation regarding the association of an obstacle with a false alarm via an assistance interface. In some embodiments, upon receiving an assistance request from electronic device 150, remote device 120 may present a target element associated with the obstacle to assistant 110 in the assistance interface to facilitate assistant 110 in verifying whether the obstacle is a false alarm. The target element may be configured to indicate the obstacle, for example, by indicating the obstacle's location, type, size, etc.

[0044] The target elements and interactive operations are illustrated below with reference to Figures 3A to 3F, which respectively show examples 300A to 300F of assistance interfaces according to some embodiments of the present disclosure. The assistance interfaces shown in Examples 300A to 300F can be provided by a remote device 120 to support an assistant 110 in assisting the vehicle 101 to get out of trouble via these assistance interfaces.

[0045] In some embodiments, the remote device 120 presents at least one video stream associated with the autonomous vehicle 101 in a first area of ​​the assistance interface. Based on the location information of the obstacle, a first element associated with the obstacle is presented at the corresponding position of the at least one video stream. The first element can be used to indicate the obstacle in the video stream to facilitate the assistant 110 in verifying whether the obstacle is a false perception through the video stream. As an example, as shown in FIG3A, the assistance interface in Example 300A includes areas 311, 312, 313, and 314. Area 311 presents a video stream in front of the autonomous vehicle 101, area 312 presents a video stream to the left of the autonomous vehicle 101, area 313 presents a video stream to the right of the autonomous vehicle 101, and area 314 presents a video stream behind the autonomous vehicle 101. Since the obstacle is located in front of the autonomous vehicle, the display size of area 311 can be increased relative to areas 312, 313, and 314 to facilitate the assistant 110 in verifying the obstacle. For a single obstacle as shown in Example 300A, a highlight layer 301 (i.e., the first element) and the obstacle number 302 (ID) can be added to the image area corresponding to the obstacle in the video stream.

[0046] As another example, as shown in Figure 3B, in Example 300B, when multiple obstacles are detected, bounding boxes 303 (i.e., first elements) or highlight layers can be added to the image regions corresponding to the multiple obstacles in the video stream. Of course, the first element is not limited to highlight layers and bounding boxes; it can also include shadows, solid color layers, etc., added to the image regions corresponding to the obstacles. The embodiments of this disclosure do not limit this.

[0047] In some embodiments, the remote device 120 may present at least one equidistant line in at least one video stream. As an example, as shown in FIG3B, an equidistant line 303 is presented in the lane corresponding to the autonomous vehicle 101 in region 311. The equidistant line 304 includes diagonal lines 306 and 307 extending along the driving direction of the lane, which can identify the width of the lane where the autonomous vehicle 101 is located. The equidistant line 304 also includes a horizontal line 308 extending perpendicular to the driving direction of the lane. The horizontal line 308 may represent a predetermined distance (e.g., 1m, 2m, 5m, etc.) between the location of the horizontal line 308 and the autonomous vehicle 101.

[0048] In some embodiments, the remote device 120 may also display a map associated with the autonomous vehicle 101 in a second area of ​​the assistance interface. Based on the location information of obstacles, a second element associated with the obstacle is displayed at the corresponding location on the map. The second element is used to indicate the obstacle on the map, so as to facilitate the assistant 110 to observe the obstacle through the map, thereby improving the assistance efficiency of the assistant 110.

[0049] As an example, as shown in Figure 3A, the assistance interface shown in Example 300A may also include area 320. The remote device 120 may display a map of the vicinity of the autonomous vehicle 101 in area 320. The map may display an icon 321 indicating the autonomous vehicle 101, a highlight layer 322 indicating obstacles, and obstacle numbers 323.

[0050] As another example, as shown in Figure 3B, when multiple obstacles are detected, the remote device can display a bounding box 324 in the map corresponding to the areas of the multiple obstacles. The bounding box 324 indicates the areas where the multiple obstacles are located, so as to facilitate the assistance personnel 110 to verify whether the multiple obstacles are false perceptions.

[0051] In some embodiments, the remote device 120 may present a three-dimensional spatial view related to the autonomous vehicle 101 in a third area of ​​the assistance interface. Based on vehicle information of the autonomous vehicle 101 and obstacle location information, a third element associated with the autonomous vehicle 101 and a fourth element associated with the obstacle are presented at corresponding positions in the three-dimensional spatial view. The three-dimensional spatial view can be configured to indicate the relative positional relationship between the autonomous vehicle 101 and the obstacle in the environmental space.

[0052] As an example, as shown in Figure 3C, the assistance interface illustrated in Example 300C includes region 330. Region 330 presents a three-dimensional spatial view modeled from the environment in which the autonomous vehicle 101 is located. The three-dimensional spatial view displays an icon 331 indicating the autonomous vehicle 101 and icons 332 indicating obstacles. Example 300C actually shows a side view of the autonomous vehicle 101, which allows the assistant 110 to see the relative positional relationship between the autonomous vehicle 101 and obstacles in the length and height directions of the autonomous vehicle 101. Region 330 may also display a window 333, through which perceived information about obstacles can be presented, such as three-dimensional size information, type information, etc.

[0053] As another example, the three-dimensional spatial view can be configured to switch the view orientation. The assistant 110 can switch the three-dimensional spatial view shown in area 330 from the view effect shown in example 300C to the view effect shown in example 300D by triggering menus, options, or controls on the assistance interface. Example 300D is actually a front view of the autonomous vehicle 101, which presents to the assistant 110 the relative positional relationship between the autonomous vehicle 101 and obstacles in the width and height directions of the autonomous vehicle 101. It should be noted that the above elements are merely exemplary. In practical applications, any appropriate elements can be selected to indicate obstacles and the autonomous vehicle 101; the embodiments of this disclosure are not limited in this regard.

[0054] Generally, after the assistant 110 verifies whether an obstacle is a false alarm through the content presented on the assistance interface, the assistant 110 can perform appropriate interactive operations through the assistance interface to indicate that the obstacle is associated with a false alarm. Of course, the assistant 110 can also indicate that the obstacle is not a false alarm through appropriate operations. The following provides an exemplary description of the interactive operations for associating an obstacle with a false alarm, but it should not be construed as being limited to using the interactive operations shown below to associate an obstacle with a false alarm. In practical applications, any appropriate operation can be selected to associate an obstacle with a false alarm, and the embodiments of this disclosure are not limited thereto.

[0055] In some embodiments, if a trigger is received from a target element, the remote device 120 may present a removal control for removing the obstacle. The remote device 120 can indicate that the obstacle is associated with error detection by receiving an interactive action on the removal control. The removal control provides the assistant 110 with an entry point to associate the obstacle with error detection. The assistant 110 can trigger or select the removal control by an interactive action that matches the removal control. The remote device 120 can determine that the obstacle is associated with error detection in response to receiving a trigger or selection on the removal control. The interactive action can include any type of operation that matches the removal control, such as a single click, double click, long press, drag, etc. Thus, a specific obstacle can be accurately associated with error detection by using the removal control.

[0056] In some examples, remote device 120 may present a removal control to indicate the removal of an obstacle in response to receiving a trigger on a first element presented in the video stream. As an example, as shown in Figure 3E, remote device 120 may present a removal control 341 if it receives a trigger on a highlighted layer 301 or number 302 presented in region 311. Remote device 120 may also present a removal control 341 in response to receiving a trigger on region 320. Removal control 341 includes options 342 and 343. If remote device 120 receives a trigger on option 342, remote device 120 may determine that the obstacle is associated with error perception.

[0057] In other examples, remote device 120 may, in response to receiving a trigger on a second element presented in the map, present a removal control to indicate the removal of an obstacle. As an example, as shown in Figure 3E, if a trigger is received on a highlighted layer 322 or number 323 presented in region 320, remote device 120 may present a removal control 341.

[0058] In other examples, remote device 120 may, in response to receiving a trigger on a fourth element in the three-dimensional spatial view, present a removal control to indicate the removal of an obstacle. As an example, as shown in FIG3E, if a trigger is received on icon 332 presented in region 330, remote device 120 may present removal control 341. It is understood that the above removal controls are merely exemplary, and any other appropriate removal control may be selected as needed. The embodiments of this disclosure do not specifically limit this.

[0059] In some embodiments, the remote device 120 can receive interactive operations via an assistance interface. The interactive operations can indicate a set of waypoints corresponding to a reference driving route, which passes through areas corresponding to obstacles. Specifically, the assistant 110 can trigger the remote device 120 to generate a set of waypoints through the interactive operations. This set of waypoints can indicate a reference driving route for the autonomous vehicle 101. The reference driving route can be a route formed by connecting the waypoints. Alternatively, the reference driving route can be a driving route generated based on the connection of the waypoints and the driving rules of the autonomous vehicle 101. If it is determined that the reference driving route passes through areas corresponding to obstacles, at least one obstacle associated with the reference driving route can be associated with the error perception. Thus, when there are many obstacles identified as being mistakenly perceived, the assistant 110 can conveniently associate at least one obstacle with the error perception by selecting a set of waypoints, which helps improve the efficiency of remote assistance.

[0060] As an example, as shown in Figure 3F, highlighted areas 325 corresponding to multiple obstacles are displayed on the map in area 320. The assistant 110 can select a set of points on the map through interactive operations such as clicking. The remote device 120 can generate a set of waypoints 326-1, 326-2, 326-3, ..., 326-N corresponding to this set of points. The remote device 120 can determine the line connecting these waypoints 326-1, 326-2, 326-3, ..., 326-N as a reference driving route 327. Clearly, the reference driving route 327 passes through the highlighted area 325 on the map.

[0061] Continuing with Figure 2, the remote device 120, based on an interactive operation, sends (220) a target message to the electronic device 150 (i.e., the autonomous vehicle 101) instructing the obstacle to be associated with the error perception. Specifically, if the remote device receives an interactive operation regarding the association of an obstacle with the error perception, the remote device 120 can send a target message to the electronic device 150. This target message instructs the electronic device 150 to associate the obstacle with the error perception.

[0062] In embodiments of this disclosure, if a target message received from remote device 120 indicates that an obstacle is associated with a false perception, electronic device 150 updates (225) the perception information of autonomous vehicle 101 to remove the obstacle. Removing an obstacle here does not mean physically removing it from the road. Removing an obstacle can be understood as removing information related to the obstacle indicated by the target message from the autonomous vehicle 101's perception information of the environment, so that the autonomous vehicle 101 can ignore the removed obstacle or determine that the removed obstacle does not exist. For example, electronic device 150 can remove an obstacle from the autonomous vehicle 101's perception list of obstacles in the environment, causing the autonomous vehicle 101's planning and navigation control (PNC) system to ignore the removed obstacle.

[0063] In some embodiments, if the target message is determined to include a removal instruction to remove an obstacle, the electronic device 150 updates the perception information of the autonomous vehicle 101 to remove the obstacle indicated by the removal instruction. In some examples, the removal instruction may carry an identifier (e.g., a number) of the obstacle, and the electronic device 150 can remove the corresponding obstacle from the perception information of the autonomous vehicle 101 based on the obstacle identifier.

[0064] In some embodiments, if the target message indicates a set of waypoints for a reference driving route of the autonomous vehicle 101, the electronic device 150 identifies at least one obstacle associated with the reference driving route. The electronic device 150 then updates the perception information of the autonomous vehicle 101 to remove the at least one obstacle.

[0065] In some examples, if the target message includes a set of waypoints, the electronic device 150 can determine a corresponding reference driving route based on that set of waypoints. If it is determined that the reference driving route passes through an area corresponding to an obstacle, the electronic device 150 can identify at least one obstacle associated with the reference driving route.

[0066] As an example, electronic device 150 can determine the line connecting the set of waypoints as a reference driving route. If the reference driving route is determined to pass through the area corresponding to an obstacle, electronic device 150 can determine the travel area of ​​the autonomous vehicle 101 along the reference driving route, and then determine the overlapping area between the travel area and the area corresponding to the obstacle. The obstacle located in this overlapping area is then identified as associated with error perception.

[0067] In other examples, at least one obstacle associated with a reference driving route can also be identified by remote device 120, and the target message may include the set of waypoints and the identifier of the at least one obstacle. Electronic device 150 can update the perception information of autonomous vehicle 101 based on the identifier of the at least one obstacle. It is understood that the above-described methods of removing obstacles are merely exemplary, and the methods by which electronic device 150 removes obstacles may also differ depending on how remote device 120 indicates that an obstacle is associated with a false perception.

[0068] In embodiments of this disclosure, electronic device 150 plans an escape route for autonomous vehicle 101 based on updated perception information. In some examples, if the target message includes a move-out instruction, electronic device 150 may instruct the PNC of autonomous vehicle 101 to plan an escape route for autonomous vehicle 101 based on the updated perception information. It is then determined whether there is a risk of collision with traffic participants in the environment when autonomous vehicle 101 travels along the planned escape route. If it is determined that there is no risk of collision when autonomous vehicle 101 travels along the escape route, electronic device 150 may instruct autonomous vehicle 101 to travel along the escape route, thereby freeing autonomous vehicle 101 from the trapped state.

[0069] In other examples, if the target message includes a set of waypoints indicating a reference driving route, the electronic device 150 can instruct the PNC to plan an escape route for the autonomous vehicle 101 based on updated perception information and the reference driving route. Of course, the electronic device 150 can determine whether there is a collision risk along the escape route. If it is determined that there is no collision risk along the escape route, the electronic device 150 can instruct the autonomous vehicle 101 to drive along the escape route to extricate the autonomous vehicle 101 from the trapped state.

[0070] In some embodiments, if it is determined that the autonomous vehicle 101 is in a trapped state, the electronic device 150 can determine the cause of the trapped state. If it is determined that the trapped state is caused by the presence of an obstacle in the environment in which the autonomous vehicle 101 is located, the electronic device 150 can determine the target area corresponding to the obstacle. The electronic device 150 can receive perception information from the autonomous vehicle 101's perception system in real time. If, based on the perception information, at least one traffic participant (e.g., a vehicle) is detected entering the target area, it indicates that there may be no obstacle in the target area. The electronic device 150 can determine that the obstacle is associated with a false perception, update the perception information of the autonomous vehicle 101, and remove the obstacle. Subsequently, the electronic device 150 can plan an escape route for the autonomous vehicle based on the updated perception information, so that the autonomous vehicle 101 can extricate itself from the trapped state. This reduces the triggering of remote assistance and improves the extrication efficiency of the autonomous vehicle 101.

[0071] In summary, when an autonomous vehicle is trapped, embodiments of this disclosure assist in determining whether a perceived obstacle is associated with a false perception. If the obstacle is determined to be associated with a false perception, the obstacle is removed by updating the perception information, enabling the autonomous vehicle to quickly escape the trapped state. Embodiments of this disclosure improve the safety and reliability of autonomous vehicles.

[0072] Example processes, apparatus and equipment

[0073] Figure 4 shows a flowchart of a process 400 for controlling an autonomous vehicle according to some embodiments of the present disclosure. Process 400 can be implemented in an electronic device 150.

[0074] In box 410, electronic device 150, in response to the autonomous vehicle being in a distressed state, sends an assistance request associated with the distressed state to a remote device, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system.

[0075] In box 420, electronic device 150 receives target messages from remote device.

[0076] In box 430, electronic device 150, in response to a target message indicating that an obstacle is associated with a false perception, updates the autonomous vehicle’s perception information to remove the obstacle.

[0077] In frame 440, electronic device 150 plans an escape route for the autonomous vehicle based on updated perception information.

[0078] In some embodiments, sending an assistance request to a remote device includes: in response to an autonomous vehicle being in a stalemate and the obstacle causing the stalemate being of a predetermined type, sending an assistance request associated with the stalemate to the remote device.

[0079] In some embodiments, updating the perception information of the autonomous vehicle includes: in response to a target message including a removal instruction to remove an obstacle, updating the perception information of the autonomous vehicle to remove the obstacle indicated by the removal instruction.

[0080] In some embodiments, updating the perception information of the autonomous vehicle includes: in response to a target message indicating a set of waypoints for a reference driving route of the autonomous vehicle, identifying at least one obstacle associated with the reference driving route; and updating the perception information of the autonomous vehicle to remove the at least one obstacle.

[0081] In some embodiments, process 400 further includes: in response to detecting that at least one traffic participant has entered a target area corresponding to an obstacle, determining that the obstacle is associated with a false perception; updating the perception information of the autonomous vehicle to remove the obstacle; and planning an escape route for the autonomous vehicle based on the updated perception information.

[0082] In some embodiments, the assistance request includes at least one of the following: an obstacle identifier, obstacle location information, obstacle type information, and image information related to the obstacle.

[0083] Figure 5 illustrates a flowchart of a remote assistance process 500 according to some embodiments of the present disclosure. The process 500 can be implemented at a remote device 120.

[0084] In frame 510, remote device 120 receives an assistance request from the autonomous vehicle, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system.

[0085] In frame 520, remote device 120 receives interactive operations regarding obstacle association with error perception via an assistance interface.

[0086] In frame 530, remote device 120 sends an instruction message to the autonomous vehicle, based on interactive operation, indicating that the obstacle is associated with the target of the erroneous perception, so that the autonomous vehicle updates the perception information to remove the obstacle.

[0087] In some embodiments, receiving an interactive action regarding an obstacle's association with error perception includes: presenting a target element associated with the obstacle in an assistance interface; presenting a removal control for removing the obstacle in response to receiving a trigger of the target element; and receiving an interactive action for the removal control to indicate that the obstacle is associated with error perception.

[0088] In some embodiments, receiving an interactive operation regarding an obstacle's association with error perception includes receiving the interactive operation via an assistance interface, wherein the interactive operation indicates a set of waypoints corresponding to a reference driving route that passes through the area corresponding to the obstacle.

[0089] In some embodiments, process 500 further includes: presenting at least one video stream associated with the autonomous vehicle in a first area of ​​the assistance interface; and presenting a first element associated with an obstacle at a corresponding location in the at least one video stream based on the obstacle's location information.

[0090] In some embodiments, process 500 further includes: presenting at least one equidistant line in at least one video stream.

[0091] In some embodiments, process 500 further includes: presenting a map associated with the autonomous vehicle in a second area of ​​the assistance interface; and presenting a second element associated with the obstacle at a corresponding location on the map based on the obstacle's location information.

[0092] In some embodiments, process 500 further includes: presenting a three-dimensional spatial view related to the autonomous vehicle in a third area of ​​the assistance interface; and presenting a third element associated with the autonomous vehicle and a fourth element associated with the obstacle at corresponding positions in the three-dimensional spatial view based on vehicle information of the autonomous vehicle and location information of the obstacle.

[0093] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. Figure 6 shows a schematic structural block diagram of an apparatus 600 for controlling an autonomous vehicle according to some embodiments of this disclosure. The apparatus 600 may be implemented in or included in electronic device 150. The various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0094] As shown in Figure 6, the device 600 includes: a first sending module 610, a first receiving module 620, an updating module 630, and a planning module 640. The first sending module 610 is configured to send an assistance request associated with the stranded state to a remote device in response to the autonomous vehicle being in a stranded state. The assistance request indicates an obstacle identified by the autonomous vehicle's perception system. The first receiving module 620 is configured to receive a target message from the remote device. The updating module 630 is configured to update the autonomous vehicle's perception information in response to the target message indicating that the obstacle is associated with a false perception, in order to remove the obstacle. The planning module 640 is configured to plan an escape route for the autonomous vehicle based on the updated perception information.

[0095] In some embodiments, the first sending module 610 is further configured to: in response to an autonomous vehicle being in a stagnant state and the obstacle causing the stagnant state being of a predetermined type, send an assistance request associated with the stagnant state to a remote device.

[0096] In some embodiments, the update module 630 is further configured to update the perception information of the autonomous vehicle in response to a target message including a removal instruction to remove an obstacle, so as to remove the obstacle indicated by the removal instruction.

[0097] In some embodiments, the update module 630 is further configured to: in response to a target message indicating a set of waypoints for a reference driving route of the autonomous vehicle, identify at least one obstacle associated with the reference driving route; and update the perception information of the autonomous vehicle to remove at least one obstacle.

[0098] In some embodiments, the apparatus 600 further includes: a detection module configured to, in response to detecting that at least one traffic participant has entered a target area corresponding to an obstacle, determine that the obstacle is associated with a false perception; update the perception information of the autonomous vehicle to remove the obstacle; and plan an escape route for the autonomous vehicle based on the updated perception information.

[0099] In some embodiments, the assistance request includes at least one of the following: an obstacle identifier, obstacle location information, obstacle type information, and image information related to the obstacle.

[0100] Figure 7 shows a schematic structural block diagram of a device 700 for remote assistance according to some embodiments of the present disclosure. The device 700 may be implemented in or included in a remote device 120. Various modules / components in the device 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0101] As shown in Figure 7, the device 700 includes: a second receiving module 710, an interaction module 720, and a second sending module 730. The second receiving module 710 is configured to receive an assistance request from an autonomous vehicle, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system. The interaction module 720 is configured to receive an interactive operation via an assistance interface regarding the association of the obstacle with a false perception. The second sending module 730 is configured to, based on the interactive operation, send a target message to the autonomous vehicle indicating that the obstacle is associated with a false perception, causing the autonomous vehicle to update its perception information to remove the obstacle.

[0102] In some embodiments, the interaction module 720 is further configured to: present a target element associated with an obstacle in an assistance interface; present a removal control for removing the obstacle in response to receiving a trigger of the target element; and receive an interactive operation on the removal control to indicate that the obstacle is associated with error perception.

[0103] In some embodiments, the interaction module 720 is further configured to receive an interaction operation via an assistance interface, wherein the interaction operation indicates a set of waypoints corresponding to a reference driving route, the reference driving route passing through areas corresponding to obstacles.

[0104] In some embodiments, the device 700 further includes: a first presentation module configured to present at least one video stream associated with the autonomous vehicle in a first area of ​​the assistance interface; and to present a first element associated with an obstacle at a corresponding position in the at least one video stream based on the obstacle's location information.

[0105] In some embodiments, the first presentation module is further configured to present at least one equidistant line in at least one video stream.

[0106] In some embodiments, the device 700 further includes: a second presentation module configured to present a map associated with the autonomous vehicle in a second area of ​​the assistance interface; and to present a second element associated with an obstacle at a corresponding location on the map based on the obstacle's location information.

[0107] In some embodiments, the device 700 further includes: a third presentation module that presents a three-dimensional spatial view related to the autonomous vehicle in a third area of ​​the assistance interface; and, based on vehicle information of the autonomous vehicle and location information of obstacles, presents a third element associated with the autonomous vehicle and a fourth element associated with the obstacles at corresponding positions in the three-dimensional spatial view.

[0108] The units and / or modules included in devices 600 and 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in devices 600 and 700 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0109] Figure 8 shows a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 800 shown in Figure 8 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 800 shown in Figure 8 may include or be implemented as the electronic device 150 of Figure 1, the remote device 120, the device 600 of Figure 6, or the device 700 of Figure 7.

[0110] As shown in Figure 8, the electronic device 800 is in the form of a general-purpose electronic device. Components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage devices 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0111] Electronic device 800 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 830 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 800.

[0112] Electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG8, disk drives for reading or writing from removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading or writing from removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0113] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 800 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0114] Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown) via communication unit 840 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 800, or with any device that enables electronic device 800 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0115] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0116] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0117] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0118] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0120] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for controlling an autonomous vehicle, comprising: In response to an autonomous vehicle being in a distressed state, an assistance request associated with the distressed state is sent to a remote device, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system; Receive the target message from the remote device; In response to the target message indicating that the obstacle is associated with a false perception, the perception information of the autonomous vehicle is updated to remove the obstacle; as well as Based on the updated perception information, a route for the autonomous vehicle to escape from trouble is planned.

2. The method of claim 1, wherein sending the assistance request to the remote device comprises: In response to the autonomous vehicle being in a trapped state and the obstacle causing the trapped state being of a predetermined type, an assistance request associated with the trapped state is sent to the remote device.

3. The method according to claim 1, wherein updating the perception information of the autonomous vehicle includes: In response to the target message including a removal instruction to remove the obstacle, the perception information of the autonomous vehicle is updated to remove the obstacle indicated by the removal instruction.

4. The method according to claim 1, wherein updating the perception information of the autonomous vehicle comprises: In response to a set of waypoints indicating a reference driving route for the autonomous vehicle in the target message, at least one obstacle associated with the reference driving route is identified; as well as Update the perception information of the autonomous vehicle to remove the at least one obstacle.

5. The method according to claim 1, further comprising: In response to detecting that at least one traffic participant has entered a target area corresponding to the obstacle, the obstacle is determined to be associated with a false perception; Update the perception information of the autonomous vehicle to remove the obstacle; as well as Based on the updated perception information, a route for the autonomous vehicle to escape from trouble is planned.

6. The method of claim 1, wherein the assistance request comprises at least one of the following: The sign of the obstacle, The location information of the obstacle The type information of the obstacle. Image information related to the obstacle.

7. A method for remote assistance, comprising: Receive assistance requests from autonomous vehicles, the assistance requests indicating obstacles identified by the autonomous vehicles' perception systems; The system receives interactive actions regarding the association of the obstacle with error perception via the assistance interface. as well as Based on the interaction, a target message indicating that the obstacle is associated with the erroneous perception is sent to the autonomous vehicle, causing the autonomous vehicle to update its perception information to remove the obstacle.

8. The method of claim 7, wherein receiving an interactive action regarding the obstacle's association with error perception comprises: The target element associated with the obstacle is presented in the assistance interface; In response to receiving a trigger from the target element, a removal control for removing the obstacle is presented; as well as Receive the interaction action for the removal control to indicate that the obstacle is associated with error perception.

9. The method of claim 7, wherein receiving an interactive operation regarding the obstacle's association with error perception comprises: The interactive operation is received via the assistance interface, wherein the interactive operation indicates a set of waypoints corresponding to a reference driving route, the reference driving route passing through the area corresponding to the obstacle.

10. The method of claim 7, further comprising: At least one video stream associated with the autonomous vehicle is presented in a first area of ​​the assistance interface; as well as Based on the location information of the obstacle, a first element associated with the obstacle is presented at the corresponding position in the at least one video stream.

11. The method of claim 10, further comprising: At least one equidistant line is presented in the at least one video stream.

12. The method of claim 7, further comprising: A map associated with the autonomous vehicle is displayed in the second area of ​​the assistance interface; as well as Based on the location information of the obstacle, a second element associated with the obstacle is presented at the corresponding location on the map.

13. The method of claim 7, further comprising: A three-dimensional spatial view related to the autonomous vehicle is presented in the third area of ​​the assistance interface; as well as Based on the vehicle information of the autonomous vehicle and the location information of the obstacle, a third element associated with the autonomous vehicle and a fourth element associated with the obstacle are presented at the corresponding positions in the three-dimensional spatial view.

14. A device for controlling an autonomous vehicle, comprising: A first sending module is configured to send an assistance request associated with the stranded state to a remote device in response to the autonomous vehicle being in a stranded state, the assistance request indicating an obstacle identified by the autonomous vehicle's perception system. The first receiving module is configured to receive the target message from the remote device; An update module is configured to update the perception information of the autonomous vehicle in response to the target message indicating that the obstacle is associated with a false perception, so as to remove the obstacle; as well as The planning module is configured to plan the autonomous vehicle's escape route based on updated perception information.

15. An apparatus for remote assistance, comprising: The second receiving module is configured to receive assistance requests from an autonomous vehicle, the assistance requests indicating obstacles identified by the autonomous vehicle's perception system; The interaction module is configured to receive interactive operations related to error perception of the obstacle via an assistance interface; as well as The second sending module is configured to send a target message indicating that the obstacle is associated with the error perception to the autonomous vehicle based on the interaction operation, so that the autonomous vehicle updates the perception information to remove the obstacle.

16. An electronic device comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 6 or 7 to 13 when executed by the at least one processing unit.

17. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 6 or 7 to 13.

18. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 6 or 7 to 13.

Images