Image generation and display method and apparatus, and electronic device, product, medium and vehicle

Abstract

An image generation and display method and apparatus, and an electronic device, a product, a medium and a vehicle. The method comprises: acquiring real-time perception data that is obtained by means of a perception device on a vehicle perceiving a surrounding environment of an ego-vehicle; on the basis of the real-time perception data, generating an electronic image of the surrounding environment, wherein the electronic image includes an ego-vehicle identifier corresponding to the ego-vehicle, an identifier corresponding to a perceived object in the surrounding environment, and a control identifier corresponding to a real-time control mode of the ego-vehicle relative to the perceived object; and displaying the electronic image on an intelligent driving perception picture of the ego-vehicle.

Description

Image generation and display methods, devices, electronic equipment, products, media, and vehicles Cross-references to related applications

[0001] This application claims priority to Chinese patent application No. 202510103852.1, filed on January 22, 2025, and Chinese patent application No. 202510105943.9, filed on January 22, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to, but is not limited to, the field of intelligent driving technology, and particularly to a method, apparatus, electronic device, program product, storage medium, and vehicle for generating and displaying electronic images. Background Technology

[0003] When driving, people need to observe the surrounding environment with their naked eyes and analyze it based on their driving experience in order to make accurate driving judgments and control the vehicle to safely drive to the destination by following traffic rules and a planned route. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This application provides a method, apparatus, electronic device, program product, storage medium, and vehicle for generating and displaying electronic images.

[0006] In a first aspect, embodiments of this application provide a method for generating and displaying an electronic image, comprising: acquiring real-time perception data obtained by a vehicle's perception device perceiving the surrounding environment of the vehicle; generating an electronic image of the surrounding environment based on the real-time perception data; the electronic image comprising a vehicle identifier corresponding to the vehicle, an identifier corresponding to a perceived object in the surrounding environment, and a control identifier corresponding to the real-time control mode of the vehicle relative to the perceived object; and displaying the electronic image on the vehicle's intelligent driving perception screen.

[0007] In some embodiments of this application, generating an electronic image of the surrounding environment based on the real-time sensing data includes: generating a vehicle identifier corresponding to the vehicle; determining the relative position of the sensing object and the vehicle in the surrounding environment based on the real-time sensing data; generating an identifier corresponding to the sensing object at the relative position of the vehicle identifier with reference to the vehicle identifier, and generating a control identifier corresponding to the real-time control mode of the vehicle relative to the sensing object.

[0008] In some embodiments of this application, generating the identifier corresponding to the sensing object at the relative position of the vehicle identifier with reference to the vehicle identifier includes: determining the number of sensing objects based on the real-time sensing data and numbering the sensing objects; generating the identifier corresponding to the sensing object at the relative position of the vehicle identifier with reference to the vehicle identifier according to a preset display ratio, and generating respective numbers on the identifier corresponding to the sensing object itself and / or around it.

[0009] In some embodiments of this application, the sensing object includes other vehicles; determining the number of sensing objects based on the real-time sensing data and numbering the sensing objects includes: determining the number of other vehicles based on the real-time sensing data and numbering the other vehicles using a first type of counting symbol; generating the identifier corresponding to the sensing object at the relative position of the self-vehicle identifier according to a preset display ratio and with the self-vehicle identifier as a reference, and generating their respective numbers on the identifier corresponding to the sensing object itself and / or around it includes: generating the identifier of other vehicles corresponding to the other vehicles at the relative position of the self-vehicle identifier according to the preset display ratio and with the self-vehicle identifier as a reference, and generating their respective numbers on the identifier of other vehicles itself and / or around it.

[0010] In some embodiments of this application, the sensing object includes a lane, which includes the lane where the vehicle is located and other lanes where the vehicle is not located; determining the number of sensing objects based on the real-time sensing data and numbering the sensing objects includes: determining the number of lanes based on the real-time sensing data and numbering the lanes using a second type of counting symbol; generating the identifier corresponding to the sensing object at the relative position of the vehicle identifier according to a preset display ratio, with the vehicle identifier as a reference, and generating their respective numbers on and / or around the identifier corresponding to the sensing object, includes: generating the identifiers corresponding to the vehicle lane and the other lane at the relative position of the vehicle identifier according to the preset display ratio, with the vehicle identifier as a reference, and generating their respective numbers on and / or around the identifiers corresponding to the vehicle lane and the other lane; wherein, the identifier corresponding to the vehicle lane is generated below the vehicle identifier, and the identifiers corresponding to the other lanes are generated side by side with the identifier corresponding to the vehicle lane.

[0011] In some embodiments of this application, after generating respective numbers for the identifiers corresponding to the respective lanes and other lanes, the method for generating and displaying the electronic image further includes: determining whether the distance between one side of the vehicle and the boundary of the corresponding side of the lane is greater than a preset safety distance; in response to the distance between the one side of the vehicle and the boundary of the corresponding side of the lane being greater than the preset safety distance, generating an operable direction indicator pointing to the boundary of the corresponding side of the lane on one side of the vehicle identifier; in response to the distance between the one side of the vehicle and the boundary of the corresponding side of the lane not being greater than the preset safety distance, generating an inoperable direction indicator pointing to the boundary of the corresponding side of the lane on one side of the vehicle identifier; wherein the attributes of the operable direction indicator are different from those of the inoperable direction indicator, the operable direction indicator is configured to indicate that the position of the vehicle is allowed to be adjusted in the direction it points, and the inoperable direction indicator is configured to indicate that the position of the vehicle is prohibited from being adjusted in the direction it points.

[0012] In some embodiments of this application, generating an identifier corresponding to the sensing object at the relative position of the vehicle identifier includes: determining, based on the real-time sensing data and a preset intelligent driving strategy, whether there is an executable real-time control method for the vehicle relative to the sensing object; in response to the existence of an executable real-time control method for the vehicle relative to the sensing object, generating an operable identifier corresponding to the sensing object at the relative position of the vehicle identifier; in response to the absence of an executable real-time control method for the vehicle relative to the sensing object, generating an inoperable identifier corresponding to the sensing object at the relative position of the vehicle identifier; wherein the attributes of the operable identifier corresponding to the sensing object are different from the attributes of the inoperable identifier corresponding to the sensing object.

[0013] In some embodiments of this application, after displaying the electronic image on the intelligent driving perception screen of the vehicle, the method for generating and displaying the electronic image further includes: receiving voice commands input by the user for the control identifier in real time; and controlling the vehicle to drive according to the voice commands.

[0014] In some embodiments of this application, controlling the vehicle to drive according to the voice command includes: determining whether the control method in the voice command matches the control method indicated by the control identifier, and obtaining a determination result; and controlling whether the vehicle drives according to the voice command based on the determination result and the real-time perception data.

[0015] In some embodiments of this application, after controlling the vehicle to drive according to the voice command, the method for generating and displaying the electronic image further includes: obtaining the user's identity; obtaining the user's driving habits from pre-recorded user driving habits based on the user's identity; and controlling the vehicle to drive according to the user's driving habits.

[0016] In some embodiments of this application, the pre-recorded user driving habits are generated through the following steps: collecting and recording real-time perception data and voice commands during each driving process, analyzing the timbre of the voice commands, and identifying users who input voice commands with different timbres as different users; generating driving habits for different users based on the real-time perception data and voice commands with different timbres during each driving process.

[0017] In some embodiments of this application, after receiving the voice command input by the user for the control identifier in real time, the method for generating and displaying the electronic image further includes: according to at least one layer of safety judgment mechanism, if it is determined that the voice command is unreasonable, receiving the next command adjacent to the voice command and overwriting the voice command with the next command; the at least one layer of safety judgment mechanism is configured to indicate that the vehicle is driven safely after determining that the voice command is reasonable; and, if it is determined that the voice command is reasonable, allowing the vehicle to drive according to the voice command.

[0018] In some embodiments of this application, the at least one layer of security judgment mechanism includes a user security judgment mechanism. The step of receiving the next instruction adjacent to the voice instruction and overwriting the voice instruction when the voice instruction is determined to be unreasonable, based on the at least one layer of security judgment mechanism, includes: receiving the next instruction re-inputted by the user when the voice instruction is determined to be unreasonable, and overwriting the voice instruction; wherein the user security judgment mechanism is configured to indicate an instruction that has been corrected as unreasonable by the user.

[0019] In some embodiments of this application, the at least one layer of safety judgment mechanism further includes a driver safety judgment mechanism. The step of receiving a next instruction adjacent to the voice instruction and overwriting the voice instruction when the voice instruction is determined to be unreasonable, based on the at least one layer of safety judgment mechanism, includes: receiving the next instruction re-inputted by the driver when the voice instruction is determined to be unreasonable, and overwriting the voice instruction; wherein the driver safety judgment mechanism is configured to indicate that the driver assists the user in correcting the unreasonable instruction.

[0020] In some embodiments of this application, the step of receiving the next instruction re-inputted by the driver to overwrite the voice instruction when it is determined that the voice instruction is unreasonable includes: receiving a disabling instruction re-inputted by the driver to overwrite the voice instruction when it is determined that the abnormal frequency of the voice instruction is greater than a first frequency threshold; wherein the disabling instruction is configured to disable the control soft switch, and the disabling instruction is input through physical button operation.

[0021] In some embodiments of this application, the at least one layer of safety judgment mechanism further includes an intelligent driving system safety judgment mechanism. The step of receiving a next instruction adjacent to the voice instruction and overwriting the voice instruction when it is determined that the voice instruction is unreasonable, according to the at least one layer of safety judgment mechanism, includes: analyzing the rationality and risk of the voice instruction according to the intelligent driving system safety judgment mechanism to determine whether the voice instruction is reasonable; if it is determined that the voice instruction is unreasonable and exceeds the range that the intelligent driving system can handle, generating a request instruction for requesting manual intervention from the driver to overwrite the voice instruction; wherein, the intelligent driving system safety judgment mechanism is configured to indicate the rationality and risk of the intelligent driving system's judgment instruction. And / or, the at least one layer of security judgment mechanism further includes a background security monitoring mechanism. The step of receiving the next instruction adjacent to the voice instruction and overwriting the voice instruction when the voice instruction is determined to be unreasonable according to the at least one layer of security judgment mechanism includes: receiving a deactivation instruction from the background when the background security monitoring mechanism detects that the frequency of disabling exceeds a second frequency threshold, and overwriting the voice instruction with the deactivation instruction, so that the vehicle owner account can apply for relearning and then reactivate the function; wherein the deactivation instruction is configured to indicate the termination of the intelligent driving function.

[0022] Secondly, embodiments of this application provide an electronic image generation and display device, comprising: an acquisition module configured to acquire real-time perception data obtained by a vehicle's perception device perceiving the vehicle's surrounding environment; a generation module configured to generate an electronic image of the surrounding environment based on the real-time perception data; the electronic image including a vehicle identifier corresponding to the vehicle, an identifier corresponding to a perceived object in the surrounding environment, and a control identifier corresponding to the real-time control mode of the vehicle relative to the perceived object; and a display module configured to display the electronic image on the vehicle's intelligent driving perception screen.

[0023] Thirdly, embodiments of this application provide an electronic device including one or more processors configured to implement the method for generating and displaying an electronic image as described in any of the first aspects.

[0024] Fourthly, embodiments of this application provide a computer program product, including a computer program stored on a computer-readable storage medium, wherein when the computer program is executed by at least one processor, it implements the method for generating and displaying an electronic image as described in any one of the first aspects.

[0025] Fifthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by at least one processor, implement the method for generating and displaying an electronic image as described in any one of the first aspects.

[0026] In a sixth aspect, embodiments of this application provide a vehicle comprising: an electronic image generation and display device as described in the second aspect; or an electronic device as described in the third aspect; or a computer program product as described in the fourth aspect; or a non-transitory computer-readable storage medium as described in the fifth aspect; or one or more processors configured to implement the electronic image generation and display method as described in any one of the first aspects.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Other aspects will become clear after reading and understanding the accompanying drawings and detailed description. Attached Figure Description

[0028] The accompanying drawings are included to provide a further understanding of the technical solutions of this application, are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application, and are used together with the embodiments to explain the principles of this application, but do not constitute a limitation on the technical solutions of this application.

[0029] Figure 1 is a flowchart illustrating the electronic image generation and display method according to an embodiment of this application.

[0030] Figure 2 is a schematic diagram of an electronic image according to an embodiment of this application.

[0031] Figure 3 is a flowchart illustrating a method for generating and displaying electronic images according to another embodiment of this application.

[0032] Figure 4 is a schematic diagram of the process of generating the identifier corresponding to the perceived object in the electronic image generation and display method of this application embodiment.

[0033] Figure 5 is a flowchart illustrating the vehicle safety control method according to an embodiment of this application.

[0034] Figure 6 is a schematic diagram of the safety judgment mechanism of the vehicle safety control method shown in Figure 5.

[0035] Figure 7 shows a schematic diagram of the soft switch of the vehicle safety control method shown in Figure 5.

[0036] Figure 8 shows a schematic diagram of the physical button used to generate a disable command in the vehicle safety control method shown in Figure 5.

[0037] Figure 9 is a structural block diagram of an electronic image generation and display device according to an embodiment of this application.

[0038] Figure 10 is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0039] The technical solutions in the embodiments of this application will be clearly and completely described herein with reference to the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0040] The embodiments described below are not representative of all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims. It should be noted that in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments. In real-world driving scenarios, the surrounding environment of a vehicle is highly complex, leading to rapidly changing road conditions. Drivers must not only observe the surrounding environment but also analyze it to make timely and accurate driving judgments. Distractions during driving can easily lead to safety risks.

[0041] In view of this, to address the technical problem of users needing to observe and analyze the vehicle's surroundings to make driving decisions and to reduce the safety risks caused by user distraction, this application provides a method for generating and displaying electronic images. Based on real-time perception data from the vehicle's sensing devices, an electronic image of the vehicle's surrounding environment is generated. This electronic image visually displays the vehicle's own identification and the identification of perceived objects in the surrounding environment, allowing users to quickly and accurately grasp information about their vehicle's surroundings by viewing the image. Furthermore, since the electronic image also includes control indicators corresponding to the real-time control methods of the vehicle relative to perceived objects in the surrounding environment, users can also determine which real-time control methods can be executed relative to which perceived objects in the surrounding environment, eliminating the need for analysis and judgment based on driving experience and reducing the safety risks caused by user distraction.

[0042] The electronic image generation and display method provided in this application can be applied to vehicles with displays, especially to intelligent driving systems of vehicles. Please refer to Figure 1, which is a schematic flowchart of the electronic image generation and display method according to an embodiment of this application. The electronic image generation and display method provided in this application includes, but is not limited to, the following steps S110 to S130.

[0043] In step S110, real-time perception data obtained by the vehicle's perception devices from perceiving the vehicle's surrounding environment is acquired.

[0044] Intelligent driving systems utilize various sensors installed on the vehicle (such as millimeter-wave radar, lidar, mono / dual-lens cameras, and satellite navigation) to perceive the vehicle's surroundings in real time during operation. This real-time perception data is used to identify, detect, and track static and dynamic objects, and combined with navigation map data for system calculations and analysis. This allows the system to proactively alert the user to potential dangers, effectively improving driving comfort and safety. Here, the vehicle's perception devices can be various sensors installed on the vehicle. By acquiring real-time perception data from these sensors, the intelligent driving system controls the vehicle to safely travel to its destination along a planned route, adhering to traffic rules.

[0045] In step S120, an electronic image of the surrounding environment is generated based on the real-time sensing data; the electronic image includes the vehicle identifier corresponding to the vehicle, the identifier corresponding to the sensing object in the surrounding environment, and the control identifier corresponding to the real-time control mode of the vehicle relative to the sensing object.

[0046] As the "eyes and ears" of an intelligent driving system, the vehicle's sensing devices acquire real-time perception data of the vehicle's surroundings through these devices, allowing the system to indirectly perceive the vehicle's environment. In this embodiment, the vehicle's surrounding environment includes lanes and vehicles along the route. Vehicles include the vehicle itself and other vehicles, and lanes include the vehicle's own lane and other lanes not occupied by the vehicle. Therefore, the objects of perception in the surrounding environment can include other vehicles and lanes including the vehicle's own lane and other lanes.

[0047] After acquiring real-time perception data from the vehicle's sensing devices, the intelligent driving system generates an electronic image of the vehicle's surroundings based on this data. This electronic image includes the vehicle's identifier, identifiers corresponding to perceived objects in the surrounding environment, and control identifiers. In this embodiment, the vehicle identifier and the identifiers corresponding to perceived objects are used to indicate the vehicle and other vehicles and lanes in the surrounding environment, respectively. The control identifiers indicate the real-time control method of the vehicle relative to perceived objects in the surrounding environment, and may include vehicle control identifiers corresponding to the real-time control method of the vehicle relative to other vehicles and / or lane control identifiers corresponding to the real-time control method of the vehicle relative to lanes including its own lane and other lanes.

[0048] Please refer to Figure 2, which is a schematic diagram of an electronic image according to an embodiment of this application. The electronic image includes a vehicle identifier, a other vehicle identifier, a lane identifier, a vehicle control identifier, and a lane control identifier. The lane identifier may include a lane graphic and / or a lane number, and the other vehicle identifier may include a other vehicle graphic and / or a other vehicle number. The electronic image shown in Figure 2 will be described in detail below.

[0049] Lane markings include lane graphics and lane numbers ①, ②, ③, and ④. A solid line and a dashed line form the lane graphic for the rightmost lane (lane ①) traveling in the same direction as the vehicle. Two dashed lines form the lane graphic for the middle lane (lane ②) traveling in the same direction as the vehicle. A dashed line and a solid line form the lane graphic for the leftmost lane (lane ③) traveling in the same direction as the vehicle. A dashed line and a solid line form the lane graphic for the leftmost lane (lane ④) traveling in the opposite direction to the vehicle.

[0050] The other vehicle identification includes a graphic of the other vehicle and its number, A, B, C, or D. Vehicles A, B, and C are traveling in the same direction as your vehicle, respectively, in lanes ①, ②, and ③. Vehicle D is traveling in the opposite direction to your vehicle, in lane ④.

[0051] Lane control signs include the first letters of the Chinese pinyin for the real-time lane control mode: B, Y, and S. B indicates lane changing, Y indicates priority lane driving, and S indicates lane locking. If the vehicle can change lanes to lane 1 or 2 and maintain priority or lock in lane 1 or 2, the lane control signs corresponding to lane 1 and 2 will be B, Y, and S, representing lane changing, priority lane driving, and lane locking, respectively. If the vehicle is currently in lane 3 and cannot change lanes to lane 3 but can maintain priority or lock in lane 3, the lane control signs corresponding to lane 3 will be Y and S, representing priority lane driving and lane locking, respectively. If the vehicle cannot change lanes to lane 4 or maintain priority or lock in lane 4, the lane control sign corresponding to lane 4 will be empty.

[0052] The vehicle control indicators include the first letters of the Chinese pinyin for the real-time vehicle control mode: C, G, and S. C indicates overtaking, G indicates priority following, and S indicates locked following. If the vehicle cannot overtake, follow, or lock-follow vehicle A or D, the vehicle control indicator corresponding to the other vehicle indicator for A and D will be empty. If the vehicle can follow or lock-follow vehicle B, the vehicle control indicator corresponding to the other vehicle indicator for B will be G and S, representing following and locked following, respectively. If the vehicle can overtake, follow, or lock-follow vehicle C, the vehicle control indicator corresponding to the other vehicle indicator for C will be C, G, and S, representing overtaking, following, and locked following, respectively.

[0053] It should be noted that the display methods of lane signs and lane control signs, and the display methods of other vehicle signs and vehicle control signs, are not limited to those shown in Figure 2. Each lane control sign is not necessarily displayed above or around its corresponding lane number, and each vehicle control sign is not necessarily displayed below or around its corresponding other vehicle sign. It is sufficient to establish and display a connection between each lane control sign and its corresponding lane sign, and between each vehicle control sign and its corresponding other vehicle sign. For example, different lane signs and other vehicle signs can be set to different colors, and each lane control sign and vehicle control sign can be set to the same color as its corresponding lane sign and other vehicle sign. This application embodiment does not limit the specific display methods of lane signs and lane control signs, and the specific display methods of other vehicle signs and vehicle control signs.

[0054] It should also be noted that in the electronic image shown in Figure 2, the vehicle's graphic is black, the graphics of other vehicles (B and C) are white, the graphics of other vehicles (A and D) are gray, the lane graphics of lanes ①, ②, and ③ are white, and the lane graphic of lane ④ is gray (represented by diagonally filled lines in the figure). A white other vehicle graphic indicates that the current vehicle can perform at least one vehicle control mode (such as overtaking or following) relative to the other vehicle represented by that graphic, while a gray other vehicle graphic indicates that the current vehicle cannot perform any vehicle control mode relative to the other vehicle represented by that graphic. Similarly, a white lane graphic indicates that the current vehicle can perform at least one lane control mode (such as changing lanes) relative to the lane represented by that lane graphic, while a gray lane graphic indicates that the current vehicle cannot perform any lane control mode relative to the lane represented by that lane graphic. In specific scenarios, the colors of the other vehicle graphics corresponding to different other vehicles and the lane graphics corresponding to different lanes can be specifically set, and the colors of the other vehicle numbers corresponding to different other vehicles and the lane numbers corresponding to different lanes can also be different accordingly. For example, the vehicle identification mark can be set to the vehicle's actual color, the other vehicle identification mark (which can be controlled by at least one vehicle control method) can be set to green, and the other vehicle identification mark (which cannot be controlled by any vehicle control method) can be set to red. This application embodiment does not limit the specific colors of different lane markings, different other vehicle identification marks, and the vehicle identification mark.

[0055] In step S130, the electronic image is displayed on the vehicle's intelligent driving perception screen.

[0056] By displaying electronic images containing the vehicle's identifier, the identifiers corresponding to perceived objects in the surrounding environment, and the control identifiers corresponding to the real-time control methods of the vehicle relative to the perceived objects on the vehicle's Advanced Driver Assistance System View (ADV), users can grasp information about the vehicle's surrounding environment and know which real-time control methods the vehicle can perform relative to which perceived objects in the surrounding environment simply by viewing the electronic images on the ADV, without needing to analyze and judge based on driving experience, thus reducing the safety risks caused by the user's distraction.

[0057] Please refer to Figures 1, 2 and 3 together. Figure 3 shows a flowchart of a method for generating and displaying electronic images according to another embodiment of this application.

[0058] In some embodiments, when generating an electronic image of the surrounding environment based on the real-time sensing data in step S120, the process specifically includes the following steps S121 to S123.

[0059] In step S121, a vehicle identifier corresponding to the vehicle is generated.

[0060] Because the vehicle's surroundings are constantly changing while it is in motion, and the sensing devices are mounted on the vehicle body, the location of objects in the surrounding environment can only be determined by using the vehicle itself as a reference. Therefore, it is necessary to first generate the vehicle's identifier corresponding to the vehicle that is "stationary" relative to the sensing devices, and then generate the identifier corresponding to the objects that are "moving" relative to the sensing devices.

[0061] In step S122, the relative position of the sensing object and the vehicle is determined based on the real-time sensing data.

[0062] By collecting real-time sensing data from sensing devices, the position of a sensing object in the surrounding environment relative to the vehicle can be determined, that is, the relative position of the sensing object and the vehicle. In this embodiment, there may be multiple sensing objects in the surrounding environment within the sensing range of the sensing device, and each sensing object has a relative position with the vehicle. For example, if the first sensing object is the vehicle's lane, then the first sensing object is located directly below the vehicle; if the second sensing object is a vehicle in front of the vehicle in the left or right adjacent lane, then the second sensing object is located in front of the vehicle on the left or right.

[0063] In step S123, with the vehicle identifier as a reference, an identifier corresponding to the sensing object is generated at the relative position of the vehicle identifier, and a control identifier corresponding to the real-time control mode of the vehicle relative to the sensing object is generated.

[0064] Treating the vehicle sign as the origin of the reference coordinate system, and using the vehicle sign as a reference, the system generates a sign corresponding to the perceived object at the relative position of the perceived object and the vehicle sign, based on the relative position of the perceived object and the vehicle sign. For example, if the vehicle lane is directly below the vehicle, then the sign corresponding to the vehicle lane is generated directly below the vehicle sign; if another vehicle in front of the vehicle lane is directly in front of the vehicle, then the sign corresponding to that other vehicle in front of the vehicle lane is generated directly in front of the vehicle sign.

[0065] In addition, the intelligent driving system also needs to determine the real-time control methods that the vehicle can perform relative to the perceived objects in the surrounding environment based on real-time perception data and preset intelligent driving strategies, and generate control icons corresponding to these real-time control methods. This allows the system to show the user which real-time control methods the vehicle can perform relative to the perceived objects in the surrounding environment, so that the user can input commands in a targeted manner, rather than blindly inputting commands according to their own wishes and then being told by the intelligent driving system whether the command can be executed. This reduces the user's operation steps, improves the user experience of autonomous driving, and saves the vehicle's network and power resources.

[0066] In some embodiments, when generating the identifier corresponding to the perceived object at the relative position of the vehicle identifier in step S123, the following steps S124 to S126 are specifically included.

[0067] In step S124, based on the real-time perception data and the preset intelligent driving strategy, it is determined whether there is an executable real-time control method for the vehicle relative to the perceived object.

[0068] Because multiple objects may exist in the surrounding environment within the sensing range of the sensing device, the vehicle may not have an executable control mode relative to each of these objects. For example, in the electronic image shown in Figure 2: vehicle B is a vehicle in another lane traveling in the same direction as the vehicle, and the vehicle can currently perform a following control mode relative to vehicle B; vehicle C is a vehicle in its own lane traveling in the same direction as the vehicle, and the vehicle can currently perform overtaking or following control modes relative to vehicle C; vehicle D is a vehicle traveling in the opposite direction to the vehicle, and the vehicle cannot currently perform any control mode relative to vehicle D. The intelligent driving system, based on real-time sensing data and preset intelligent driving strategies, can determine whether the vehicle has an executable real-time control mode relative to each sensing object.

[0069] In step S125, if the vehicle has an executable real-time control mode relative to the sensing object, an operable identifier corresponding to the sensing object is generated at the relative position of the vehicle identifier.

[0070] If the vehicle has an executable real-time control method relative to a certain sensing object, then in addition to generating a corresponding control identifier, based on the relative position of the sensing object and the vehicle, the identifier corresponding to the sensing object generated at the relative position of the vehicle's identifier is an identifier indicating operability. For example, in the electronic image shown in Figure 2: if the vehicle has an executable real-time control method relative to vehicles B and C, then the identifiers corresponding to vehicles B and C are white, indicating that they are operable; if the vehicle has an executable real-time control method relative to lanes ①, ②, and ③, then the identifiers corresponding to lanes ①, ②, and ③ are white, indicating that they are operable. It should be noted that in the electronic image, the operable identifier corresponding to the sensing object is not limited to white, but can also be other colors, such as green. This application embodiment does not limit the specific color of the operable identifier corresponding to the sensing object.

[0071] In step S126, if there is no executable real-time control method for the vehicle relative to the sensing object, an inoperable identifier corresponding to the sensing object is generated at the relative position of the vehicle identifier.

[0072] If there is no executable real-time control method for the vehicle relative to a certain sensing object, then in addition to not generating a corresponding control identifier, the identifier corresponding to the sensing object generated at the relative position of the vehicle's identifier, based on the relative position of the sensing object and the vehicle, will be an identifier indicating inoperability. For example, in the electronic image shown in Figure 2: if there is no executable real-time control method for the vehicle relative to vehicles A and D, then the identifiers corresponding to vehicles A and D will be gray, indicating that they are inoperable; if there is no executable real-time control method for the vehicle relative to lane ④, then the identifier corresponding to lane ④ will be gray, indicating that it is inoperable. It should be noted that the inoperable identifier corresponding to the sensing object in the electronic image is not limited to gray, but can also be other colors, such as red. This application embodiment does not limit the specific color of the inoperable identifier corresponding to the sensing object.

[0073] It should be noted that the operable identifiers corresponding to the perceived object and the non-operable identifiers corresponding to the perceived object are not limited to different colors; they can also be different in solid or dashed form and / or other ways. For example, an operable identifier can be generated using a solid line of one color, and a non-operable identifier can be generated using a dashed line of another color; a larger operable identifier can be generated using a solid line, and a smaller non-operable identifier can be generated using a dashed line. This application does not limit the specific manifestation of the difference between the operable identifiers corresponding to the perceived object and the non-operable identifiers corresponding to the perceived object, as long as the attributes of the operable identifiers corresponding to the perceived object are different from the attributes of the non-operable identifiers corresponding to the perceived object.

[0074] In vehicles equipped with autonomous driving capabilities, Advanced Driver Assistance Systems (ADAS) typically control the vehicle according to preset driving modes. If the preset driving mode does not suit the user's preferences, and the user wants to drive the vehicle according to their own instructions, they need to request the ADAS to control the vehicle according to their input through human-machine interaction. However, the user's input instructions may pose safety risks and may not conform to the ADAS's preset intelligent driving strategy. This could result in the ADAS informing the user that the input instruction cannot be executed, requiring the user to try inputting different instructions. This process may occur multiple times, wasting the vehicle's network and power resources, increasing the user's operational steps, and negatively impacting the user experience of autonomous driving.

[0075] To address the technical problem that user-inputted commands cannot be executed and require attempts to re-input other commands, in some embodiments, after displaying the electronic image on the vehicle's intelligent driving perception screen in step S130 above, the electronic image generation and display method of this application embodiment further includes the following steps S140 and S150.

[0076] In step S140, voice commands sent by the user for the control identifier are received in real time.

[0077] Because users may want their vehicles to perform different real-time control actions relative to perceived objects in the surrounding environment under different road conditions, the voice command sent by the user one second may be canceled the next second. Therefore, it is necessary to receive the voice commands sent by the user to the control signs in the electronic image in real time to meet the user's real-time needs.

[0078] Receiving voice commands from users can be achieved through the vehicle's audio recording equipment (such as a microphone). For specific implementation details, please refer to relevant documentation in the field; these will not be elaborated upon here.

[0079] In step S150, the vehicle is controlled to move according to the voice command.

[0080] The electronic image generation and display method provided in this application embodiment displays the electronic image on the intelligent driving perception screen of the vehicle. The electronic image includes the vehicle's corresponding vehicle identifier, the identifiers corresponding to the perceived objects in the surrounding environment, and the control identifiers corresponding to the real-time control methods of the vehicle relative to the perceived objects in the surrounding environment. This allows the user to perceive the vehicle's surrounding environment in real time by viewing the electronic image and to know which real-time control methods can be executed relative to which perceived objects in the surrounding environment. Furthermore, after the electronic image is displayed on the vehicle's intelligent driving perception screen, voice commands sent by the user in response to the control identifiers in the electronic image are received in real time. This allows the user to send targeted voice commands based on the control identifiers in the electronic image, reducing the need to resend other voice commands when they cannot be executed during autonomous driving. This saves the vehicle's network and power resources, reduces the user's operation steps, and improves the user experience of autonomous driving.

[0081] In some embodiments, when controlling the vehicle to drive according to the voice command in step S150, the steps S151 and S152 are specifically included.

[0082] In step S151, it is determined whether the control method in the voice command matches the control method indicated by the control identifier, and a determination result is obtained.

[0083] Because the control method in the voice command sent by the user may not be consistent with the control method indicated by the control identifier, it is necessary to determine whether the control method in the voice command matches the control method indicated by the control identifier before executing the user's voice command, and then obtain the judgment result.

[0084] For example, in the electronic image shown in Figure 2, the vehicle control icons corresponding to the vehicle B's other vehicle icon are G and S, representing following and locked following, indicating that the user's vehicle can currently follow or lock onto following vehicle B. If the user sends a voice command to follow vehicle B, the vehicle control method in the voice command matches the vehicle control method indicated by the vehicle control icon, and the result is a match result; if the user sends a voice command to overtake vehicle B, the vehicle control method in the voice command does not match the vehicle control method indicated by the vehicle control icon, and the result is a mismatch result.

[0085] In step S152, based on the judgment result and the real-time perception data, the system controls whether the vehicle drives according to the voice command.

[0086] Even if the judgment result obtained in step S151 is a match, the intelligent driving system will not control the vehicle to drive according to the voice command if the real-time road conditions do not permit it. In other words, regardless of whether the judgment result obtained in step S151 is a match or a non-match, the intelligent driving system must still determine whether the real-time road conditions are safe before deciding whether to control the vehicle to drive according to the voice command. For example, in the electronic image shown in Figure 2, if the user sends a voice command to overtake vehicle C, although the control method in the voice command matches the control method indicated by the control icon, and the judgment result is a match, if vehicle B slows down at this time, and another vehicle is approaching from behind vehicle B in lane ②, then the real-time road conditions are not safe, and the system cannot immediately control the vehicle to overtake vehicle C. Only when the judgment result obtained in step S151 is a match and the real-time road conditions are safe will the intelligent driving system control the vehicle to drive according to the voice command. Specifically, the intelligent driving system can determine whether the real-time road conditions are safe by acquiring real-time perception data obtained by the vehicle's perception devices from perceiving the surrounding environment. For guidance on how to determine whether real-time road conditions are safe based on real-time sensing data, please refer to relevant explanations in this field, which will not be elaborated here.

[0087] In some embodiments, the intelligent driving system may wait until the real-time road conditions are safe before controlling the vehicle to drive according to the voice command. In other embodiments, the intelligent driving system may also discard the currently received voice command and wait for the user to resend the voice command before executing the above steps S151 and S152. This application does not limit the specific processing method of voice commands when real-time road conditions are unsafe.

[0088] In the electronic image generation and display method provided in this application embodiment, after receiving the user's voice command input for the control mark in the electronic image in real time, and deciding whether to control the vehicle to drive according to the voice command based on the judgment result, the method also comprehensively considers whether the real-time road conditions indicated by the real-time perception data are safe. This improves the personalization of the intelligent driving system while ensuring the safety of autonomous driving, thereby further improving the user experience of autonomous driving.

[0089] In addition to the above-mentioned method of judging the safety of user-sent commands by determining whether real-time road conditions are safe, this application embodiment also provides a vehicle safety control method to realize the safety judgment of user-issued commands. It will be described in detail below with reference to Figures 5 to 8, and will not be described in detail here.

[0090] In some embodiments, after controlling the vehicle to drive according to the voice command in step S150 above, the electronic image generation and display method of this application embodiment further includes the following steps S160 to S180.

[0091] In step S160, the identity of the current user is obtained.

[0092] Considering that most families currently cannot afford to have each family member with their own dedicated vehicle, and that most families with private cars typically have two or more family members sharing a single car, it's highly likely that different people will drive the same car, and these drivers will have different driving habits. Before starting the vehicle, the current user's identity can be obtained through login authentication; after starting the vehicle, it can also be obtained through voice recognition. Of course, other methods can also be used to obtain the current user's identity before and / or after starting the vehicle, which will not be listed here.

[0093] In step S170, the driving habits of the current user are obtained from the pre-recorded driving habits of the user based on the current user's identity.

[0094] Intelligent driving systems can pre-record each user's driving habits. After obtaining the current user's identity, they can retrieve the current user's driving habits from the pre-recorded driving habits.

[0095] It's important to note that driving habits are not a single piece of data, but rather a complex set of data combining real-time road conditions indicated by real-time perception data and the control methods executed by the user under those specific conditions. For insights into recording driving habits, please refer to the shadow model in the field of computer algorithms; further details will not be elaborated upon here.

[0096] In step S180, the vehicle is controlled to drive according to the current user's driving habits.

[0097] In some embodiments, the pre-recorded user driving habits in step S170 above are generated by the following steps: collecting and recording the real-time perception data and the voice command during each driving process, analyzing the timbre of the voice command, identifying users who send voice commands with different timbres as different users; and generating driving habits for different users based on the real-time perception data and the voice commands with different timbres during each driving process.

[0098] By analyzing the timbre of the voice commands sent by users, it is possible to distinguish the different identities of users who send voice commands with different timbres, and generate different driving habits of different users based on the real-time perception data collected and recorded during each driving process and the voice commands with different timbres.

[0099] In some embodiments, after recognizing the user sending a voice command based on its timbre, the intelligent driving system can generate corresponding driving habits by recording only the voice commands sent by that user that differ from its historical records. Generating user driving habits based on discrepancy data can more accurately target user preferences while also saving vehicle network and storage resources.

[0100] In other embodiments, after recognizing the user sending the voice command based on its timbre, the intelligent driving system can record all the voice commands sent by that user to generate corresponding driving habits. Generating user driving habits based on all data allows for a more comprehensive understanding of user preferences and also benefits the learning and training of the intelligent driving system.

[0101] Please refer to Figures 2, 3 and 4 together. Figure 4 is a schematic flowchart of the process of generating the identifier corresponding to the perceived object in the electronic image generation and display method of this application embodiment.

[0102] In some embodiments, when generating the identifier corresponding to the perceived object at the relative position of the vehicle identifier with reference to the vehicle identifier in step S123, the steps S210 and S220 are specifically included.

[0103] In step S210, the number of the sensing objects is determined based on the real-time sensing data, and the sensing objects are numbered.

[0104] By collecting real-time sensing data from sensing devices, the position of sensed objects in the surrounding environment relative to the vehicle can be determined, as can the number of sensed objects in the surrounding environment. To facilitate differentiation of these sensed objects, each sensed object can be numbered. For example, each sensed object can be numbered according to the order in which it appears in the electronic image from right to left. Alternatively, it can be numbered according to the order in which it appears in the electronic image from left to right. This application embodiment does not limit the specific numbering order of the sensed objects. In this application embodiment, sensed objects of the same type can be numbered using one sequence number, and sensed objects of different types can be numbered using another sequence number.

[0105] In step S220, according to a preset display ratio, with the vehicle logo as a reference, an identifier corresponding to the sensing object is generated at the relative position of the vehicle logo, and a number is generated on the identifier corresponding to the sensing object itself and / or around it.

[0106] In reality, the distance between a perceived object and the vehicle may range from several meters to tens of meters. However, the distance between the sign corresponding to the perceived object in the electronic image and the vehicle's sign cannot actually be set to several meters to tens of meters. Therefore, it is necessary to generate the sign corresponding to the perceived object at the relative position of the vehicle's sign, using the vehicle's sign as a reference, according to a preset display ratio, such as a 1:500 ratio. Furthermore, a number for the perceived object should be generated around the sign itself and / or around the sign corresponding to the perceived object. For example, if a real-world perceived object is located 15 meters due north relative to the vehicle, then the relative position of the sign corresponding to that perceived object in the electronic image would be 3 centimeters due north.

[0107] In some embodiments, the sensing object in step S210 is another vehicle; when determining the number of sensing objects based on the real-time sensing data and numbering the sensing objects in step S210, it specifically includes the following step S211; when generating the identifier corresponding to the sensing object at the relative position of the vehicle identifier according to the preset display ratio and with the vehicle identifier as a reference, and generating their respective numbers on the identifier corresponding to the sensing object itself and / or around it in step S220, it specifically includes the following step S221.

[0108] In step S211, the number of other vehicles is determined based on the real-time sensing data, and the first type of counting symbols is used to number the other vehicles.

[0109] When the object of perception in the surrounding environment is another vehicle, the real-time perception data collected by the sensing device can determine the position of the other vehicle relative to the own vehicle, as well as the number of other vehicles in the surrounding environment. In this case, the first type of counting symbol is used to number the other vehicles in the surrounding environment.

[0110] For example, in the electronic image shown in Figure 2, uppercase English letters are used to number the vehicles in the surrounding environment according to their order of appearance from right to left, resulting in vehicles A, B, C, and D. It should be noted that the first type of counting symbols is not limited to uppercase English letters; it can also be other types of counting symbols, such as Roman numerals. This application does not limit the specific type of the first type of counting symbols.

[0111] In step S221, according to a preset display ratio, with the vehicle identifier as a reference, a corresponding vehicle identifier is generated at the relative position of the vehicle identifier, and a number is generated on the vehicle identifier itself and / or around it.

[0112] For example, in the electronic image shown in Figure 2, based on the relative positions of vehicles A, B, C, and D with the vehicle itself, and according to a preset display ratio, using the vehicle's identifier as a reference, the identifiers of other vehicles corresponding to vehicles A, B, C, and D are generated at their relative positions. The identifiers A, B, C, and D are then generated on these other vehicle identifiers. It should be noted that the identifiers A, B, C, and D may not be generated on the other vehicle identifiers themselves, but rather around them. This embodiment of the application does not limit the specific location of the other vehicle identifier numbers.

[0113] In some embodiments, the sensing object in step S210 is a lane, which includes the lane where the vehicle is located and other lanes where the vehicle is not located; when determining the number of sensing objects and numbering the sensing objects based on the real-time sensing data in step S210, it specifically includes the following step S212; when generating the corresponding identifier of the sensing object at the relative position of the vehicle identifier according to a preset display ratio and with the vehicle identifier as a reference, and generating their respective numbers on the identifier corresponding to the sensing object itself and / or around it in step S220, it specifically includes the following step S222.

[0114] In step S212, the number of lanes is determined based on the real-time sensing data, and the lanes are numbered using the second type of counting symbols.

[0115] When the perceived object in the surrounding environment is a lane, the real-time sensing data collected by the sensing device can determine the position of the lane relative to the vehicle, thus distinguishing the vehicle's lane from other lanes, and also determining the number of lanes in the surrounding environment. In this case, to distinguish the lanes from other vehicles, a second type of counting symbol is used to number the lanes in the surrounding environment.

[0116] For example, in the electronic image shown in Figure 2, circled Arabic numerals are used to number the lanes in the surrounding environment in the order they appear in the electronic image from right to left, resulting in lane ①, lane ②, lane ③, and lane ④. It should be noted that the second type of counting symbols is not limited to circled Arabic numerals; it can also be other types of counting symbols, such as Chinese numerals. This application does not limit the specific type of the second type of counting symbols.

[0117] In step S222, according to a preset display ratio, with the vehicle sign as a reference, a sign corresponding to the vehicle lane and a sign corresponding to the other lane are generated at the relative position of the vehicle sign, and their respective numbers are generated on the vehicle lane and the other lane's respective signs and / or around them; wherein, the sign corresponding to the vehicle lane is generated below the vehicle sign, and the sign corresponding to the other lane is generated side by side with the sign corresponding to the vehicle lane.

[0118] For example, in the electronic image shown in Figure 2, lane 3 can be determined as the vehicle's lane based on the relative positions of lanes 1, 2, 3, and 4 with the vehicle. Then, according to the preset display ratio, with the vehicle's sign as a reference, a lane sign corresponding to lane 3 is generated directly below the vehicle's sign. At the relative position of the vehicle's sign, lane signs corresponding to lanes 1, 2, and 4 are generated side by side with the lane sign corresponding to lane 3. Finally, the numbers ①, ②, ③, and ④ are generated on the lane signs corresponding to lanes 1, 2, 3, and 4, respectively.

[0119] In some embodiments, after generating the sign corresponding to the lane and the sign corresponding to the other lane at the relative position of the vehicle sign according to the preset display ratio in step S222, and generating their respective numbers on and / or around the signs corresponding to the lane and the other lane, the electronic image generation and display method of this application embodiment further includes the following steps S223 to S225.

[0120] In step S223, it is determined whether the distance between one side of the vehicle and the boundary of the corresponding side of the lane is greater than a preset safety distance.

[0121] Because a vehicle cannot drive too close to one side of its lane while in motion, otherwise it may scrape against other vehicles in the adjacent lane and / or roadside greenery. Therefore, a safety distance needs to be preset so that the distance between the two sides of the vehicle and the corresponding side boundaries of the lane is not less than the safety distance.

[0122] In step S224, in response to the distance between one side of the vehicle and the boundary of the corresponding side of the lane being greater than a preset safety distance, an operable direction indicator pointing to the boundary of the corresponding side of the lane is generated on the side of the vehicle identifier; wherein, the operable direction indicator is used to indicate that the position of the vehicle can be adjusted in the direction it points.

[0123] In step S225, in response to the distance between one side of the vehicle and the boundary of the corresponding side of the lane not being greater than a preset safety distance, an inoperable direction indicator pointing to the boundary of the corresponding side of the lane is generated on the side of the vehicle identifier; wherein, the inoperable direction indicator is used to indicate that the position of the vehicle is prohibited from being adjusted in the direction it points to.

[0124] While the vehicle is in motion, the user may want to fine-tune the distance between the vehicle and the boundaries of its lane. Therefore, directional indicator signs can be generated on both sides of the vehicle icon to indicate how to fine-tune the distance between the vehicle and the lane boundaries.

[0125] If the distance between the left side of the vehicle and the left edge of the lane is greater than the safe distance, an operable directional indicator is generated to the left of the vehicle's indicator; if the distance between the left side of the vehicle and the left edge of the lane is not greater than the safe distance, an inoperable directional indicator is generated to the left of the vehicle's indicator.

[0126] If the distance between the right side of the vehicle and the right edge of the lane is greater than the safe distance, an operable directional indicator is generated to the right of the vehicle indicator; if the distance between the right side of the vehicle and the right edge of the lane is not greater than the safe distance, an inoperable directional indicator is generated to the right of the vehicle indicator.

[0127] In this embodiment, the attributes of the operable direction indicators are different from those of the non-operable direction indicators. For example, the operable direction indicators can be set to one color, and the non-operable direction indicators to another color, using the difference in color to represent the difference between the operable and non-operable direction indicators; alternatively, solid lines can be used to generate the operable direction indicators, and dashed lines can be used to generate the non-operable direction indicators, using the difference between solid and dashed lines to represent the difference between the operable and non-operable direction indicators; furthermore, larger operable direction indicators and smaller non-operable direction indicators can be generated, using the difference in size to represent the difference between the operable and non-operable direction indicators. This embodiment does not limit the specific manifestation of the difference between the operable and non-operable direction indicators, as long as the attributes of the operable and non-operable direction indicators are different.

[0128] This application also provides a vehicle safety control method to provide a mechanism for making safety judgments based on user-issued commands. Please refer to Figures 5 to 8. Figure 5 is a flowchart illustrating the vehicle safety control method of this application embodiment; Figure 6 is a schematic diagram of the safety judgment mechanism of the vehicle safety control method shown in Figure 5; Figure 7 is a schematic diagram of the soft switch in the vehicle safety control method shown in Figure 5; and Figure 8 is a schematic diagram of the physical button used to generate a disable command in the vehicle safety control method shown in Figure 5.

[0129] As shown in Figure 5, the vehicle safety control method may include, but is not limited to, steps 310 to 340.

[0130] In step 310, electronic images of the intelligent driving perception screen are acquired in real time.

[0131] In this embodiment, the electronic image is used to display vehicle identification information and surrounding environment identification information. The electronic image may include one or more of the vehicle identification and surrounding environment identification information. For example, the electronic image may display information about the actual distance of the vehicle relative to the surrounding environment; correspondingly, the display distance of the vehicle identification relative to the surrounding environment identification is displayed on the electronic image. Surrounding environment identification may include, but is not limited to, other vehicle identification and lane markings in the surrounding environment. Vehicle identification, other vehicle identification, and lane markings represent information indicating road conditions for the vehicle and the surrounding environment, respectively representing the actual distribution of the vehicle, other vehicles, and lanes in the actual environment. Thus, this identification information is used to describe the distribution of the actual environment, thereby reflecting the real-world situation.

[0132] The electronic images in this article can be displayed on the intelligent driving perception screen (ADV) of the HMI (Human-Machine Interface) to achieve human-machine interaction.

[0133] As described above, the electronic image may include, but is not limited to, vehicle identification, vehicle control identification, lane identification, and lane control identification. However, the embodiments of this application are not limited to this. For example, the electronic image may also include other control identification for controlling the vehicle, such as, but not limited to, a speed adjustment identification for indicating speed adjustment.

[0134] The following description uses a control identifier as an example to illustrate the method for generating the corresponding identifier. Specifically, the control identifier can be generated using steps A through C.

[0135] In step A, a vehicle icon is generated.

[0136] Because the vehicle's surroundings are constantly changing while it is in motion, and the sensing devices are mounted on the vehicle body, the location of objects in the surrounding environment can only be determined by using the vehicle itself as a reference. Therefore, it is necessary to first generate a vehicle identifier corresponding to the vehicle that is "stationary" relative to the sensing devices.

[0137] In step B, the relative position of the sensing object and the vehicle is determined based on real-time sensing data.

[0138] By collecting real-time sensing data from sensing devices, the position of objects in the surrounding environment relative to the vehicle can be determined; that is, the relative position of the objects to the vehicle. Within the sensing range of the sensing devices, there may be multiple objects in the surrounding environment, and each object has a relative position to the vehicle. For example, when the first object is in the vehicle's lane, the first object is directly below the vehicle; when the second object is a vehicle in front of the vehicle in the left or right adjacent lane, the second object is located to the left or right front of the vehicle.

[0139] In step C, with the vehicle identifier as a reference, an identifier for the perceived object is generated at the relative position of the vehicle identifier, and a control identifier is generated.

[0140] The vehicle sign is considered as the origin of the reference coordinate system. Using the vehicle sign as a reference, the sign of the perceived object is generated at the corresponding position relative to the vehicle sign, based on the relative position of the perceived object and the vehicle. For example, when the first perceived object (e.g., the vehicle lane) is directly below the vehicle, the sign of the first perceived object (the vehicle lane) is generated directly below the vehicle sign; when the third perceived object (e.g., a vehicle ahead in the vehicle lane) is directly in front of the vehicle, the sign of the second perceived object (the vehicle ahead in the vehicle lane) is generated directly in front of the vehicle sign.

[0141] In step 320, the current instructions input by the user for the electronic image (e.g., the vehicle's identification information and the identification information of the surrounding environment in the electronic image) are received in real time.

[0142] In step 330, based on at least one layer of security judgment mechanism, if it is determined that the current instruction is unreasonable, the next instruction adjacent to the current instruction is received and the next instruction overwrites the current instruction.

[0143] In the embodiments of this application, the next instruction in "overwrite the current instruction with the next instruction" can be a reasonable instruction, or it can be the instruction that is finally determined to be reasonable after a series of unreasonable instructions.

[0144] The at least one layer of security judgment mechanism in the embodiments of this application is used to prioritize security and ensure that all instructions are ultimately reasonable, thereby improving security.

[0145] The safety judgment mechanism in this paper is used to indicate whether the current instruction is reasonable, thus enabling safe intelligent driving of the vehicle.

[0146] "Unreasonable current instruction" indicates that the current instruction will affect driving safety. This unreasonable current instruction can include, but is not limited to, instructions for dangerous or malicious driving. An example of an unreasonable current instruction is one that is frequently issued within a short period. Another example is when the surrounding environment is surrounded by seawater on both sides without roads, but the current instruction is to turn left or right.

[0147] Next, the input method for the aforementioned current command can be via touch input or via voice input, and this application does not impose any restrictions on this.

[0148] In step 340, if the current instruction is determined to be reasonable, the vehicle is allowed to drive according to the current instruction. This enables intelligent driving by controlling the vehicle to follow the current instruction.

[0149] This article not only supports the driver (referred to as the primary driver) in judging whether the instruction is reasonable, but also supports the co-driver in judging whether the current instruction is reasonable. Please see below for details.

[0150] Continuing with Figures 5 and 6, the safety judgment mechanism of the vehicle safety control method may include, but is not limited to, one or more of the following: user safety judgment mechanism, driver safety judgment mechanism, intelligent driving system safety judgment mechanism, and background safety monitoring mechanism. In practical use, one or more of the above safety judgment mechanisms can be used. When using all safety judgment mechanisms, the user safety judgment mechanism, driver safety judgment mechanism, intelligent driving system safety judgment mechanism, and background safety monitoring mechanism can be performed sequentially, which will not be elaborated further here.

[0151] In this regard, for step 330 above, at least one optional embodiment can be used to implement the next instruction overwriting the current instruction.

[0152] In a first optional embodiment, based on the user security judgment mechanism within the security judgment mechanism, if it is determined that the current instruction is unreasonable, the next instruction re-entered by the user is received and overwritten. In this embodiment, the user security judgment mechanism is used to indicate an instruction that has been corrected as unreasonable by the user.

[0153] The next instruction mentioned above may include, but is not limited to, deceleration and / or braking instructions. This can create a relatively safe operating environment for the vehicle.

[0154] The user security assessment mechanism is as follows.

[0155] 1. After a user issues a command, if the intelligent driving system agrees to execute it, the user can cancel the execution of the command at any time during the process using the cancel button on the screen or a voice cancellation command. The user here can include the driver and other people besides the driver. For example, a front passenger issues a command. Another example is a rear-seat user issuing a command.

[0156] 2. After issuing a command, if the user finds the command unreasonable, they can override the previous command with another command, allowing the intelligent driving system to perform a more reasonable operation.

[0157] 3. If a risk is detected after the user issues a command, the user can use a deceleration command to immediately slow down the vehicle to avoid the risk.

[0158] 4. If a risk is detected after the user issues a command, the risk can be avoided by using the braking command.

[0159] In this embodiment, if a user finds that they have entered an incorrect current command, they can re-enter the next command to overwrite the unreasonable or incorrect command, thereby correcting the error command in a timely manner and improving the security and accuracy of the command.

[0160] In this embodiment, the driver and the intelligent driving system need to have the power, authority, and means to effectively disable this function. See below for details.

[0161] In an optional embodiment, in the human-machine co-driving mode, when a user controls the vehicle via screen touch or voice, there may be situations where the operation is not performed by the driver. Such situations, without the driver's consent, may pose safety risks. The vehicle safety control method provided in this application eliminates the risks associated with non-driver human-machine co-driving by allowing the driver to override non-driver operations through the screen and voice.

[0162] Referring again to Figures 5 and 6, in the second optional embodiment, based on the driver safety judgment mechanism in the safety judgment mechanism, if it is determined that the current instruction is unreasonable, the driver will receive the next instruction re-entered by the driver to overwrite the current instruction; wherein, the driver safety judgment mechanism is used to indicate that the driver is assisting the user in correcting unreasonable instructions.

[0163] It should be noted that the driver has the highest authority and can take over or disable touchscreen and voice control at any time. Therefore, even if the intelligent driving system errs in its perception, misses or misjudges the risk of a command, the driver can cancel, terminate, or take over.

[0164] The above-mentioned driver safety judgment mechanism is as follows.

[0165] 1. After the user issues a command, the driver can cancel or terminate the execution of the command by using the cancel command on the screen or by using the voice cancel command.

[0166] 2. After the user issues the command, the driver can directly take over the vehicle to terminate the co-driving command.

[0167] 3. The driver can disable touchscreen and voice control with a single button press using the steering wheel combination. Once disabled, the system can only be unlocked via the steering wheel buttons; the soft switch is ineffective to prevent unauthorized access after unlocking via the soft switch. The driver-vehicle co-driving button has lower priority than the two buttons that disable touchscreen and voice control.

[0168] In an optional embodiment, when using the touch screen or voice-activated human-machine interaction to control intelligent driving functions during autonomous driving, the passenger in the front seat or other people in the vehicle cabin may maliciously interfere (e.g., children, pets). In this embodiment, the risk of malicious interference is eliminated through various methods such as gesture intent inference, central control shortcut key locking, cloud-based identification of malicious interference in the cabin, and driver manual takeover.

[0169] In this embodiment of the application, the driver can assist the user in correcting unreasonable commands, thereby improving the safety and rationality of command input.

[0170] Continuing as shown in Figures 5 and 7, if the current instruction is determined to be unreasonable, the next instruction re-entered by the driver is received and overridden. This includes: if the abnormal frequency of the current instruction is determined to be greater than a first frequency threshold, a disabling instruction re-entered by the driver is received and overridden. The disabling instruction is used to disable the soft switch, and it is entered via a physical button. The disabling instruction also indicates that all software input control permissions are disabled during this power-on cycle.

[0171] It should be noted that the intelligent driving system will analyze the rationality of touch screen and voice control. If multiple or high-frequency unreasonable operations are detected, touch screen and voice control will be temporarily disabled until the owner confirms that they can be released and passes the retest.

[0172] The aforementioned first frequency threshold indicates the permissible range of erroneous commands allowed by the vehicle. This first frequency threshold is set based on user requirements. The higher the first frequency threshold, the more frequently erroneous commands are tolerated.

[0173] The abnormal frequency of the current command exceeding a first frequency threshold indicates that a malicious input command has been received, such as touch or voice input without a clear instruction. In this embodiment, such malicious input commands must be rejected. To avoid further losses caused by malicious input commands, a disable command is input to disable the soft switch function, rendering it ineffective. Thus, even if malicious input commands continue to exist, they are considered invalid, reducing the losses caused by malicious input commands.

[0174] As shown in Figure 7, the soft switch function will be activated using any of the following methods: 1) The human-vehicle co-driving touch interaction method is not disabled; 2) NOA (Navigation On Autopilot) is activated, ADV (Advanced Driver Assistance System) map is opened, and this function will automatically take effect.

[0175] The soft switch function will be deactivated by any of the following methods: (1) The touch interaction mode of human-vehicle co-driving is disabled; (2) ADV map is deactivated; (3) NOA is deactivated; (4) Manual takeover is initiated.

[0176] In this embodiment, the system can fully withstand user misoperation and erratic operation, while also preventing such operations, thereby improving vehicle control safety. Furthermore, the system can reject intentions that may pose risks to the outside world, intentions that may pose risks to the user's own vehicle, and intentions that violate laws or regulations. The intelligent driving system has the final decision-making power; if the user misoperates or operates erratically, the intelligent driving system will reject the operation if it deems it unreasonable, unsafe, or impossible in the current environment.

[0177] As shown in Figure 8, the physical buttons include physical buttons located on both sides of the vehicle's steering wheel, and when the physical buttons on both sides are pressed simultaneously, a disable command is generated.

[0178] As shown in Figure 8, the above vehicle safety control method may also include, but is not limited to, the following two steps: 1. Receiving a disable command from the vehicle's physical buttons (which may be referred to as buttons), the physical buttons being used to disable the soft switch function; 2. The disable command overwrites the current command and responds to the disable command, controlling the disable of the soft switch function on the touch screen, and / or controlling the disable of the touch screen's touch function.

[0179] In addition, after disabling the command that overrides the current command, the above methods can also include, but are not limited to, exiting the function of the aforementioned soft switch, disabling the touch function of the aforementioned touch screen, and switching the touch screen to other screens. Other screens are commonly used screens convenient for the driver. Other screens may include navigation screens, or entertainment / music screens.

[0180] In this embodiment of the application, physical buttons are manually operated to forcibly disable them, thereby reducing interference from other users to the driver and improving the safety of the driver.

[0181] Because vehicles have numerous physical buttons in various locations—for example, buttons on the steering wheel or on the center console—the number of such buttons can be one or more, and there is no limitation on this. A detailed explanation follows.

[0182] Continuing with the physical buttons shown in the box in Figure 8, the physical buttons in this article include the physical buttons located on both sides of the vehicle's steering wheel, and when the physical buttons on both sides are pressed simultaneously, a disable command is generated.

[0183] The physical buttons arranged on opposite sides of the steering wheel can be two or more (excluding two). For example, symmetrical arrangement of two physical buttons is not only aesthetically pleasing, but also makes it faster and more convenient for the driver to operate the steering wheel with both hands on it.

[0184] In this embodiment, physical buttons are manually operated to forcibly disable them, reducing interference from the passenger to the driver. Furthermore, simultaneous operation of multiple physical buttons achieves forced disabling, not only making it convenient for the driver to disable other users' operations but also preventing accidental operation by the driver.

[0185] In some practical application scenarios, after the current driving session ends and the vehicle is powered on again, the process returns to steps 310 to 340 to continue executing the judgment on whether the current instruction is reasonable, etc.

[0186] For the driver, the next instruction mentioned above includes one or more of the following: cancel the current instruction, terminate co-pilot instruction for physical takeover of the driver, and disable instruction for controlling soft switches to be ineffective; the terminate co-pilot instruction has the highest authority.

[0187] In this embodiment of the application, physical takeover may include, but is not limited to, forcibly disengaging from co-driving by having the user operate one or more of the following: the steering wheel, the accelerator pedal, and the brake pedal.

[0188] Referring again to Figures 5 and 6, in the third optional embodiment, the safety judgment mechanism further includes an intelligent driving system safety judgment mechanism. Accordingly, in step 1, the rationality and risk of the current instruction are analyzed according to the intelligent driving system safety judgment mechanism to determine whether the current instruction is reasonable. In step 2, if the current instruction is determined to be unreasonable and exceeds the scope that the intelligent driving system can handle, a request instruction for manual takeover by the driver is generated to override the current instruction. In step 3, if the current instruction is determined to be unreasonable but within the scope that the intelligent driving system can handle, control of the intelligent driving system is implemented. In step 4, if the current instruction is determined to be reasonable, step 340 above is executed.

[0189] In this embodiment of the application, the intelligent driving system safety judgment mechanism is used to indicate the rationality and risk of the intelligent driving system's judgment instruction.

[0190] It's important to note that while the intelligent driving system is granted high-level privileges, touchscreen or voice commands are merely weak requests. Whether and how these commands are executed is determined by the intelligent driving system. The system incorporates commands into its safety assessment, rejecting those that are risky or beyond its capabilities (similar to lane changing via a lever; the system will execute promptly, delay execution, or refuse execution depending on the situation). Therefore, even if someone maliciously manipulates the system and issues dangerous commands, the vehicle will not execute them.

[0191] The safety judgment mechanism of the above-mentioned intelligent driving system is as follows.

[0192] 1. Upon receiving a command from the user, the intelligent driving system analyzes the rationality and risk of the command based on its own safety rules. Commands that pose a risk will be refused or delayed.

[0193] 2. During the execution of user commands, if the intelligent driving system finds that the dynamic environment is no longer suitable for continued execution, it will pause or terminate the execution of the commands and automatically adopt the most appropriate strategy to deal with the situation (i.e., machine management).

[0194] 3. During the execution of user instructions, if the intelligent driving system finds that the dynamic environment is no longer suitable for continued execution, and the emergency situation exceeds the scope of the intelligent driving system's capabilities, the system will promptly request the user to take over.

[0195] 4. After receiving commands from the user, the system will analyze each command. If malicious operations are detected, a warning will be issued. If the situation does not improve, all touchscreen and voice control permissions will be disabled for the remainder of the current power cycle. For example, receiving multiple obviously unreasonable commands in a short period of time, or frequent, irregular multi-finger gestures appearing on the screen in a short period of time.

[0196] Referring again to Figures 5 and 6, in the fourth optional embodiment, the method further includes receiving a deactivation command from the background if the background security monitoring mechanism in the security judgment mechanism detects that the frequency of disabling is greater than a second frequency threshold, and overwriting the current command with the deactivation command, so that the vehicle owner account can apply for relearning and then enable it; wherein, the deactivation command is used to indicate the termination of the intelligent driving function. In this way, the deactivation command can be used to overwrite the current command. In this embodiment, the frequency of disabling refers to the number of times a disabling command is generated within a certain period. As can be seen from the above description, the disabling command can be a disabling command generated in the driver safety judgment mechanism (e.g., a disabling command input by the driver through a physical button based on subjective judgment), or a disabling command generated in other safety judgment mechanisms (e.g., a disabling command generated by the intelligent driving system when malicious operation is detected in the intelligent driving system safety judgment mechanism), and the disabling command is used to disable all touch screen operations (e.g., soft switches on the touch screen, etc.) and voice operations during the vehicle power-on cycle.

[0197] For example, if the intelligent driving system's safety judgment mechanism temporarily disables functions multiple times within a certain period, the backend will deactivate the vehicle's touchscreen and voice controls. This involves receiving a deactivation command from the backend so that the vehicle owner's account can apply for relearning and reactivation. Only after passing the learning and testing process can the functions be restored to normal use.

[0198] As one embodiment, the method further includes: Step 1, after allowing the vehicle to drive according to the current instruction, displaying an interruption icon for interrupting the execution of the current instruction and a continuation icon for continuing the execution of the current instruction in the electronic image, wherein the interruption icon is used to interrupt the execution of the current instruction and the continuation icon is used to continue the execution of the current instruction; Step 2, if an operation is received for the interruption icon, then the intelligent driving of the vehicle according to the current instruction is interrupted; Step 3, if an operation is received for the continuation icon, then the vehicle is controlled to continue intelligent driving according to the current instruction.

[0199] In one example, step A involves displaying an interruption touch icon and / or a continue execution touch icon within the electronic image; the interruption touch icon is used to interrupt the execution of the first instruction; the continue execution touch icon is used to continue the execution of the first instruction. Step B involves interrupting the control of the vehicle to drive intelligently according to the first instruction if the interruption touch icon is touched. Step C involves controlling the vehicle to continue driving intelligently according to the first instruction if the continue execution touch icon is touched.

[0200] In another example, step a involves displaying an interruption indicator and / or a continue execution indicator within an electronic image; the interruption indicator is used to interrupt the execution of the first instruction; the continue execution indicator is used to continue the execution of the first instruction. Step b involves, if a voice command is received for the interruption indicator, the control of the vehicle to intelligently drive according to the first instruction is interrupted. Step c involves, if a voice command is received for the continue execution indicator, the control of the vehicle to continue intelligently driving according to the first instruction is resumed.

[0201] In the embodiments of this application, the execution of instructions can be interrupted in a timely manner, facilitating timely human-computer interaction with the user.

[0202] In this article, the authority of the human-vehicle co-driving command is lower than the authority of the intelligent driving system's judgment, and the authority of the intelligent driving system's judgment is lower than the authority of physical takeover. The driving command sent by the user is merely a wish (a desire for the vehicle to fulfill their wish), not an order or control, and certainly not driving itself; it is simply communication with the intelligent driving system. Human-vehicle co-driving establishes a channel for transmitting the user's intentions to the system, and each intention is ensured to undergo feasibility and risk assessment by the system. Both the system and the driver can reject or cancel the intention midway. How to realize the intention is entirely decided by the system, and the decision-making process is unaffected by the sender.

[0203] Next, it supports reasonable operations by the front passenger (through touch swipes) or other occupants. The front passenger simply sends their intentions without directly operating the vehicle. Previously, the front passenger relayed their intentions to the driver; in the shared driving framework, this is achieved through the intelligent driving system. Both the driver and the intelligent driving system are licensed drivers and fully comply with the existing architecture of "intelligent driving + driver supervision."

[0204] Finally, current intelligent driving systems are a black hole for users. While the systems focus on functionality, they lack interaction with users during use, resulting in issues such as excessive control, lack of personalization, lack of confidence, and lack of security. Co-driving introduces an abstract intermediary layer between driving control and the driver, encapsulating driving behavior and decoupling driving control from the driver, making the vehicle oriented towards the driver's intentions rather than their control. Conversely, co-driving also encapsulates the system's perception and decisions into a reasonable form and presents them to the user through an HMI, thereby building a two-way interaction channel between people and the vehicle, giving users a fully controlled intelligent driving experience.

[0205] This application also provides an electronic image generation and display device. Please refer to FIG9, which is a structural block diagram of the electronic image generation and display device according to an embodiment of this application. The electronic image generation and display device 21 provided in this application embodiment may include: an acquisition module 211, a generation module 212, and a display module 213.

[0206] The acquisition module 211 is configured to acquire real-time perception data obtained by the vehicle's perception devices from perceiving the vehicle's surrounding environment.

[0207] The generation module 212 is configured to generate an electronic image of the surrounding environment based on the real-time sensing data; the electronic image includes the vehicle identifier corresponding to the vehicle, the identifier corresponding to the sensing object in the surrounding environment, and the control identifier corresponding to the real-time control mode of the vehicle relative to the sensing object.

[0208] Display module 213 is configured to display the electronic image on the vehicle's intelligent driving perception screen.

[0209] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, which can achieve the same technical effect, and will not be repeated here.

[0210] In some embodiments, when the generation module 212 generates an electronic image of the surrounding environment based on the real-time sensing data, it is specifically configured to: generate a vehicle identifier corresponding to the vehicle; determine the relative position of the sensing object in the surrounding environment and the vehicle based on the real-time sensing data; generate an identifier corresponding to the sensing object at the relative position of the vehicle identifier with reference to the vehicle identifier, and generate a control identifier corresponding to the real-time control mode of the vehicle relative to the sensing object.

[0211] In some embodiments, when the generation module 212 generates the identifier corresponding to the sensing object at the relative position of the vehicle identifier with reference to the vehicle identifier, it is specifically configured to: determine the number of sensing objects based on the real-time sensing data and number the sensing objects; generate the identifier corresponding to the sensing object at the relative position of the vehicle identifier with reference to the vehicle identifier according to a preset display ratio, and generate their respective numbers on the identifier corresponding to the sensing object itself and / or around it.

[0212] In some embodiments, the sensing object is another vehicle; when the generation module 212 determines the number of sensing objects based on the real-time sensing data and numbers the sensing objects, it is specifically configured to: determine the number of other vehicles based on the real-time sensing data and number the other vehicles using a first type of counting symbol; when the generation module 212 generates an identifier corresponding to the sensing object at the relative position of the self-vehicle identifier according to a preset display ratio and with the self-vehicle identifier as a reference, and generates its own number on and / or around the identifier corresponding to the sensing object, it is specifically configured to: generate an identifier for another vehicle corresponding to the other vehicle at the relative position of the self-vehicle identifier according to a preset display ratio and with the self-vehicle identifier as a reference, and generate its own number on and / or around the identifier for the other vehicle.

[0213] In some embodiments, the sensing object is a lane, which includes the lane where the vehicle is located and other lanes where the vehicle is not located. When the generation module 212 determines the number of sensing objects and numbers them based on the real-time sensing data, it is specifically configured to: determine the number of lanes based on the real-time sensing data and number the lanes using a second type of counting symbol. When the generation module 212 generates an identifier corresponding to the sensing object at the relative position of the vehicle identifier according to a preset display ratio, with the vehicle identifier as a reference, and generates its own number on and / or around the identifier corresponding to the sensing object, it is specifically configured to: generate the identifiers corresponding to the vehicle lane and the other lane at the relative position of the vehicle identifier according to a preset display ratio, with the vehicle identifier as a reference, and generate their own numbers on and / or around the identifiers corresponding to the vehicle lane and the other lane. The identifier corresponding to the vehicle lane is generated below the vehicle identifier, and the identifiers corresponding to the other lanes are generated side by side with the identifier corresponding to the vehicle lane.

[0214] In some embodiments, after generating the corresponding signs for the driving lane and the other lane at the relative position of the driving lane and the other lane according to a preset display ratio and with the driving lane sign as a reference, and generating their respective numbers on the driving lane and the other lane sign themselves and / or around them, the generation module 212 is further configured to: determine whether the distance between one side of the driving lane and the boundary of the corresponding side of the driving lane is greater than a preset safety distance; in response to the distance between one side of the driving lane and the boundary of the corresponding side of the driving lane being greater than the preset safety distance, generate an operable direction indicator sign pointing to the boundary of the corresponding side of the driving lane on the side of the driving lane sign; wherein the operable direction indicator sign is used to indicate that the position of the driving lane can be adjusted in the direction it points to; in response to the distance between one side of the driving lane and the boundary of the corresponding side of the driving lane not being greater than the preset safety distance, generate an inoperable direction indicator sign pointing to the boundary of the corresponding side of the driving lane on the side of the driving lane sign; wherein the inoperable direction indicator sign is used to indicate that the position of the driving lane cannot be adjusted in the direction it points to, and the attributes of the inoperable direction indicator sign are different from those of the operable direction indicator sign.

[0215] In some embodiments, when the generation module 212 generates the identifier corresponding to the perceived object at the relative position of the vehicle identifier, it is specifically configured to: determine whether there is an executable real-time control method for the vehicle relative to the perceived object based on the real-time perception data and a preset intelligent driving strategy; in response to the existence of an executable real-time control method for the vehicle relative to the perceived object, generate an operable identifier corresponding to the perceived object at the relative position of the vehicle identifier; in response to the absence of an executable real-time control method for the vehicle relative to the perceived object, generate an inoperable identifier corresponding to the perceived object at the relative position of the vehicle identifier; wherein the attributes of the operable identifier corresponding to the perceived object are different from the attributes of the inoperable identifier corresponding to the perceived object.

[0216] In some embodiments, the electronic image generation and display device 21 may further include a receiving module and a control module.

[0217] The receiving module is configured to receive voice commands sent by the user for the control icon in real time after the display module 213 displays the electronic image on the intelligent driving perception screen of the vehicle.

[0218] The control module is configured to control the vehicle's movement based on the voice command.

[0219] In some embodiments, when the control module controls the vehicle to drive according to the voice command, it is specifically configured to: determine whether the control method in the voice command matches the control method indicated by the control identifier, and obtain a determination result; and control whether the vehicle drives according to the voice command based on the determination result and the real-time perception data.

[0220] In some embodiments, the acquisition module 211 is further configured to acquire the identity of the current user after the control module controls the vehicle to drive according to the voice command, and to acquire the driving habits of the current user from the pre-recorded driving habits of the user based on the identity of the current user; the control module is further configured to control the vehicle to drive according to the driving habits of the current user after the acquisition module 211 acquires the driving habits of the current user.

[0221] In some embodiments, the electronic image generation and display device 21 may further include a recording module.

[0222] The recording module is configured to collect and record the real-time perception data and voice commands during each driving process, and analyze the timbre of the voice commands to identify users who send voice commands with different timbres as different users; the generation module 212 is further configured to generate the driving habits of different users based on the real-time perception data and voice commands with different timbres during each driving process after the recording module identifies users who send voice commands with different timbres as different users.

[0223] Furthermore, based on the same inventive concept as the vehicle safety control method described above with reference to Figures 5 to 8, the electronic image generation and display device of the present application embodiment can also be further used to implement the vehicle safety control method described above.

[0224] In some embodiments, as described above, in the electronic image generation and display device, the acquisition module 211 can be configured to acquire electronic images of the intelligent driving perception screen in real time, and the receiving module can be configured to receive current instructions input by the user for the electronic image in real time.

[0225] In addition, the electronic image generation and display device may also include an instruction update module, which is configured to receive the next instruction adjacent to the current instruction and overwrite the current instruction if the current instruction is determined to be unreasonable according to at least one layer of safety judgment mechanism; the safety judgment mechanism is used to indicate that the vehicle is driving safely after determining that the current instruction is reasonable.

[0226] Furthermore, the control module of the electronic image generation and display device can also be configured to allow the vehicle to drive according to the current instruction if it is determined that the current instruction is reasonable.

[0227] As one embodiment, the display module of the aforementioned electronic image generation and display device can also be configured to display, within the electronic image, an interruption icon indicating the interruption of the current instruction and a continuation icon indicating the continuation of the current instruction, after allowing the vehicle to drive according to the current instruction. The interruption icon is used to interrupt the execution of the current instruction, and the continuation icon is used to continue the execution of the current instruction. The electronic image generation and display device also includes an interruption control module, configured to interrupt intelligent driving of the vehicle according to the current instruction if an operation is received for the interruption icon. Furthermore, the control module is also configured to control the vehicle to continue intelligent driving according to the current instruction if an operation is received for the continuation icon.

[0228] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, which can achieve the same technical effect, and will not be repeated here.

[0229] This application also provides an electronic device that may include the electronic image generation and display device 21 described above. Please refer to FIG10, which is a structural block diagram of the electronic device according to an embodiment of this application. The electronic device 20 may include one or more processors 22, which are configured to implement the electronic image generation and display method described above.

[0230] In some embodiments, the electronic device 20 may further include one or more computer-readable storage media 23, which may store a program that can be invoked by the processor 22, and may include non-volatile storage media. In other embodiments, the electronic device 20 may further include memory 24 and an interface 25. In still other embodiments, the electronic device 20 may also include other hardware depending on the specific application.

[0231] The computer-readable storage medium 23 provided in this application embodiment stores a program thereon, which, when executed by the processor 22, is used to implement the above-described method for generating and displaying electronic images.

[0232] This application may take the form of a computer program product implemented on one or more computer-readable storage media 23 (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. The computer-readable storage media 23 includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented using any method or technology. This information may be computer-readable instructions, data structures, program modules, or other data. The computer-readable storage media 23 includes, but is not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium, which can be used to store information accessible by a computing device.

[0233] Of course, in some embodiments of this application, the electronic device may be one or more of a server device and a PC (Personal Computer) device. The server device and the PC device may include, but are not limited to, a server, a desktop computer, a tablet computer, or a laptop computer.

[0234] In some embodiments, this electronic device may include, but is not limited to, an in-vehicle terminal connected to the vehicle and a mobile terminal independent of the vehicle. The in-vehicle terminal connected to the vehicle may be, but is not limited to, a body processor, controller, center console, or vehicle HUD (Head-Up Display). The mobile terminal independent of the vehicle may include, but is not limited to, a smartphone, smartwatch, tablet, or laptop.

[0235] This application also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by at least one processor, it is able to perform the electronic image generation and display methods described in the above embodiments.

[0236] This application also provides a vehicle, including an electronic image generation and display device as described in the above embodiments, or an electronic device as described in the above embodiments, or a computer-readable storage medium as described in the above embodiments, or a computer program product as described in the above embodiments.

[0237] For example, the vehicle may be one or more of heavy trucks, light commercial vehicles, and passenger cars. Thus, the method of this application embodiment can be applied to various application scenarios such as heavy trucks, light commercial vehicles, and passenger cars, thereby improving the versatility of the method.

[0238] The electronic image generation and display method, apparatus, electronic device, program product, storage medium, and vehicle provided in this application generate electronic images of the vehicle's surrounding environment based on real-time perception data obtained from the vehicle's sensing devices. These electronic images can intuitively display the vehicle's corresponding identification and the identification of perceived objects in the surrounding environment, allowing users to quickly and accurately grasp information about the vehicle's surroundings by viewing the electronic images. Furthermore, since the electronic images also contain control identifiers corresponding to the real-time control methods of the vehicle relative to perceived objects in the surrounding environment, users can determine which real-time control methods can be executed relative to which perceived objects in the surrounding environment simply by viewing the electronic images, eliminating the need for analysis and judgment based on driving experience and reducing the safety risks caused by user distraction.

[0239] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware (e.g., a processor), and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware, such as by using an integrated circuit to implement its corresponding function, or it can be implemented in the form of a software functional module, such as by a processor executing a program / instruction stored in memory to implement its corresponding function. This application is not limited to any particular combination of hardware and software.

[0240] The above description is merely an exemplary embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification shall be included within the scope of protection of this specification.

[0241] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element qualified by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

A method for generating and displaying an electronic image, comprising: Acquire real-time perception data obtained by the vehicle's perception devices from perceiving the vehicle's surrounding environment. Based on the real-time sensing data, an electronic image of the surrounding environment is generated; the electronic image includes the vehicle identifier corresponding to the vehicle, the identifier corresponding to the sensing object in the surrounding environment, and the control identifier corresponding to the real-time control mode of the vehicle relative to the sensing object. The electronic image is displayed on the vehicle's intelligent driving perception screen. The electronic image generation display method according to claim 1, wherein The step of generating an electronic image of the surrounding environment based on the real-time sensed data includes: Generate the vehicle identifier corresponding to the vehicle; Based on the real-time sensing data, the relative position of the sensing object in the surrounding environment and the vehicle is determined; Using the vehicle identifier as a reference, an identifier corresponding to the sensing object is generated at the relative position of the vehicle identifier, and a control identifier corresponding to the real-time control mode of the vehicle relative to the sensing object is generated. The electronic image generation display method according to claim 2, wherein The step of generating an identifier corresponding to the perceived object at the relative position of the vehicle identifier, with reference to the vehicle identifier, includes: Based on the real-time sensing data, determine the number of sensing objects and assign numbers to the sensing objects; According to the preset display ratio, with the vehicle logo as a reference, the logo corresponding to the sensing object is generated at the relative position of the vehicle logo, and a number is generated on the logo corresponding to the sensing object itself and / or around it. The electronic image generation display method according to claim 3, wherein The sensing objects include other vehicles; The step of determining the number of sensing objects based on the real-time sensing data and numbering the sensing objects includes: Based on the real-time sensing data, the number of other vehicles is determined, and the other vehicles are numbered using the first type of counting symbol; The step of generating the identifier corresponding to the sensing object at the relative position of the vehicle identifier according to a preset display ratio, with the vehicle identifier as a reference, and generating a number for the identifier corresponding to the sensing object itself and / or around it, includes: According to the preset display ratio, with the vehicle identifier as a reference, a vehicle identifier corresponding to the other vehicle is generated at the relative position of the vehicle identifier, and a number is generated on the vehicle identifier itself and / or around it. The electronic image generation display method according to claim 3, wherein The sensing object includes lanes, and the lanes include the lane where the vehicle is located and other lanes where the vehicle is not located. The step of determining the number of sensing objects based on the real-time sensing data and numbering the sensing objects includes: Based on the real-time sensing data, the number of lanes is determined, and the lanes are numbered using a second type of counting symbol; The step of generating the identifier corresponding to the sensing object at the relative position of the vehicle identifier according to a preset display ratio, with the vehicle identifier as a reference, and generating a number for the identifier corresponding to the sensing object itself and / or around it, includes: According to the preset display ratio, with the vehicle identification as a reference, the identification corresponding to the vehicle lane and the other lane is generated at the relative position of the vehicle identification, and the identification corresponding to the vehicle lane and the other lane is generated with their respective numbers on themselves and / or around the identification. The sign corresponding to the self-lane is generated below the self-lane sign, and the sign corresponding to the other lane is generated side by side with the sign corresponding to the self-lane. The electronic image generation display method according to claim 5, wherein After generating respective numbers for the identifiers corresponding to the respective lanes and other lanes, the method for generating and displaying the electronic image further includes: Determine whether the distance between one side of the vehicle and the boundary of the corresponding side of the lane is greater than a preset safety distance; In response to the distance between the boundary of one side of the vehicle and the corresponding side of the lane being greater than a preset safety distance, an operable direction indicator pointing to the boundary of the corresponding side of the lane is generated on the side of the vehicle identifier. In response to the fact that the distance between the boundary of the side of the vehicle and the corresponding side of the lane is not greater than a preset safety distance, an inoperable directional indicator pointing to the boundary of the corresponding side of the lane is generated on the side of the vehicle identifier. The operable direction indicator has different attributes from the non-operable direction indicator. The operable direction indicator is configured to indicate that the position of the vehicle can be adjusted in the direction it points to, while the non-operable direction indicator is configured to indicate that the position of the vehicle cannot be adjusted in the direction it points to. The electronic image generation display method according to any one of claims 2 to 6, wherein Generating the identifier corresponding to the perceived object at the relative position of the vehicle identifier includes: Based on the real-time perception data and the preset intelligent driving strategy, determine whether there is an executable real-time control method for the vehicle relative to the perceived object; In response to the existence of an executable real-time control mode for the vehicle relative to the sensing object, an operable identifier corresponding to the sensing object is generated at the relative position of the vehicle identifier; In response to the absence of an executable real-time control method for the vehicle relative to the sensing object, an inoperable identifier corresponding to the sensing object is generated at the relative position of the vehicle identifier. The attributes of the operable identifier corresponding to the sensing object are different from the attributes of the inoperable identifier corresponding to the sensing object. The electronic image generation display method according to any one of claims 1 to 7, wherein After displaying the electronic image on the intelligent driving perception screen of the vehicle, the method for generating and displaying the electronic image further includes: It can receive voice commands input by the user in real time for the control identifier; The vehicle is controlled to move according to the voice commands. The electronic image generation display method according to claim 8, wherein The step of controlling the vehicle's movement according to the voice command includes: Determine whether the control method in the voice command matches the control method indicated by the control identifier, and obtain the determination result; Based on the judgment result and the real-time perception data, control whether the vehicle drives according to the voice command. The electronic image generation display method according to claim 8, wherein After controlling the vehicle to move according to the voice command, the method for generating and displaying the electronic image further includes: Obtain the user's identity; Based on the user's identity, the user's driving habits are obtained from pre-recorded user driving habits; Control the vehicle to drive according to the user's driving habits. The electronic image generation display method according to claim 10, wherein The pre-recorded user driving habits are generated through the following steps: Collect and record real-time perception data and voice commands during each driving process, analyze the timbre of the voice commands, and identify users who input voice commands with different timbres as different users; Based on the real-time perception data and the voice commands with different timbres during each driving process, different users' driving habits are generated. The electronic image generation display method according to claim 8, wherein After receiving the user's voice command input for the control identifier in real time, the method for generating and displaying the electronic image further includes: According to at least one layer of safety judgment mechanism, if it is determined that the voice command is unreasonable, the next command adjacent to the voice command is received, and the next command overwrites the voice command; the at least one layer of safety judgment mechanism is configured to indicate that the vehicle drives safely after determining that the voice command is reasonable; and, If the voice command is deemed reasonable, the vehicle may be controlled to drive in accordance with the voice command. The electronic image generation display method according to claim 12, wherein The at least one layer of security judgment mechanism includes a user security judgment mechanism. The step of receiving the next instruction adjacent to the voice instruction and overwriting the voice instruction with the next instruction, based on the at least one layer of security judgment mechanism, includes: Based on the user security judgment mechanism, if it is determined that the voice command is unreasonable, the system receives the next command re-inputted by the user and overwrites the voice command. The user security judgment mechanism is configured to indicate an instruction that the user has corrected an unreasonable situation. The electronic image generation display method according to claim 12 or 13, wherein The at least one layer of safety judgment mechanism further includes a driver safety judgment mechanism. The step of receiving the next instruction adjacent to the voice instruction and overwriting the voice instruction with the next instruction, based on the at least one layer of safety judgment mechanism, includes: Based on the driver safety judgment mechanism, if it is determined that the voice command is unreasonable, the driver will re-enter the next command and overwrite the original voice command. The driver safety judgment mechanism is configured to indicate that the driver assists the user in correcting unreasonable instructions. The electronic image generation display method according to claim 14, wherein The step of receiving the next command re-entered by the driver when it is determined that the voice command is unreasonable, and overwriting the voice command, includes: If it is determined that the abnormal frequency of the voice command is greater than a first frequency threshold, the driver re-enters a disable command to override the voice command. The disable command is configured to disable the soft switch, and the disable command is input via physical button operation. The electronic image generation display method according to any one of claims 12 to 14, wherein The at least one layer of safety judgment mechanism also includes an intelligent driving system safety judgment mechanism. According to the at least one layer of safety judgment mechanism, if it is determined that the voice command is unreasonable, receiving the next command adjacent to the voice command and overwriting the voice command with the next command includes: Based on the safety judgment mechanism of the intelligent driving system, the rationality and risks of the voice command are analyzed to determine whether the voice command is reasonable. If it is determined that the voice command is unreasonable and exceeds the processing range of the intelligent driving system, a request command for the driver to take over is generated to override the voice command. The intelligent driving system's safety judgment mechanism is configured to indicate the rationality and risk of the intelligent driving system's judgment commands, and / or The at least one layer of security judgment mechanism also includes a background security monitoring mechanism. The step of receiving the next instruction adjacent to the voice instruction and overwriting the voice instruction with the next instruction, based on the at least one layer of security judgment mechanism, includes: If the frequency of disabling is detected to be greater than the second frequency threshold according to the background security monitoring mechanism, a deactivation command is received from the background and the deactivation command is overwritten to enable the vehicle owner account to apply for relearning and then be reactivated. The deactivation command is configured to indicate the termination of the intelligent driving function. An electronic image generation and display device, comprising: The acquisition module is configured to acquire real-time perception data obtained by the vehicle's perception devices from perceiving the vehicle's surrounding environment. The generation module is configured to generate an electronic image of the surrounding environment based on the real-time sensing data; the electronic image includes the vehicle identifier corresponding to the vehicle, the identifier corresponding to the sensing object in the surrounding environment, and the control identifier corresponding to the real-time control mode of the vehicle relative to the sensing object. The display module is configured to display the electronic image on the vehicle's intelligent driving perception screen. An electronic device includes one or more processors configured to implement a method for generating and displaying an electronic image as described in any one of claims 1 to 16. A non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by at least one processor, implement the method for generating and displaying an electronic image as described in any one of claims 1 to 16. A vehicle comprising: The electronic image generation and display device as described in claim 17, or The electronic device as described in claim 18, or The non-transitory computer-readable storage medium as described in claim 19, or One or more processors configured to implement the method for generating and displaying an electronic image as described in any one of claims 1 to 16.

Images