Information processing method, model training method, and related device

Abstract

An information processing method, a model training method, and a related device, applied to the field of intelligent driving. The information processing method comprises: acquiring first environmental information around a vehicle, and then obtaining category information on the basis of the first environmental information, wherein the category information comprises the category of an object in a low-speed state; the object in the low-speed state is an object having a driving speed less than or equal to a first speed; the category indicates whether the object can be bypassed; the category corresponds to the low-speed state of the object, or the category corresponds to a cause that leads the object to be in the low-speed state, and is provided to a downstream module as the category of the object in the low-speed state; the category indicates whether the object in the low-speed state can be bypassed, and then the downstream module can directly determine, by means of the described category information, whether the object in the low-speed state can be bypassed.

Description

An information processing method, a model training method, and related equipment. Technical Field

[0001] This application relates to intelligent driving technology, and more particularly to a training method for an information processing model and related equipment. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Intelligent driving is a common application area of ​​artificial intelligence technology. For example, sensors can acquire environmental information around the vehicle and perceive the situation of surrounding vehicles based on the aforementioned environmental information. Specifically, the vehicle can detect multiple dimensions of attributes such as whether the brake lights, malfunction lights, turn signals, and doors of surrounding vehicles are on based on the aforementioned environmental information. The detected attributes can be passed to downstream modules, which then generate the vehicle's driving strategy based on the aforementioned multiple dimensions of attributes and predefined judgment rules. The vehicle's driving strategy includes whether to detour around the vehicle or not.

[0004] However, the above solution not only requires accurate acquisition of attributes in each dimension, but also requires processing of attributes in multiple dimensions, resulting in high complexity. A simpler solution is urgently needed. Summary of the Invention

[0005] This application provides an information processing method, a model training method, and related equipment for providing downstream modules with the categories of objects in a low-speed state. The aforementioned categories indicate whether objects in a low-speed state can be bypassed. The downstream modules can directly determine whether objects in a low-speed state can be bypassed by using the aforementioned category information.

[0006] This application provides the following technical solution:

[0007] Firstly, this application provides an information processing method applicable to the field of intelligent driving. In this method, a first vehicle acquires first environmental information surrounding the vehicle, and then obtains category information based on the first environmental information. The category information includes the category of each of at least one object in a low-speed state in the traffic environment surrounding the first vehicle at the current moment. The category of each object in a low-speed state indicates whether the object can be bypassed. For ease of description, any one of the at least one object in a low-speed state will be referred to as the first object. For example, when the category of the first object is a first category, it indicates that the first object can be bypassed; when the category of the first object is a second category, it indicates that the first object cannot be bypassed.

[0008] For example, the first environmental information can be the environmental information at the current moment, that is, the first environmental information can include the traffic environment around the vehicle (hereinafter referred to as "the first vehicle" for ease of distinction) at the current moment. The first environmental information can include at least one image and / or point cloud data corresponding to the traffic environment at the current moment. The first vehicle in this application can also be referred to as a self-driving vehicle. "The traffic environment around the first vehicle" can be understood as the environment within the field of view of the sensors deployed on the first vehicle.

[0009] An object in a low-speed state is an object whose speed is less than or equal to the first speed. For example, an object in a low-speed state includes an object moving at a low speed and an object that is stationary.

[0010] Each object in a low-speed state is categorized as either the object's low-speed state category or the category of the cause that led to the object being in a low-speed state.

[0011] In this implementation, after obtaining the first environmental information around the vehicle, category information can be obtained based on the first environmental information. This category information includes the category of objects in a low-speed state. Objects in a low-speed state are those whose driving speed is less than or equal to the first speed. The aforementioned category indicates whether objects in a low-speed state can be bypassed. Here, the category is the category of the object's low-speed state, or the category is the category of the reason that causes the object to be in a low-speed state. That is, what this application can provide to the downstream module is the category of objects in a low-speed state. The aforementioned category indicates whether objects in a low-speed state can be bypassed. Then, the downstream module can directly determine whether the aforementioned objects in a low-speed state can be bypassed by using the aforementioned category information. Thus, the downstream module no longer needs to process attributes of multiple dimensions, providing a simpler processing solution.

[0012] In one possible implementation, if the category of each object in a low-speed state is the category of the object's low-speed state, then the category of each object in a low-speed state can represent whether the object's low-speed state is long-term or short-term. For example, a first category can represent that the object's low-speed state is long-term; in other words, when the category of the first object represents that the object's low-speed state is long-term, it indicates that the first object can be bypassed. A second category represents that the object's low-speed state is short-term; in other words, when the category of the first object represents that the object's low-speed state is short-term, it indicates that the first object does not need to be bypassed.

[0013] Alternatively, if the category of each object in a low-speed state is the category of the cause of the object being in a low-speed state, the category of each object in a low-speed state can represent whether the object's low-speed state is caused by the object itself or by the environment. For example, a first category can represent that the object's low-speed state is caused by the object itself; in other words, when the category of a first object represents that the object's low-speed state is caused by the object itself, it indicates that the first object can be bypassed. A second category represents that the object's low-speed state is caused by the environment; in other words, when the category of a first object represents that the object's low-speed state is caused by the environment, it indicates that the first object cannot be bypassed.

[0014] In this implementation, the specific categories of objects in a low-speed state are further refined, which further improves the feasibility of this solution. If the object's low-speed state is long-term, the vehicle cannot wait behind an object in a long-term low-speed state indefinitely, so it will often choose to detour around the object. If the object's low-speed state is short-term, meaning the vehicle can pass while following the aforementioned object, it can choose not to detour around the object. If the cause of the object's low-speed state is environmental, the vehicle is often unable to detour around the object due to environmental limitations, so it can choose not to detour around the object. If the cause of the object's low-speed state is caused by the object itself, the vehicle can often detour around the object in the environment, so it can choose to detour around the object. Thus, the category of objects in a low-speed state not only indicates whether or not to detour around the object, but also provides the reason for detouring, enhancing the interpretability of the intelligent driving process.

[0015] In one possible implementation, the first environmental information is the environmental information at the current moment, and the first vehicle obtains category information based on the first environmental information, including: the first vehicle can generate a first feature corresponding to the current moment through a machine learning model based on the first environmental information, and then generate category information through a machine learning model based on the first feature and the second feature corresponding to each of the N historical moments, where N is an integer greater than or equal to 1, and the second feature corresponding to each historical moment is obtained based on the second environmental information of that historical moment.

[0016] In this implementation, after obtaining the first feature corresponding to the current moment based on the first environmental information at the current moment, the first machine learning model generates category information based on the first feature and the second feature corresponding to the historical moment. That is, it generates the category of objects in a low-speed state in the current moment's environment. The second feature is obtained based on the second environmental information at the historical moment. In other words, it uses environmental information from multiple moments to generate the category of objects in a low-speed state in the current moment's environment. This not only helps improve the accuracy of the category information corresponding to a single moment obtained by the first machine learning model, but also helps improve the continuity and stability between the category information obtained by the first machine learning model at multiple different moments. In addition, directly using the second feature at the historical moment, instead of using the second environmental information at the historical moment, avoids repetitive feature extraction of the second environmental information at the historical moment, which helps to reduce the computational resources consumed by the first machine learning model in the process of generating category information.

[0017] In one possible implementation, the category information includes the category of each object among at least one object in a low-speed state in the traffic environment surrounding the first vehicle at the current moment. The first vehicle generates category information using a machine learning model based on a first feature and a second feature corresponding to each of N historical moments. This includes: the first vehicle obtaining a third feature for each object among at least one object in a low-speed state in the traffic environment surrounding the first vehicle at the current moment, based on the first feature and the second feature corresponding to each of the N historical moments. The third feature includes the features of the same low-speed object at the current moment and at historical moments. Furthermore, based on the third feature of each object among at least one object in a low-speed state in the traffic environment surrounding the first vehicle at the current moment, the category of each object among at least one object in a low-speed state in the traffic environment surrounding the first vehicle at the current moment can be generated using a machine learning model.

[0018] In this implementation, based on the first feature and the second feature corresponding to each of the N historical features, the features of the object in the current low-speed state at the current time and at least one historical time are obtained. Then, based on the features of the object in the low-speed state at multiple times, the category of the object in the low-speed state is determined, which is beneficial to obtaining a more accurate category of the object in the low-speed state.

[0019] In one possible implementation, the first vehicle generates a first feature corresponding to the current moment based on the first environmental information through a machine learning model. This includes: the first vehicle inputs the first environmental information and the second feature corresponding to each of the M historical moments into the machine learning model, and extracts features from the first environmental information and the second feature corresponding to each of the M historical moments (i.e., the M second features) through the machine learning model to obtain the first feature. M is an integer greater than or equal to 1, and the values ​​of M and N can be the same or different.

[0020] Optionally, the feature extraction module of the machine learning model can be the feature extraction module of an object detection model. An object detection model refers to a machine learning model used to detect objects in the current traffic environment. In other words, an object detection model can be understood as a machine learning model that detects objects moving at low speeds in the current traffic environment. For example, the feature extraction module of this machine learning model may include: a convolutional neural network layer + a region proposal network (RPN) layer.

[0021] In this implementation, a first machine learning model is used to extract features from the first environmental information and the second features corresponding to historical moments to obtain the first feature corresponding to the current moment. Thus, the first feature corresponding to the current moment not only includes the features extracted from the first environmental information of the current moment, but also includes the features extracted from the second features corresponding to historical moments. Therefore, when generating the category information corresponding to the current moment using the first feature, it is possible to understand not only the environmental information of the current moment, but also the environmental information of historical moments, which is beneficial to further improve the accuracy of the category information corresponding to a single moment obtained by the first machine learning model.

[0022] In one possible implementation, the category information is used to obtain the vehicle's planning and control information, which includes at least one of the following: the vehicle's planned location, the vehicle's driving strategy, or the vehicle's control information. In this implementation, the category information can also be used to obtain the planning and control information of a first vehicle, which includes at least one of the following: the vehicle's planned location, the vehicle's driving strategy, or the vehicle's control information. This provides multiple application scenarios for the category information, which is beneficial for expanding the application scenarios of this application.

[0023] Secondly, this application provides a model training method that can be applied to the field of intelligent driving. In this method, the training device acquires first environmental information around the vehicle. Based on the first environmental information, a machine learning model is used to obtain predicted category information, which includes the predicted category of an object in a low-speed state. An object in a low-speed state is an object whose driving speed is less than or equal to a first speed. A loss function is used to train the machine learning model. The loss function indicates the similarity between the predicted category information and the expected category information. The expected category information includes the correct category of the object in a low-speed state. Both the predicted category and the correct category indicate whether the object can be bypassed. In this case, the predicted category and the correct category are both categories of the object in a low-speed state, or the predicted category and the correct category are categories of the reasons that cause the object to be in a low-speed state.

[0024] In the second aspect of this application, the training equipment is also used to perform the steps performed by the first vehicle in the first aspect and various possible implementations of the first aspect. The specific implementations of the steps in the second aspect, the meanings of the terms, and the beneficial effects are all described in the first aspect and will not be repeated here.

[0025] Thirdly, this application provides an information processing device that can be used in the field of artificial intelligence. The device includes: an acquisition module for acquiring first environmental information around a vehicle; and a processing module for obtaining category information based on the first environmental information. The category information includes the category of an object in a low-speed state, wherein the object in a low-speed state is an object whose driving speed is less than or equal to a first speed, and the category indicates whether the object can be bypassed. The category is either the category of the object's low-speed state or the category of the cause that leads to the object being in a low-speed state.

[0026] In the third aspect of this application, the information processing device is also used to perform the steps performed by the first vehicle in the first aspect and various possible implementations of the first aspect. The specific implementations of the steps in the third aspect, the meanings of the terms, and the beneficial effects are all described in the first aspect and will not be repeated here.

[0027] Fourthly, this application provides a model training device that can be used in the field of artificial intelligence. The device includes: an acquisition module for acquiring first environmental information around a vehicle; a processing module for obtaining predicted category information based on the first environmental information using a machine learning model, wherein the predicted category information includes the predicted category of an object in a low-speed state, and the object in a low-speed state is an object whose driving speed is less than or equal to a first speed; and a training module for training the machine learning model using a loss function, wherein the loss function indicates the similarity between the predicted category information and the expected category information, and the expected category information includes the correct category of the object in a low-speed state, wherein both the predicted category and the correct category indicate whether the object can be bypassed, wherein the predicted category and the correct category are both categories of the object in a low-speed state, or the predicted category and the correct category are categories of the reasons that cause the object to be in a low-speed state.

[0028] In the fourth aspect of this application, the training device for the model is also used to execute the steps performed by the first vehicle in the first aspect and various possible implementations of the first aspect. The specific implementation methods of the steps in the fourth aspect, the meanings of the terms, and the beneficial effects they bring can all be found in the first aspect, and will not be repeated here.

[0029] Fifthly, this application provides an apparatus including a processor and a memory, the processor being coupled to the memory, the memory storing program instructions, which, when executed by the processor, implement the methods described in the first or second aspect.

[0030] In a sixth aspect, this application provides a vehicle including a processor and a memory, the processor being coupled to the memory, the memory storing program instructions, which, when executed by the processor, implement the methods of the first or second aspect described above.

[0031] In a seventh aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect.

[0032] Eighthly, this application provides a computer program product comprising a program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect.

[0033] Ninthly, this application provides a chip system including a processor for supporting the implementation of the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the terminal device or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0034] The second to ninth aspects of this application correspond to the first aspect or multiple possible ways of the first aspect, and have corresponding beneficial effects. Attached Figure Description

[0035] Figure 1 is a structural diagram of an artificial intelligence main framework provided in this application;

[0036] Figure 2 is a system architecture diagram of an information processing system provided in an embodiment of this application;

[0037] Figure 3 is a flowchart illustrating an information processing method provided in an embodiment of this application.

[0038] Figure 4 is a schematic diagram of various scenarios of a vehicle in a low-speed state provided in the embodiments of this application;

[0039] Figure 5 is a schematic diagram of an identifier for a sub-feature provided in an embodiment of this application;

[0040] Figure 6 is a schematic diagram of obtaining the third feature of each object in at least one object in a low-speed state in the traffic environment at the current moment based on the first feature and N second features, according to an embodiment of this application.

[0041] Figure 7 is another schematic diagram of the information processing method provided in the embodiments of this application;

[0042] Figure 8 is a flowchart illustrating a training method for a model provided in an embodiment of this application;

[0043] Figure 9 is a schematic diagram of an information processing device provided in an embodiment of this application;

[0044] Figure 10 is a schematic diagram of a training device for a model provided in an embodiment of this application;

[0045] Figure 11 is a schematic diagram of a device provided in an embodiment of this application;

[0046] Figure 12 is a structural schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0047] The embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0048] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0049] In the embodiments of this application, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information (hereinafter referred to as instruction information) is called the information to be instructed. In specific implementation, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly indicate the information to be instructed by indicating other information, where there is an association between the other information and the information to be instructed; or it can indicate only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction can be implemented by using a pre-agreed (e.g., protocol predefined) arrangement of various information, thereby reducing the instruction overhead to a certain extent. This application does not limit the specific method of instruction. It is understood that for the sender of the instruction information, the instruction information can be used to indicate the information to be instructed; for the receiver of the instruction information, the instruction information can be used to determine the information to be instructed.

[0050] First, the overall workflow of the artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of one aspect of the artificial intelligence framework provided in this application. The framework is then elaborated on from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0051] (1) Infrastructure

[0052] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically employ hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0053] (2) Data

[0054] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0055] (3) Data processing

[0056] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0057] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0058] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0059] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0060] (4) General ability

[0061] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0062] (5) Smart Products and Industry Applications

[0063] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, intelligent driving, and smart cities.

[0064] The method provided in this application can be applied to the field of intelligent driving. In related technologies, sensors are often used to acquire environmental information around a vehicle (hereinafter referred to as the "first vehicle" for ease of distinction). Based on the aforementioned environmental information, the situation of surrounding vehicles (hereinafter referred to as the "second vehicle" for ease of distinction) is perceived. For example, based on the aforementioned environmental information, multiple specific attributes of the second vehicle can be detected, such as whether the brake lights are on, whether the malfunction indicator lights are on, the properties of the turn signals, and whether the doors are open. The detected attributes can be passed to downstream modules, which generate a driving strategy for the vehicle based on the aforementioned multiple specific attributes and predefined judgment rules. The driving strategy of the vehicle includes whether to detour around the second vehicle or not. However, the aforementioned solutions not only require accurate acquisition of attributes in each dimension but also require processing of attributes in multiple dimensions, resulting in high complexity for this solution.

[0065] To address the aforementioned issues, this application discloses that after obtaining the first environmental information surrounding the vehicle, category information can be obtained based on this first environmental information. This category information includes the category of objects in a low-speed state. Objects in a low-speed state are those with a driving speed less than or equal to a first speed. The aforementioned category indicates whether objects in a low-speed state can be bypassed. Here, the category is either the category of the object's low-speed state or the category of the cause that leads to the object being in a low-speed state. In other words, this application can provide downstream modules with the category of objects in a low-speed state. The aforementioned category indicates whether objects in a low-speed state can be bypassed. Therefore, downstream modules can directly determine whether objects in a low-speed state can be bypassed using the aforementioned category information. This eliminates the need for downstream modules to process attributes across multiple dimensions, providing a simpler processing solution.

[0066] Before detailing the method provided in this application, the architecture of the information processing system provided in this application will be described first. Please refer to Figure 2, which is a system architecture diagram of the information processing system provided in an embodiment of this application. In Figure 2, the information processing system 200 includes a training device 210, a database 220, an execution device 230, and a data storage system 240. The execution device 230 includes a computing module 231. The database 220 stores a training dataset. During the training phase of the first machine learning model 201, the training device 210 can use the training dataset to perform training operations to obtain the trained first machine learning model 201.

[0067] In the application phase of the first machine learning model 201, the inference process of the first machine learning model 201 can be executed by the computing module 231 of the execution device 230. Optionally, as shown in Figure 2, the execution device 230 can be integrated into the vehicle, allowing users to directly interact with the vehicle where the execution device 230 is deployed. For example, the execution device 230 can be a module in the vehicle's host CPU that uses the machine learning model for data processing. The execution device 230 can also be a graphics processing unit (GPU), neural network processing unit (NPU), or tensor processing unit (TPU) in the vehicle, etc. The aforementioned GPU, NPU, or TPU is mounted on the vehicle's host CPU as a coprocessor, and the vehicle's host CPU allocates tasks, etc.

[0068] The execution device 230 can access data, code, etc., in the data storage system 240, and can also store data, instructions, etc., in the data storage system 240. The data storage system 240 can be located within the execution device 230, or it can be an external memory relative to the execution device 230.

[0069] It should be noted that Figure 2 is merely a schematic diagram of one architecture of the information processing system provided in the embodiments of this application, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in some other embodiments of this application, the execution device 230 and the vehicle can be separate and independent devices. The execution device 230 is configured with an input / output (I / O) interface, through which the execution device 230 can interact with the vehicle for data. For example, in the application phase, the intelligent driving system in the vehicle can send environmental information to the execution device 230 through the I / O interface. After obtaining category information through the trained first machine learning model 201 deployed in the computing module 231, the execution device 230 can send the aforementioned category information to the intelligent driving system in the vehicle through the I / O interface.

[0070] For example, in some other embodiments of this application, the training device 210 and the execution device 230 may also be integrated into the same device, and the specific architecture of the information processing system can be determined according to the actual application scenario. The following only describes the specific implementation process of the application stage and the training stage of the first machine learning model 201.

[0071] I. Application Phase

[0072] Please refer to Figure 3, which is a flowchart illustrating an information processing method provided in an embodiment of this application. The information processing method provided in this embodiment may include:

[0073] 301. Obtain the initial environmental information around the vehicle.

[0074] For example, the first environmental information can be the environmental information at the current moment, that is, the first environmental information can include the traffic environment around the vehicle (hereinafter referred to as "the first vehicle" for easy distinction) at the current moment. The first environmental information can include at least one image and / or point cloud data corresponding to the traffic environment at the current moment. The first vehicle in this application can also be referred to as a self-driving vehicle.

[0075] For example, step 301 may include: the first vehicle can acquire at least one image of the traffic environment around the first vehicle at the current moment through a first sensor; optionally, the acquired at least one image can be an image from a perspective view (PV); and / or, the first vehicle can acquire point cloud data of the traffic environment around the first vehicle at the current moment through a second sensor. It should be understood that in this application, the executing entity is described as "the first vehicle," but the actual executing entity can be the intelligent driving system within the first vehicle.

[0076] For example, "traffic environment around the first vehicle" can be understood as the environment within the field of view of the sensors deployed on the first vehicle (such as the aforementioned first or second sensor). Exemplarily, the first sensor can be a photoelectric sensor, such as a camera or event camera; for example, the second sensor can be an ultrasonic sensor, a lidar sensor, a millimeter-wave radar sensor, or other sensors capable of measuring and obtaining point cloud data, etc., which are not exhaustively listed in the embodiments of this application.

[0077] For example, the vehicle can be a car, truck, motorcycle, bus, boat, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, airplane, and helicopter, etc., and the embodiments of this application do not impose any particular limitation. It should be noted that the executing entity in this application can be the intelligent driving system in the vehicle, and this application uses "vehicle" as the executing entity for illustration.

[0078] 302. Based on the first environmental information, category information is obtained. The category information includes the category of objects in a low-speed state. Objects in a low-speed state are objects whose driving speed is less than or equal to the first speed. The category indicates whether the object can be bypassed. The category is either the category of the object's low-speed state or the category of the reason that causes the object to be in a low-speed state.

[0079] For example, objects in a low-speed state include objects moving at low speed and objects in a stationary state. For instance, objects in a low-speed state may include vehicles in a low-speed state (hereinafter referred to as "second vehicles" for ease of distinction). The second vehicle in this application may also be referred to as another vehicle. Alternatively, objects in a low-speed state may also include electric vehicles, bicycles, or other objects moving in the traffic environment around the first vehicle, etc. The specific details can be determined in conjunction with the actual application scenario.

[0080] For example, the category information includes the category of each of at least one object in a low-speed state in the traffic environment surrounding the first vehicle at the current moment, and the category of each low-speed object indicates whether the object can be bypassed. For ease of description, any one of the at least one low-speed object will be referred to as the first object. For example, when the category of the first object is a first category, it indicates that the first object can be bypassed; when the category of the first object is a second category, it indicates that the first object cannot be bypassed.

[0081] To further understand this solution, please refer to Figure 4. Figure 4 is a schematic diagram of various scenarios of a vehicle in a low-speed state provided by the embodiments of this application. Figure 4 shows multiple scenarios in which the second vehicle is classified as Category 2 (i.e., the second vehicle cannot be bypassed) and multiple scenarios in which the second vehicle is classified as Category 1 (i.e., the second vehicle can be bypassed). As shown in Figure 4, when the second vehicle in a low-speed state is stuck in traffic, waiting at a red light, or queuing to turn right, the second vehicle is classified as Category 2, that is, the second vehicle in a low-speed state cannot be bypassed in the aforementioned scenarios. When a second vehicle that is stopped has its hazard lights flashing continuously or has a warning sign placed behind it (i.e., the stopped second vehicle is a disabled vehicle), or when the trunk of a stopped second vehicle is open, or when a stopped second vehicle is parked on the side of the road with its brake lights off or facing the wrong way (i.e., the stopped second vehicle is illegally parked on the side of the road), or when a stopped second vehicle is abnormally parked in the middle of the road, or when a stopped second vehicle is parked on the side of the road to pick up or drop off passengers, or when a second vehicle traveling at low speed has no vehicle in front of it and is not waiting at a red light, the second vehicle falls into category one. This means that in the aforementioned scenarios, a second vehicle traveling at low speed can be bypassed. It should be understood that the examples in Figure 4 are only for illustrative purposes and are not intended to limit the scope of this solution.

[0082] In one scenario, the category of each object in a low-speed state can be the category of the object's low-speed state; alternatively, the category of each object in a low-speed state can represent whether the object's low-speed state is long-term or short-term.

[0083] For example, the first category can represent that the object's low-speed state is long-term. For instance, the first category can be "long-term low-speed state", "long-term stop", "long-term" or other words that represent that the object's low-speed state is long-term. The specific category can be determined in combination with the actual application scenario. In other words, when the category of the first object represents that the first object's low-speed state is long-term, it indicates that the first object can be bypassed.

[0084] The second category indicates that the object's low-speed state is short-term. For example, the second category can be "short-term low-speed state", "short stop", "short-term" or other words that indicate that the object's low-speed state is short-term. The specific category can be determined based on the actual application scenario. In other words, when the category of the first object indicates that the first object's low-speed state is short-term, it indicates that the first object does not need to be detoured.

[0085] In another scenario, the category of each object in a low-speed state can be the category of the cause that led to the object being in a low-speed state; for example, the category of each object in a low-speed state can represent whether the object's low-speed state is caused by the object itself or by the environment.

[0086] For example, the first category can represent that the low speed state of the object is caused by the object itself. For example, the first category can be "self-initiated low speed" or other words that represent that the low speed state of the object is caused by the object itself. The specific category can be determined in combination with the actual application scenario. In other words, when the category of the first object represents that the low speed state of the first object is caused by the object itself, it indicates that the first object can be bypassed.

[0087] The second category indicates that the object's low-speed state is caused by the environment. For example, the second category could be "environment-induced low speed" or other words that indicate that the object's low-speed state is caused by the environment. The specific category can be determined based on the actual application scenario. In other words, when the category of the first object indicates that the first object's low-speed state is caused by the environment, it indicates that the first object cannot be bypassed.

[0088] In this embodiment, the specific categories of objects in a low-speed state are further refined, which further improves the feasibility of this solution. If the object's low-speed state is long-term, since the vehicle needs to pass, it cannot wait behind the object indefinitely, so it will often choose to detour around it. If the object's low-speed state is short-term, that is, the vehicle can pass while following the object, it can choose not to detour around it. If the reason for the object's low-speed state is caused by the environment, the vehicle is often unable to detour around the object due to environmental limitations, so it can choose not to detour around it. If the reason for the object's low-speed state is caused by the object itself, the vehicle can often detour around the object in the environment, so it can choose to detour around it. Thus, the category of objects in a low-speed state not only indicates whether the object can be detoured, but also provides the reason for detouring, enhancing the interpretability of the intelligent driving process.

[0089] For example, step 302 may include: the first vehicle inputting first environmental information into a first machine learning model, and obtaining the category of each object in at least one object in a low-speed state through the first machine learning model.

[0090] Further, in one scenario, step 302 may include: the first vehicle generating a first feature corresponding to the current moment using a first machine learning model based on first environmental information; and generating category information using the first machine learning model based on the first feature and a second feature corresponding to each of the N historical moments. Here, N is an integer greater than or equal to 1; for example, N can be 1, 2, 3, 4, 5, or other values, which can be determined based on the actual application scenario. The time interval between the current moment and two adjacent moments in the N historical moments can be understood as a time step; for example, the length of a time step can be 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, or other time lengths, which can be determined based on the actual application scenario.

[0091] The second feature corresponding to each historical moment in N time points is obtained based on the second environmental information of that historical moment. For example, the current moment can be called moment t. Taking N as 3, the N historical moments can be N-1, N-2, and N-3. The second feature of moment N-1 can be obtained based on the second environmental information 1 of moment N-1, the second feature of moment N-2 can be obtained based on the second environmental information 2 of moment N-2, and the second feature of moment N-3 can be obtained based on the second environmental information 3 of moment N-3. It should be understood that the example here is only for the convenience of understanding the correspondence between the second feature and the second environmental information and is not intended to limit this solution. The method of obtaining the second feature of each historical moment can be the same as the method of obtaining the first feature of the current moment. For details, please refer to the subsequent description of obtaining the first feature based on the first environmental information of the current moment. It will not be described in detail here.

[0092] In this embodiment, after obtaining the first feature corresponding to the current time based on the first environmental information at the current time, category information is generated by the first machine learning model based on the first feature and the second feature corresponding to the historical time. That is, the category of objects in a low-speed state in the current environment is generated. The second feature is obtained based on the second environmental information at the historical time. That is, the category of objects in a low-speed state in the current environment is generated by using environmental information from multiple times. This not only helps to improve the accuracy of the category information corresponding to a single time obtained by the first machine learning model, but also helps to improve the continuity and stability between the category information obtained by the first machine learning model at multiple different times. In addition, directly using the second feature at the historical time, instead of using the second environmental information at the historical time, avoids the repetitive feature extraction of the second environmental information at the historical time, which helps to reduce the computational resources consumed by the first machine learning model in the process of generating category information.

[0093] Regarding the specific implementation method of the first vehicle generating the first feature corresponding to the current moment through the first machine learning model, in one implementation method, the first vehicle can input the first environmental information and the second feature corresponding to each of the M historical moments into the first machine learning model. The feature extraction module in the first machine learning model extracts features from the first environmental information and the second features corresponding to each of the M historical moments (i.e., the M second features) to obtain the first feature. Here, M is an integer greater than or equal to 1, for example, the value of M can be 1, 2, 3, 4, 5 or other values, etc. The values ​​of M and N can be the same or different, and can be determined according to the actual application scenario.

[0094] In this embodiment, a first machine learning model is used to extract features from the first environmental information and the second features corresponding to historical moments to obtain the first feature corresponding to the current moment. Thus, the first feature corresponding to the current moment not only includes the features extracted from the first environmental information at the current moment, but also includes the features extracted from the second features corresponding to historical moments. Therefore, when generating the category information corresponding to the current moment using the first feature, it is possible to understand not only the environmental information at the current moment, but also the environmental information at historical moments, which is beneficial to further improve the accuracy of the category information corresponding to a single moment obtained by the first machine learning model.

[0095] In another implementation, the first vehicle can input the first environmental information at the current moment into the first machine learning model, and extract features from the first environmental information through the feature extraction module in the first machine learning model to obtain the first feature.

[0096] In another implementation, the first vehicle can input the first environmental information at the current moment and the second environmental information of each of the M historical moments into the first machine learning model. The feature extraction module in the first machine learning model extracts features from the first environmental information and the second environmental information of each of the M historical moments (i.e., the M second environmental information) to obtain the first feature.

[0097] For example, in one case, if the input of the first machine learning model includes first environmental information and M second features, then the first feature may include the first environmental information and the features of the M second features; or, if the input of the first machine learning model only includes the first environmental information, then the first feature may include the features of the first environmental information; or, if the input of the first machine learning model includes the first environmental information and M second environmental information, then the first feature may include the first environmental information and the features of the M second environmental information.

[0098] For example, the feature extraction module of the first machine learning model may include at least one of the following: a convolutional neural network (CNN) layer, a fully connected neural network layer, a neural network layer based on an attention mechanism, or other types of neural network layers, which can be determined in combination with the actual application scenario.

[0099] In another scenario, the first feature may include at least one sub-feature corresponding one-to-one with at least one object in the traffic environment at the current moment that is in a low-speed state. Each sub-feature is a feature of each of the aforementioned at least one object in a low-speed state. For example, each sub-feature may include a value, or each sub-feature may include a vector, or each sub-feature may include a matrix, etc., which can be determined in combination with the actual application scenario.

[0100] Optionally, the feature extraction module of the first machine learning model can be the feature extraction module of the object detection model. The object detection model refers to the machine learning model used to detect objects in the traffic environment at the current moment. In other words, the object detection model can be understood as the first machine learning model that detects objects in the traffic environment at the current moment that are in a low-speed state.

[0101] For example, the feature extraction module of the first machine learning model may include: a convolutional neural network layer + a region proposal network (RPN) layer. The features of the first environmental information (optionally, also including M second features or M second environmental information) can be obtained through the convolutional neural network layer. The aforementioned features can be represented in the form of a feature map. The RPN determines the target region of each object in the aforementioned feature map among at least one object in a low-speed state. Thus, the sub-features corresponding to each object in at least one object in a low-speed state in the traffic environment at the current moment can be obtained from the features of the first environmental information (optionally, also including M second features or M second environmental information).

[0102] Alternatively, the feature extraction module of the first machine learning model may include: a convolutional neural network layer + a transformer neural network layer. The features of the first environmental information (optionally, also including M second features or M second environmental information) can be obtained through the convolutional neural network layer. Then, based on the aforementioned features, the transformer neural network layer can be used to obtain the sub-features corresponding to each object in at least one object in a low-speed state in the traffic environment at the current moment.

[0103] Alternatively, the feature extraction module of the first machine learning model can also be composed of other types of neural network layers. This example is only to demonstrate the feasibility of this solution, and the specific implementation can be determined based on the actual application scenario.

[0104] Optionally, the first vehicle is also equipped with a feature processing module for an object detection model. For example, the feature processing module can be a decoder. It may include at least one of the following: a fully connected neural network layer, a residual neural network layer, a multilayer perceptron (MLP), a support vector machine, or other types of neural network layers, depending on the specific application scenario. The first vehicle can also input a first feature into the feature processing module of the object detection model. Based on the sub-features corresponding to each object in the at least one low-speed object included in the first feature, the feature processing module generates the position information of each low-speed object in the traffic environment at the current moment. Optionally, the feature processing module can also generate the confidence score corresponding to each of the aforementioned position information. This can be understood as the first machine learning model and the object detection model sharing the same feature extraction module.

[0105] For example, the position information of each object in a low-speed state in the traffic environment at the current moment can be the position information of each object in a low-speed state in the vehicle coordinate system of the first vehicle. In other words, the position information of each object in a low-speed state in the traffic environment at the current moment can be the position of each object in a low-speed state relative to the first vehicle; or, the position information of each object in a low-speed state in the traffic environment at the current moment can be the position information of each object in a low-speed state in the front view image of at least one of the above-mentioned images, etc., which can be determined according to the actual application scenario. The confidence level corresponding to the position information of each object in a low-speed state can indicate the confidence level of the position information of the object in a low-speed state, or it can also be understood as the confidence level of the object in a low-speed state in terms of its position information.

[0106] Regarding the specific implementation method of generating category information for a first vehicle based on a first feature and a second feature corresponding to each of N historical moments using a first machine learning model, in one implementation method, the first vehicle obtains fused features through the feature fusion module of the first machine learning model based on the first feature and the second feature corresponding to each of the N historical moments. The fused features include a third feature of each object in at least one low-speed state. The third feature includes features of the same low-speed object at the current moment and at historical moments. Specifically, the third feature includes a first sub-feature of the same low-speed object at the current moment and a second sub-feature of at least one historical moment. Based on the fused features including the third feature of each object in at least one low-speed state, the first vehicle generates a category for each object in at least one low-speed state through the feature processing module of the first machine learning model.

[0107] For example, the first feature includes a sub-feature corresponding to each of at least one object in the traffic environment at the current time that is in a low-speed state, and the second feature corresponding to each of the N historical times includes a sub-feature corresponding to each of at least one object in the traffic environment at that historical time that is in a low-speed state; optionally, the first vehicle may also obtain the identity (ID) of the sub-feature corresponding to each object in the low-speed state included in each second feature, and may also generate the ID of the sub-feature corresponding to each object in the low-speed state included in the first feature, and the sub-feature of the same object at different times may have the same ID.

[0108] For example, in one implementation, the first device can generate the ID of each sub-feature of the first feature by calculating the similarity between the sub-features included in the first feature and the sub-features in the second feature corresponding to the historical time. The higher the similarity between two sub-features, the higher the probability that the two sub-features are assigned the same ID. In another implementation, the first device can input the first feature, multiple second features corresponding one-to-one with multiple historical time points, and the ID of each sub-feature in each second feature into a second machine learning model to obtain the ID of each sub-feature in the first feature generated by the second machine learning model.

[0109] To more intuitively understand this solution, please refer to Figure 5. Figure 5 is a schematic diagram of the identification of sub-features provided in an embodiment of this application. In Figure 5, N is taken as 2. Time t in Figure 5 represents the current time, and time t-1 and time t-2 in Figure 5 represent two historical times. Each small box in Figure 5 represents a sub-feature corresponding to an object in a low-speed state. Each sub-feature corresponds to an ID. The sub-features of the same object at different times can have the same ID. As shown in Figure 5, the second feature 2 corresponding to time t-2, the second feature 1 corresponding to time t-1, and the first feature corresponding to time t all include a sub-feature corresponding to the ID 1. The three sub-features corresponding to ID 1 are the three sub-features of the same object at times t-2, t-1, and t, respectively; the second feature 2 corresponding to time t-2, the second feature 1 corresponding to time t-1, and the first feature corresponding to time t all include the sub-features corresponding to ID 3, indicating that the three sub-features corresponding to ID 3 are the three sub-features of the same object at times t-2, t-1, and t, respectively; Figure 5 also shows IDs 7 and 5; please refer to the above description of IDs 1 and 3, which will not be elaborated here. It should be understood that the examples in Figure 5 are only for the convenience of understanding this scheme and are not intended to limit this scheme.

[0110] The first vehicle, based on a first feature and a second feature corresponding to each of the N historical moments, obtains a third feature for each object in at least one low-speed object through the feature fusion module of the first machine learning model. This can include: the first vehicle, based on the first feature, the ID of each sub-feature of the first feature, the second feature corresponding to each of the N historical moments, and the ID of each sub-feature of each second feature, obtains a third feature for each object in at least one low-speed object in the traffic environment at the current moment through the feature fusion module of the first machine learning model. Each third feature includes the features of a low-speed object in the traffic environment at the current moment in both the current moment and historical moments.

[0111] For ease of description, any one of the multiple sub-features included in the first feature is referred to as the first sub-feature. For example, the first vehicle can obtain at least one second sub-feature from the N second features corresponding to N historical moments through the feature fusion module of the first machine learning model, based on the ID of the first sub-feature and the ID of each sub-feature in each second feature. The ID of each second sub-feature is the same as the ID of the first sub-feature. Based on the first sub-feature and at least one second sub-feature, the first vehicle obtains a third feature of an object in a low-speed state in the traffic environment at the current moment through the feature fusion module of the first machine learning model.

[0112] Optionally, the first vehicle can use the feature fusion module of the first machine learning model to concatenate the first sub-feature and at least one second sub-feature to obtain a third feature of an object in a low-speed state in the traffic environment at the current moment.

[0113] For example, the number of second sub-features included in at least one second sub-feature can be less than or equal to N. If each of the N second features contains a second sub-feature with the same ID as the first sub-feature, then at least one second sub-feature includes N second sub-features. If any one of the N second features does not contain a sub-feature with the same ID as the first sub-feature, then the number of second sub-features included in at least one second sub-feature is less than N. Optionally, when the number of second sub-features included in at least one second sub-feature is less than N, the third feature may include not only the first sub-feature and at least one second sub-feature, but may also include preset information. In other words, preset information can be filled into the third feature. For example, the preset information may be a value consisting entirely of 0s, or the preset information may include values ​​generated according to a preset rule, such as 0101…0101, etc.

[0114] The vehicle can perform the above operation on each of the multiple sub-features included in the first feature to obtain the third feature of each object in the traffic environment at the current moment that is in a low-speed state, that is, to obtain the fused feature.

[0115] To understand this solution more intuitively, please refer to Figure 6. Figure 6 is a schematic diagram of obtaining the third feature of each object in at least one object in a low-speed state in the traffic environment at the current moment based on the first feature and N second features provided by the embodiment of this application. Figure 6 includes two sub-schematic diagrams, left and right. The left sub-schematic diagram of Figure 6 shows the first feature at the current moment (i.e., time t in Figure 6) and the two second features corresponding to the two historical moments, i.e., the second feature 1 corresponding to time t-1 and the second feature 2 corresponding to time t-2 in Figure 6. The left sub-schematic diagram of Figure 6 can be understood in conjunction with the above description of Figure 5, and the repeated parts will not be repeated.

[0116] The right sub-schematic of Figure 6 represents the third feature of each object in at least one low-speed object in the traffic environment at the current time, obtained based on the first feature at time t, the second feature 1 corresponding to time t-1, and the second feature 2 corresponding to time t-2. As shown in the right sub-schematic of Figure 6, the first row represents the third feature 1 of an object 1 in a low-speed state corresponding to ID 1. The third feature 1 includes the features of object 1 at time t-2, time t-1, and time t; the second row represents the third feature 2 of an object 2 in a low-speed state corresponding to ID 3. The third feature 2 includes the features of object 2 at time t-2, t-1, and t; the third row represents the third feature 3 of object 3 in a low-speed state corresponding to ID 5, which includes the features of object 3 at time t-2, t-1, and t; the fourth row represents the third feature 4 of object 4 in a low-speed state corresponding to ID 7, which includes the features of object 4 at time t-2, t-1, and t. It should be understood that the example in Figure 6 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0117] In this embodiment of the application, based on the first feature and the second feature corresponding to each of the N historical features, the features of the object in the current low-speed state at the current time and at least one historical time are obtained. Then, based on the features of the object in the low-speed state at multiple times, the category of the object in the low-speed state is determined, which is beneficial to obtaining a more accurate category of the object in the low-speed state.

[0118] After obtaining the fused features, the first vehicle can generate the category of each object in at least one low-speed object in the traffic environment at the current moment based on the fused features through the feature processing module of the first machine learning model; the feature processing module of the first machine learning model in this application can also be replaced by the semantic prediction module of the first machine learning model.

[0119] For example, the feature processing module of the first machine learning model may include at least one of the following: a fully connected neural network layer, a multilayer perceptron, a residual neural network layer, a classifier, or other types of neural network layers.

[0120] Optionally, the feature processing module of the first machine learning model may further include a neural network layer based on an attention mechanism; for example, the aforementioned neural network layer based on an attention mechanism may be a self-attention neural network layer. After the first vehicle obtains the fused features, it can process them through the aforementioned self-attention neural network layer to obtain updated features; and then, based on the updated features, generate the category of each object in at least one object in the traffic environment at the current moment that is in a low-speed state.

[0121] Alternatively, the aforementioned attention-based neural network layer can be a cross-attention neural network layer. After obtaining the fused features, the first vehicle can process the fused features and the first environmental information (optionally, also including M second features or M second environmental information) using a cross-attention neural network layer to obtain updated features. Then, based on the updated features, the category of each object in at least one object in the traffic environment at the current moment that is in a low-speed state is generated.

[0122] In another implementation, where the first feature includes sub-features corresponding to each object in at least one low-speed object in the current traffic environment, and the second feature includes sub-features corresponding to each object in at least one low-speed object in the traffic environment at a historical time, the first vehicle can further obtain the ID of each sub-feature included in each second feature and the ID of each sub-feature included in the first feature. Based on the first feature, the ID of each sub-feature in the first feature, the second feature corresponding to each historical time in N historical times, and the ID of each sub-feature in each second feature, the feature fusion module of the first machine learning model determines the first sub-feature and at least one second sub-feature corresponding to each low-speed object in the current traffic environment. The specific implementation of the aforementioned steps can be found in the above description and will not be repeated here. The first vehicle can select a target sub-feature from the first sub-feature and at least one second sub-feature corresponding to each low-speed object in the current traffic environment through the feature fusion module of the first machine learning model to obtain the fused feature, wherein the fused feature includes the target sub-feature corresponding to each object in at least one low-speed object in the current traffic environment.

[0123] Optionally, the first vehicle selects a target sub-feature from the first sub-feature and at least one second sub-feature corresponding to each object in the traffic environment at the current moment that is in a low-speed state. This may include: the first vehicle selecting a target sub-feature from the first sub-feature and at least one second sub-feature based on the confidence level of the first sub-feature and the confidence level of each of the at least one second sub-feature. Optionally, the target sub-feature may be the one with the highest confidence level among the first sub-feature and at least one second sub-feature.

[0124] For example, the confidence scores of the first sub-feature and each second sub-feature can be obtained through the object detection model described above. Since the feature processing module of the object detection model generates the position information and corresponding confidence scores of each object based on the sub-features corresponding to each object, the confidence scores corresponding to the position information of each object can be determined as the confidence scores corresponding to each sub-feature.

[0125] Alternatively, the first vehicle may randomly select one sub-feature as the target sub-feature from the first sub-feature and at least one second sub-feature corresponding to each object in a low-speed state; the specific selection rules can be determined in combination with the actual application scenario.

[0126] After obtaining the fused features, the first vehicle can generate the category of each object in at least one low-speed object in the traffic environment at the current moment through the feature processing module of the first machine learning model, based on the fused features, that is, generate category information; the specific implementation of the aforementioned steps and the meaning of the terms can be found in the above description, and will not be repeated here.

[0127] In another implementation, regardless of whether the first feature includes first environmental information (optionally, it also includes M second features or M second environmental information) or the first feature includes at least one sub-feature corresponding to at least one object in the traffic environment at the current moment that is in a low-speed state, after the first vehicle obtains the first feature and the second feature corresponding to each of the N historical moments, that is, after obtaining the first feature and the N second features, it can also directly concatenate the first feature and the N second features through the feature fusion module of the first machine learning model to obtain the fused feature.

[0128] After obtaining the fused features, the first vehicle can generate the category of each object in at least one low-speed object in the traffic environment at the current moment through the feature processing module of the first machine learning model, that is, generate category information; the specific implementation of the above steps and the meaning of the terms can be found in the above description, and will not be repeated here.

[0129] To better understand this solution, please refer to Figure 7. Figure 7 is another schematic diagram of the information processing method provided in the embodiment of this application. As shown in Figure 7, the first environmental information at the current moment and the second features (i.e., the M second features) corresponding to each of the M historical moments are input into the first machine learning model. The feature extraction module of the first machine learning model extracts features from the first environmental information and the M second features to obtain the first feature corresponding to the current moment (i.e., time t in Figure 7). Based on the first feature corresponding to time t and the second features corresponding to each of the N historical moments (including time t-1 and time t-2 in Figure 7), the feature fusion module of the first machine learning model processes the features to obtain the third features of each object in at least one object in a low-speed state at the current moment and at least one historical moment. Then, based on the aforementioned third features of each object in at least one object in a low-speed state at the current moment and at least one historical moment, the feature processing module of the first machine learning model generates category information. It should be understood that the example in Figure 7 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0130] In another implementation, regardless of whether the first feature includes first environmental information (optionally, it also includes M second features or M second environmental information) or the first feature includes at least one sub-feature corresponding to at least one object in the traffic environment at the current moment that is in a low-speed state, after the first vehicle obtains the first feature and the second feature corresponding to each of the N historical moments, it can directly input the first feature and the N second features sequentially into the feature processing module of the first machine learning model to obtain the category information generated by the feature processing module of the first machine learning model.

[0131] The feature processing module of the first machine learning model may include a neural network layer for processing time-series data. For example, the feature processing module of the first machine learning model may include a recurrent neural network layer, a gate recurrent unit (GRU) or a long short-term memory (LSTM) neural network layer, etc. The embodiments of this application do not exhaustively list them.

[0132] In another scenario, step 302 may include: the first vehicle generating a first feature corresponding to the current moment based on the first environmental information through the feature extraction module in the first machine learning model; the specific implementation methods and the meanings of the terms in the aforementioned steps can be found in the above description, and will not be repeated here. Furthermore, the first vehicle processes the first feature through the feature processing module of the first machine learning model to generate category information. In other words, the first vehicle may not need the second feature corresponding to the historical moment in the process of generating category information.

[0133] Optionally, the category information generated by the first machine learning model is used to obtain the control information of the first vehicle. The control information of the first vehicle may include at least one of the following: the planned position of the first vehicle corresponding to each of at least one time after the current time, the driving strategy of the first vehicle corresponding to each of at least one time after the current time, the control information of the first vehicle corresponding to each of at least one time after the current time, or other types of control information, which can be determined in combination with the actual application scenario.

[0134] For example, the planned position of the first vehicle corresponding to each time moment can also be understood as the planned trajectory point of the first vehicle corresponding to each time moment. If at least one time moment is specifically at least two time moments, then the at least two planned trajectory points of the first vehicle corresponding one-to-one with the at least two time moments can also be understood as the planned trajectory of the first vehicle at the aforementioned at least two time moments.

[0135] The driving strategy for the first vehicle at each time point may include the lateral driving strategy and / or longitudinal driving strategy corresponding to each time point. The lateral driving strategy corresponding to a certain time point may be left turn, straight, right turn, lane change or lateral avoidance, etc., and the longitudinal driving strategy corresponding to a certain time point may be acceleration, constant speed or deceleration, etc. The specific manifestation of the driving strategy can be determined in combination with the actual application scenario.

[0136] The control information of the first vehicle corresponding to each moment may include: control signals of at least one component of the first vehicle corresponding to each moment, and / or, the planned state information of the first vehicle corresponding to each moment. For example, at least one component may include a steering wheel, engine, brakes, clutch, turn signals, or other components used during the driving of the first vehicle; the planned state information of the first vehicle corresponding to each moment after the current moment can be understood as the state information that the first vehicle needs to reach.

[0137] In this embodiment of the application, the category information can also be used to obtain the control information of the first vehicle. The vehicle control information includes at least one of the following: the planned location of the vehicle, the driving strategy of the vehicle, or the control information of the vehicle. This provides multiple application scenarios for the category information, which is beneficial to expanding the application scenarios of this application.

[0138] Optionally, the first vehicle can also output this type of information. For example, the first vehicle can play this type of information via voice, allowing passengers to clearly understand the reason for the vehicle's decision to change lanes or not. For instance, the voice message could be, "The vehicle ahead is moving at a consistently low speed; our vehicle will initiate a lane change to improve traffic flow," or "The vehicle ahead is moving at a low speed due to environmental factors; therefore, we recommend following and waiting." Alternatively, the first vehicle can output this type of information through a display interface. For example, it can display the second vehicle ahead of the first vehicle on a map and add text to the second vehicle to display this type of information. The text could be, for example, "constantly low speed," "short-term low speed," "low speed caused by environmental factors," "low speed caused by a vehicle," or other text. Optionally, the first vehicle can also highlight the second vehicle on the map, for example, by highlighting the second vehicle, flashing the second vehicle, etc. The specific implementation method can be determined based on the actual application scenario.

[0139] In this embodiment, after obtaining the first environmental information around the vehicle, category information can be obtained based on the first environmental information. The category information includes the category of objects in a low-speed state. Objects in a low-speed state are those with a driving speed less than or equal to a first speed. The aforementioned category indicates whether objects in a low-speed state can be bypassed. Here, the category is the category of the object's low-speed state, or the category is the category of the reason that causes the object to be in a low-speed state. That is, what can be provided to the downstream module in this application is the category of objects in a low-speed state. The aforementioned category indicates whether objects in a low-speed state can be bypassed. Then, the downstream module can directly determine whether the aforementioned objects in a low-speed state can be bypassed by using the aforementioned category information. Thus, the downstream module no longer needs to process attributes of multiple dimensions, providing a simpler processing solution.

[0140] II. Training Phase

[0141] Please refer to Figure 8, which is a flowchart illustrating a model training method provided in an embodiment of this application. The model training method provided in an embodiment of this application may include:

[0142] 801. Obtain the initial environmental information around the vehicle.

[0143] For example, the training device may be equipped with a training dataset, which may include at least one training sample and at least one expected category information corresponding to each training sample. Each training sample includes at least first environmental information around the vehicle. The meaning of the first environmental information can be found in the description in the embodiment corresponding to Figure 3 above, and will not be repeated here.

[0144] 802. Based on the first environmental information, the predicted category information is obtained through the first machine learning model. The predicted category information includes the predicted category of objects in a low-speed state. Objects in a low-speed state are objects whose driving speed is less than or equal to the first speed.

[0145] The specific implementation of step 802 can be found in the description of step 302 in the corresponding embodiment of Figure 3. The difference is that the execution subject is replaced by the training device instead of the first vehicle, and the meaning of the predicted category information is similar to that of the category information in the corresponding embodiment of Figure 3. Therefore, they will not be repeated here.

[0146] Optionally, each training sample may include environmental information at multiple times. The environmental information at multiple times is the environmental information at multiple consecutive times obtained by the same vehicle during its driving in a certain traffic environment. For example, the first environmental information is the environmental information at time t. The aforementioned environmental information at multiple times may also include the second environmental information at each time in at least one time before time t.

[0147] 803. The first machine learning model is trained using a loss function, which indicates the similarity between the predicted category information and the expected category information. The expected category information includes the correct category of the object in a low-speed state. Both the predicted category and the correct category indicate whether the object can be bypassed. The predicted category and the correct category are both categories of the object's low-speed state, or the predicted category and the correct category are categories of the reasons that cause the object to be in a low-speed state.

[0148] For example, since a training sample may include environmental information at one or more times, the corresponding training sample may include a correct category information corresponding to each of the aforementioned one or more times. A correct category information may indicate the correct category of each object in at least one object in a low-speed state in the traffic environment at the corresponding time. The meaning of the correct category information is similar to the meaning of the category information in the corresponding embodiment of Figure 3, and will not be repeated here.

[0149] For example, after performing step 802 at least once, the training device can generate a predicted category information corresponding to each of the above one or more time points. Step 803 may include: the training device generating a function value of a loss function based on a predicted category information corresponding to each time point and a correct category information corresponding to each time point; the training device may perform gradient differentiation on the function value of the loss function and update the weight parameters of the first machine learning model using the backpropagation algorithm to achieve one training of the first machine learning model.

[0150] The loss function indicates the similarity between a predicted class at each time step and a correct class at each time step. The goal of training using the loss function is to improve the similarity between the predicted class at each time step and the correct class at each time step.

[0151] Optionally, if step 803 further generates predicted location information for each object in a low-speed state in the traffic environment at each time step through the feature processing module of the object detection model, each training data may also include correct location information for each object in a low-speed state in the traffic environment at each time step. The meanings of predicted location information and correct location information can be found in the description of location information in the corresponding embodiment of Figure 3, and will not be repeated here. The loss function can also indicate the similarity between the predicted location information and the correct location information for each object in a low-speed state in the traffic environment at each time step. The goal of training using the loss function also includes improving the similarity between the predicted location information and the correct location information for each object in a low-speed state in the traffic environment at each time step.

[0152] Step 803 may include: the training device generating a loss function value based on a predicted category information corresponding to each time step, a correct category information corresponding to each time step, the predicted position information and the correct position information of each object in a low-speed state in the traffic environment at each time step; the training device may perform gradient differentiation on the loss function value and use the backpropagation algorithm to update the weight parameters of the first machine learning model (optionally, it may also include updating the weight parameters of the feature processing module of the object detection model) to achieve one training of the first machine learning model.

[0153] The training device can repeatedly execute steps 801 to 803 to train the first machine learning model (optionally, also including the feature processing module of the object detection model) multiple times until the convergence condition is met, thus obtaining the trained first machine learning model (optionally, also including the trained feature processing module). For example, the convergence condition may include: satisfying the convergence condition of the loss function and / or training the first machine learning model a preset number of times.

[0154] Based on the embodiments corresponding to Figures 1 to 8, in order to better implement the above-described solutions of the embodiments of this application, related equipment for implementing the above-described solutions is also provided below. Specifically, refer to Figure 9, which is a schematic diagram of an information processing device provided in an embodiment of this application. The information processing device 900 includes: an acquisition module 901, used to acquire first environmental information surrounding the vehicle; and a processing module 902, used to obtain category information based on the first environmental information. The category information includes the category of objects in a low-speed state, where objects in a low-speed state are those with a driving speed less than or equal to a first speed. The category indicates whether the object can be bypassed. The category is either the category of the object's low-speed state or the category of the cause leading to the object being in a low-speed state.

[0155] Optionally, if the category is a category of the object's low-speed state, the category represents whether the object's low-speed state is long-term or short-term; or, if the category is a category of the cause that leads to the object being in a low-speed state, the category represents whether the object's low-speed state is caused by the object or by the environment.

[0156] Optionally, the first environmental information is the environmental information at the current moment. The processing module 902 is specifically used to: generate a first feature corresponding to the current moment through a machine learning model based on the first environmental information; and generate category information through a machine learning model based on the first feature and the second feature corresponding to each of the N historical moments, where N is an integer greater than or equal to 8, and the second feature is obtained based on the second environmental information of the historical moment.

[0157] Optionally, the category information includes the category of each object in at least one object in a low-speed state. The processing module 902 is specifically used to: obtain a third feature of each object in at least one object in a low-speed state based on the first feature and the second feature corresponding to each historical moment in N historical moments. The third feature includes the features of the same object in a low-speed state at the current moment and at historical moments; and generate the category of each object in at least one object in a low-speed state through a machine learning model based on the third feature of each object in at least one object in a low-speed state.

[0158] Optionally, the processing module 902 is specifically used to: input the first environmental information and the second feature corresponding to each of the M historical moments into the machine learning model, and extract features from the first environmental information and the second feature corresponding to each of the M historical moments through the machine learning model to obtain the first feature, where M is an integer greater than or equal to 8.

[0159] Optionally, the category information is used to obtain the vehicle's control information, which includes at least one of the following: the vehicle's planned location, the vehicle's driving strategy, or the vehicle's control information.

[0160] It should be noted that the information interaction and execution process between the modules / units in the information processing device 900 are based on the same concept as the various method embodiments corresponding to Figures 1 to 8 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0161] Please refer to Figure 10, which is a schematic diagram of a model training device provided in an embodiment of this application. The model training device 1000 includes: an acquisition module 1001, used to acquire first environmental information around the vehicle; a processing module 1002, used to obtain predicted category information based on the first environmental information through a machine learning model, the predicted category information including the predicted category of an object in a low-speed state, the object in a low-speed state being an object whose driving speed is less than or equal to a first speed; and a training module 1003, used to train the machine learning model using a loss function, the loss function indicating the similarity between the predicted category information and the expected category information, the expected category information including the correct category of the object in a low-speed state, both the predicted category and the correct category indicating whether the object can be bypassed, wherein the predicted category and the correct category are both categories of the object in a low-speed state, or the predicted category and the correct category are categories of the reasons that cause the object to be in a low-speed state.

[0162] It should be noted that the information interaction and execution process between the modules / units in the model training device 1000 are based on the same concept as the various method embodiments corresponding to Figures 1 to 8 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0163] The following describes a device provided in an embodiment of this application. Please refer to Figure 11, which is a structural schematic diagram of a device provided in an embodiment of this application. Optionally, the device 1100 performs the functions of the first vehicle or training device in the various method embodiments corresponding to Figures 1 to 8.

[0164] Device 1100 includes a memory 1102 and at least one processor 1101. Optionally, device 1100 further includes at least one accelerator 1103. Optionally, processor 1101 implements the method in the above embodiments by reading program instructions stored in memory 1102; or, processor 1101 reads program instructions stored in memory 1102 and implements the steps executed by the machine learning model in the method in the above embodiments through accelerator 1103; or, processor 1101 may also implement the method in the above embodiments by reading program instructions stored internally; or, processor 1101 may also read program instructions stored internally and implement the steps executed by the machine learning model in the method in the above embodiments through accelerator 1103.

[0165] When the processor 1101 reads the program instructions stored in the memory 1102 to implement the method in the above embodiments, the memory 1102 stores the program instructions that implement the method provided in the above embodiments of this application.

[0166] Optionally, at least one processor 1101 is one or more CPUs, either a single-core CPU or a multi-core CPU. For example, memory 1102 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, or optical memory. Memory 1102 stores program instructions for the operating system. For example, at least one accelerator 1103 may include at least one of the following: GPU, NPU, TPU, ASIC, FPGA, or other types of accelerators. After the program instructions stored in memory 1102 are read by the at least one processor 1101, device 1100 executes the corresponding operations in the foregoing embodiments.

[0167] Optionally, the device 1100 also includes a network interface 1104, which can be a wired interface or a wireless interface. The network interface 1104 is used to perform data transmission and reception in the various method embodiments corresponding to Figures 1 to 8.

[0168] It should be understood that network interface 1104 has the functions of receiving and sending data. The functions of "receiving data" and "sending data" can be integrated into the same transceiver interface, or the functions of "receiving data" and "sending data" can be implemented in different interfaces, which is not limited here. In other words, network interface 1104 may include one or more interfaces for implementing the functions of "receiving data" and "sending data".

[0169] After the processor 1101 reads the program instructions from the memory 1102, other functions that the device 1100 can perform are described in the preceding method embodiments.

[0170] Optionally, the device 1100 also includes a bus 1105, through which the processor 1101 and memory 1102 are typically interconnected, or in other ways.

[0171] The device 1100 provided in this application embodiment is used to execute the methods executed by the first vehicle or training device in the above-described method embodiments, and to achieve the corresponding beneficial effects. The specific implementation of the device 1100 shown in FIG11 can be referred to the descriptions in the foregoing method embodiments, and will not be repeated here.

[0172] This application also provides a vehicle, as shown in Figure 12. Figure 12 is a structural schematic diagram of a vehicle provided in this application embodiment. The vehicle 100 is configured for fully or partially automated driving mode. For example, the vehicle 100 can control itself while in automated driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the probability of other vehicles performing possible behaviors, and control the vehicle 100 based on the determined information. When the vehicle 100 is in automated driving mode, the vehicle 100 can also be set to operate without human interaction.

[0173] Vehicle 100 may include various subsystems, such as a mobility system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112, and a user interface 116. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0174] The mobility system 102 may include components that provide powered motion to the vehicle 100. In one embodiment, the mobility system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels / tires 121.

[0175] Engine 118 can be an internal combustion engine, an electric motor, an air-compressed engine, or other combinations of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. Engine 118 converts energy source 119 into mechanical energy. Examples of energy source 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 can also provide energy to other systems of vehicle 100. Transmission 120 transmits mechanical power from engine 118 to wheels 121. Transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 121.

[0176] Sensor system 104 may include several sensors for sensing information about the environment surrounding vehicle 100. For example, sensor system 104 may include a positioning system 122 (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. Sensor system 104 may also include sensors for the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensing data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the autonomous vehicle 100.

[0177] The positioning system 122 can be used to estimate the geographical location of the vehicle 100. An IMU 124 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope. A radar 126 can use radio signals to sense objects in the surrounding environment of the vehicle 100, specifically millimeter-wave radar or lidar. In some embodiments, in addition to sensing objects, the radar 126 can also be used to sense the speed and / or direction of travel of objects. A laser rangefinder 128 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. A camera 130 can be used to capture multiple images of the surrounding environment of the vehicle 100. The camera 130 can be a still camera or a video camera.

[0178] The control system 106 controls the operation of the vehicle 100 and its components. The control system 106 may include various components, including a steering system 132, a throttle 134, a braking unit 136, a computer vision system 140, a trajectory control system 142, and an obstacle avoidance system 144.

[0179] The steering system 132 is operable to adjust the forward direction of the vehicle 100. For example, in one embodiment, it may be a steering wheel system. The throttle 134 controls the operating speed of the engine 118 and thus the speed of the vehicle 100. The braking unit 136 controls the deceleration of the vehicle 100. The braking unit 136 may use friction to slow down the wheels 121. In other embodiments, the braking unit 136 may convert the kinetic energy of the wheels 121 into electrical current. The braking unit 136 may also take other forms to slow down the rotational speed of the wheels 121 to control the speed of the vehicle 100. The computer vision system 140 is operable to process and analyze images captured by the camera 130 to identify objects and / or features in the environment surrounding the vehicle 100. The objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 may be used to map the environment, track objects, estimate the speed of objects, etc. The route control system 142 is used to determine the driving route and speed of the vehicle 100. In some embodiments, the route control system 142 may include a lateral planning module 1421 and a longitudinal planning module 1422, which are respectively used to combine data from the obstacle avoidance system 144, GPS 122, and one or more predetermined maps to determine the driving route and speed for the vehicle 100. The obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise traverse obstacles in the environment of the vehicle 100, which may specifically be physical obstacles and virtual moving bodies that may collide with the vehicle 100. In one example, the control system 106 may add or alternatively include components other than those shown and described. Alternatively, some of the components shown above may be reduced.

[0180] Vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral device 108. Peripheral device 108 may include wireless communication system 146, on-board computer 148, microphone 150, and / or speaker 152. In some embodiments, peripheral device 108 provides a means for a user of vehicle 100 to interact with user interface 116. For example, on-board computer 148 may provide information to a user of vehicle 100. User interface 116 may also operate on-board computer 148 to receive user input. On-board computer 148 may be operated via a touchscreen. In other cases, peripheral device 108 may provide a means for vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from a user of vehicle 100. Similarly, speaker 152 may output audio to a user of vehicle 100. Wireless communication system 146 may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 146 may utilize a wireless local area network (WLAN) for communication. In some embodiments, the wireless communication system 146 may utilize an infrared link, Bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various vehicle communication systems, may also be used. For example, the wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices that can enable public and / or private data communication between the vehicle and / or a roadside station.

[0181] Power source 110 can provide power to various components of vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, as is the case in some fully electric vehicles.

[0182] Some or all of the functions of vehicle 100 are controlled by computer system 112. Computer system 112 may include at least one processor 113, which executes program instructions 115 stored in a non-transitory computer-readable medium such as memory 114. Computer system 112 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner. Processor 113 may include any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, processor 113 may also include a dedicated device such as a GPU, NPU, TPU, ASIC, FPGA, or other hardware-based processor. Although FIG12 functionally illustrates the processor, memory, and other components of computer system 112 in the same block, those skilled in the art will understand that the processor or memory may actually include multiple processors or memories not stored in the same physical housing. For example, memory 114 may be a hard disk drive or other storage medium located in a housing different from that of computer system 112. Therefore, references to processor 113 or memory 114 will be understood to include references to a collection of processors or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration components, can each have their own processor that performs only calculations related to the component's specific function.

[0183] In all the aspects described herein, processor 113 may be located remotely from vehicle 100 and may communicate wirelessly with vehicle 100. In other aspects, some of the processes described herein are executed on processor 113 located within vehicle 100, while others are executed by remote processor 113, including taking the necessary steps to perform a single operation.

[0184] In some embodiments, memory 114 may contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of vehicle 100, including those described above. Memory 114 may also contain additional program instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the mobility system 102, sensor system 104, control system 106, and peripheral devices 108. In addition to instructions 115, memory 114 may also store data such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information may be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes. A user interface 116 is provided to or receives information from a user of vehicle 100. Optionally, user interface 116 may include one or more input / output devices within the set of peripheral devices 108, such as wireless communication system 146, on-board computer 148, microphone 150, and speaker 152.

[0185] Computer system 112 can control the functions of vehicle 100 based on input received from various subsystems (e.g., driving system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 can utilize input from control system 106 to control steering system 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 is operable to provide control over many aspects of vehicle 100 and its subsystems.

[0186] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 114 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0187] Optionally, the above components are merely examples. In practical applications, components in each of the above modules may be added or removed according to actual needs. Figure 12 should not be construed as a limitation on the embodiments of this application. A vehicle traveling on a road, such as vehicle 100 above, can identify objects in its surrounding environment to determine adjustments to its current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed adjustment to be made by the vehicle can be determined.

[0188] Optionally, vehicle 100 or computing devices associated with vehicle 100, such as computer system 112, computer vision system 140, and memory 114 as shown in Figure 12, can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can also be considered together to predict the behavior of a single identified object. Vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, vehicle 100 can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered in determining the speed of vehicle 100, such as the lateral position of vehicle 100 in the road, the curvature of the road, the proximity of static and dynamic objects, etc. In addition to providing program instructions to adjust the speed of the vehicle, the computing device may also provide program instructions to modify the steering angle of the vehicle 100 so that the vehicle 100 follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the vehicle 100 (e.g., cars in adjacent lanes on the road).

[0189] In this embodiment, the processor 113 in the vehicle 100 is used to execute the method performed by the first vehicle in the embodiments corresponding to Figures 1 to 8. It should be noted that the specific manner in which the processor 113 executes the aforementioned steps is based on the same concept as the method embodiments corresponding to Figures 1 to 8 in this application, and the resulting technical effects are the same as those in the method embodiments corresponding to Figures 1 to 8 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0190] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps performed by the first vehicle or training device in the methods described in the embodiments shown in Figures 1 to 8.

[0191] This application also provides a computer program product, which includes a program that, when run on a computer, causes the computer to perform the steps performed by the first vehicle or training device in the methods described in the embodiments shown in Figures 1 to 8.

[0192] This application also provides a circuit system including a processing circuit configured to perform the steps of the first vehicle or training device in the method described in the embodiments shown in Figures 1 to 8 above.

[0193] The execution device or information processing apparatus provided in this application embodiment can specifically be a chip. The chip includes a processing unit, such as a processor. Optionally, the chip also includes a communication unit, such as an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip to execute the methods described in the embodiments shown in Figures 1 to 8. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0194] The processor mentioned above can be a general-purpose central processing unit, microprocessor, GPU, NPU, TPU, ASIC, FPGA, or one or more integrated circuits used to control the execution of the program in the first aspect of the above method.

[0195] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0196] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0197] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0198] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. An information processing method, characterized in that, The method includes: Obtain initial environmental information around the vehicle; Category information is obtained based on the first environmental information. The category information includes the category of objects in a low-speed state. The objects in a low-speed state are objects whose speed is less than or equal to a first speed. The category indicates whether the object can be bypassed. The category is either the category of the object's low-speed state or the category of the reason that causes the object to be in a low-speed state.

2. The method according to claim 1, characterized in that, If the category is a category of the object's low-speed state, the category represents whether the object's low-speed state is long-term or short-term; or, If the category is a category of causes that lead to the object being in a low-speed state, the category represents that the low-speed state of the object is caused by the object or by the environment.

3. The method according to claim 1 or 2, characterized in that, The first environmental information is the environmental information at the current moment, and the step of obtaining category information based on the first environmental information includes: Based on the first environmental information, a first feature corresponding to the current moment is generated through a machine learning model; Based on the first feature and the second feature corresponding to each of the N historical moments, the category information is generated by the machine learning model, where N is an integer greater than or equal to 1, and the second feature is obtained based on the second environmental information of the historical moment.

4. The method according to claim 3, characterized in that, The category information includes the category of each object in at least one of the objects in the low-speed state. The generation of the category information using the machine learning model, based on the first feature and the second feature corresponding to each of the N historical moments, includes: Based on the first feature and the second feature corresponding to each of the N historical moments, a third feature is obtained for each of the at least one objects in a low-speed state. The third feature includes the features of the same object in a low-speed state at the current moment and at the historical moment. Based on the third feature of each of the at least one objects in a low-speed state, the machine learning model generates a category for each of the at least one objects in a low-speed state.

5. The method according to claim 3, characterized in that, The step of generating a first feature corresponding to the current moment based on the first environmental information through a machine learning model includes: The first environmental information and the second feature corresponding to each of the M historical moments are input into the machine learning model. The machine learning model extracts features from the first environmental information and the second feature corresponding to each of the M historical moments to obtain the first feature, where M is an integer greater than or equal to 1.

6. The method according to claim 1 or 2, characterized in that, The category information is used to obtain the vehicle's control information, which includes at least one of the following: the vehicle's planned location, the vehicle's driving strategy, or the vehicle's control information.

7. A method for training a model, characterized in that, The method includes: Obtain initial environmental information around the vehicle; Based on the first environmental information, predicted category information is obtained through a machine learning model. The predicted category information includes the predicted category of objects in a low-speed state, and the objects in a low-speed state are objects whose driving speed is less than or equal to the first speed. The machine learning model is trained using a loss function that indicates the similarity between the predicted category information and the expected category information. The expected category information includes the correct category of an object in a low-speed state. Both the predicted category and the correct category indicate whether the object can be bypassed. Either the predicted category and the correct category are categories of the object's low-speed state, or the predicted category and the correct category are categories of the reasons that cause the object to be in a low-speed state.

8. An information processing device, characterized in that, The device includes: The acquisition module is used to acquire the initial environmental information around the vehicle; The processing module is configured to obtain category information based on the first environmental information. The category information includes the category of objects in a low-speed state, wherein the objects in a low-speed state are objects whose speed is less than or equal to a first speed. The category indicates whether the object can be bypassed. The category is either the category of the object's low-speed state or the category of the cause that leads to the object being in a low-speed state.

9. The apparatus according to claim 8, characterized in that, If the category is a category of the object's low-speed state, the category represents whether the object's low-speed state is long-term or short-term; or, If the category is a category of causes that lead to the object being in a low-speed state, the category represents that the low-speed state of the object is caused by the object or by the environment.

10. The apparatus according to claim 8 or 9, characterized in that, The first environmental information is the environmental information at the current moment, and the processing module is specifically used for: Based on the first environmental information, a first feature corresponding to the current moment is generated through a machine learning model; Based on the first feature and the second feature corresponding to each of the N historical moments, the category information is generated by the machine learning model, where N is an integer greater than or equal to 8, and the second feature is obtained based on the second environmental information of the historical moment.

11. The apparatus according to claim 10, characterized in that, The category information includes the category of each of at least one of the objects in the low-speed state, and the processing module is specifically used for: Based on the first feature and the second feature corresponding to each of the N historical moments, a third feature is obtained for each of the at least one objects in a low-speed state. The third feature includes the features of the same object in a low-speed state at the current moment and at the historical moment. Based on the third feature of each of the at least one objects in a low-speed state, the machine learning model generates a category for each of the at least one objects in a low-speed state.

12. The apparatus according to claim 10, characterized in that, The processing module is specifically used for: The first environmental information and the second feature corresponding to each of the M historical moments are input into the machine learning model. The machine learning model extracts features from the first environmental information and the second feature corresponding to each of the M historical moments to obtain the first feature, where M is an integer greater than or equal to 8.

13. The apparatus according to claim 8 or 9, characterized in that, The category information is used to obtain the vehicle's control information, which includes at least one of the following: the vehicle's planned location, the vehicle's driving strategy, or the vehicle's control information.

14. A training device for a model, characterized in that, The device includes: The acquisition module is used to acquire the initial environmental information around the vehicle; The processing module is used to obtain predicted category information based on the first environmental information through a machine learning model. The predicted category information includes the predicted category of objects in a low-speed state, wherein the objects in a low-speed state are objects whose driving speed is less than or equal to the first speed. A training module is used to train the machine learning model using a loss function that indicates the similarity between the predicted category information and the expected category information. The expected category information includes the correct category of an object in a low-speed state. Both the predicted category and the correct category indicate whether the object can be bypassed. Either the predicted category and the correct category are categories of the object's low-speed state, or the predicted category and the correct category are categories of the reasons that cause the object to be in a low-speed state.

15. A device, characterized in that, The method includes a processor coupled to a memory storing program instructions, which, when executed by the processor, implement the method of any one of claims 1 to 7.

16. A vehicle, characterized in that, The method includes a processor coupled to a memory storing program instructions that, when executed by the processor, implement the method of any one of claims 1 to 6.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 7.

18. A computer program product, characterized in that, The computer program product includes a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 7.

19. A chip, characterized in that, The chip includes a processor for performing the steps of the method according to any one of claims 1 to 7.

Images