Autonomous driving system

Abstract

An autonomous driving system (600) for a vehicle, comprising: a prompt unit (608) configured to generate a natural language prompt based on at least one of: a user input and one or more driving parameters; and a language model unit (612) configured to use a machine learning language model to generate one or more driving commands and / or one or more textual data outputs based on an environmental signal corresponding to an environment of the vehicle and the natural language prompt.

Description

[0001] AUTONOMOUS DRIVING SYSTEM

[0002] Technical Field

[0003] This invention relates to autonomous vehicles, such as autonomous cars, trucks, buses, vans, and such like. Particularly, though not exclusively, the present invention is concerned with 'end-to-end' methods for autonomous driving.

[0004] Background Art

[0005] Modern scalable advanced driver-assistance systems and autonomous vehicles systems (collectively termed herein "autonomous driving systems"), whether they adopt an end-to-end approach using a single network, or a component-based configuration that combines learnable perception and motion planning modules, face common challenges. These systems often behave as 'black-boxes’ in the decision-making process, making it especially difficult to endow them with out-of-distribution (OOD) reasoning and interpretability capabilities.

[0006] A fundamental challenge with modern autonomous driving systems is the lack of interpretability in the decision-making process. Current autonomous driving systems often struggle with the interpretation and prediction of complex, real-world driving scenarios, especially for rare and unusual scenarios. Understanding why a decision is made is crucial for understanding areas of uncertainty, building trust, enabling effective human-AI collaboration, and ensuring safety.

[0007] The Applicant has therefore appreciated a need for improved systems and methods for autonomous driving that, inter alia, provide for enhanced explainability and, therefore, interpretability. In particular, the Applicant has appreciated a need to provide an improved architecture for autonomous driving systems and controllers which is not a 'black box', but rather one that can be inspected and interrogated by human operators. Such an arrangement would be technically advantageous for a number of reasons, including, but not limited to, improving the safety, reliability, and performance of autonomous vehicles. While regulatory and legal considerations may impose a need forexplainability, critically there is a technical imperative to being able to understand the behaviour of the technical system (the autonomous driving systems and the autonomous vehicle as a whole); explainability is therefore of high technical importance.In addition to the need for enhanced explainability in autonomous driving systems, the Applicant has further appreciated a need for improved adaptability / customisability of autonomous driving systems.

[0008] The current autonomous driving systems on the market are rule-based systems or based on elementary machine learning models. Rule-based systems are designed to react within a narrow set of predefined scenarios, utilising straightforward 'if-then' logic without the depth needed to navigate the complexities and variabilities inherent in real-world driving. The currently available approaches are usually limited in their adaptability because: (i) the underlying model usually requires significant work to re-train or adjust the 'if-else' / 'if-then' conditions to adapt to a new 'set of rules' (e.g. due to different driving regulations / laws that apply in a new or different jurisdiction); (ii) the models struggle to balance driving style, either compromising on efficiency or adopting an overly aggressive approach that requires the user to pay significant attention; and / or (iii) the user interaction with these systems is often manual and cumbersome.

[0009] In essence, despite making some progress to autonomous driving systems to satisfy user demands and expectations, current technologies do not yet achieve the level of intelligence, adaptability, and user-friendliness that fully satisfy user demands and expectations.

[0010] The Applicant has therefore appreciated the need for autonomous driving systems to be more easily adapted and influenced by human operators and other external inputs, dynamically and at varying levels of specificity, improving adaptability in varying situations and further enhancing communication (and therefore trust) between the human operator and the autonomous driving system.

[0011] The invention of the current application builds on the Applicant's previous work as disclosed in WO 2025 / 036885 A1 , which is herein incorporated by reference in its entirety.

[0012] Summary of the Invention

[0013] In accordance with a broadest aspect of the invention there is provided an autonomous driving system for a vehicle, the autonomous driving system comprising a language model unit configured to use a machine learning language model to generate one or more driving commands and / or one or more textual data outputs. The one or more driving commands and / or the one or more textual data outputs are generated by the language model unit / machine learning language model based on an environmental signal corresponding toan environment of the vehicle and a natural language prompt. The natural language prompt is based on at least one of: a user input or one or more driving parameters.

[0014] The language model unit may comprise the machine learning language model. Alternatively, the machine learning language model may be separate from the language model unit, e.g. stored on a separate / remote server, with the language model unit configured to be in (wireless) communication with the machine learning language model / remote server to thereby provide the necessary functionality. The one or more driving commands and / or one or more textual data outputs may be generated by the language model unit based on input of the environmental signal corresponding to an environment of the vehicle and the natural language prompt to the language model unit / machine learning language model.

[0015] The system may comprise a prompt unit configured to generate the natural language prompt based on (e.g., based on input of) the at least one of: the user input or the driving parameter(s). That is, the prompt unit may be configured to construct a natural language prompt based on (an aggregation of) one or more user inputs and / or one or more driving parameters. However, where the system does not comprise a prompt unit, the user input and / or the driving parameters may form the (whole) natural language prompt, i.e. the user input and / or the driving parameters may be directly received as input by the language model unit as the natural language prompt without any (textual) processing or modification.

[0016] Thus, in accordance with a first aspect of the invention (which may be considered as an embodiment of the broadest aspect of the invention) there is provided an autonomous driving system for a vehicle, comprising: a prompt unit configured to generate a natural language prompt based on at least one of: a user input or one or more driving parameters; and a language model unit configured to use a machine learning language model to generate one or more driving commands and / or one or more textual data outputs. The driving command(s) and / or the textual data output(s) are generated based on: an environmental signal corresponding to an environment of the vehicle, and the natural language prompt.

[0017] The language model unit is thus in communication with the prompt unit and, thereby, the language model unit is configured to receive from the prompt unit the generated natural language prompt.The autonomous driving system may comprise a controller configured to operate (control) the autonomous vehicle based on the said one or more driving commands. Accordingly, the controller may be in communication with the language model unit and, thereby, the controller is configured to receive from the language model unit the one or more driving commands.

[0018] In some examples, the one or more driving commands (e.g. the driving actions / instructions to be passed to a controller) may be, or comprise, textual data outputs. The controller may therefore be configured to receive driving commands in the form of textual data outputs. The one or more driving commands may be in another form of data (e.g. non-textual data).

[0019] Accordingly, the controller may be configured to receive driving commands in the form of another form of data (e.g. non-textual data).

[0020] The one or more textual data outputs (which may be, for example, vehicle intention and / or commentary) may be communicated to a user, e.g. via a user interface (discussed further below). However, the textual data output may not be provided to a user interface and / or relayed to the user. For example, the textual data output may be used as intermediate reasoning step in a 'Chain-of-Thought' reasoning of the machine learning language model and hence is not necessarily relayed to the user.

[0021] In some examples, the one or more driving commands may be generated first, and then may be converted to one or more textual data outputs. In other examples, the one or more driving commands may be derived / generated from (e.g. based on) the generated one or more textual data outputs.

[0022] The environmental signal corresponding to an environment of the vehicle may be received by the language model unit from a sensor unit. Accordingly, the language model unit may be in communication with the sensor unit and, thereby, the language model unit is configured to receive the environmental signal from the sensor unit.

[0023] The sensor unit may be configured to generat e / provide an environmental signal which corresponds to the environment surrounding the (autonomous) vehicle. The autonomous driving system may comprise the sensor unit, or the sensor unit may be a part of the vehicle. Those skilled in the art will appreciate that there are many different types of sensors, known in the art per se, that may be used on a vehicle to sense its surroundings. In some embodiments, the sensor unit comprises one or more sensors from a group comprising: a camera; a microphone; a sonar sensor; an ultrasonic sensor; a radar sensor; an ultra-wideband radio sensor; a lidar sensor; an optical flow sensor; a magnetometer; athermometer; a barometer; a moisture sensor; a humidity sensor; an inertial measurement unit; a gyroscope; an accelerometer; a touch sensor; a laser rangefinder; an infrared sensor; a vehicle speed sensor; a yaw rate sensor; a wheel speed sensor; a current sensor; a fuel level sensor; a voltage sensor; an altitude sensor; a tilt sensor; a Hall effect sensor; an odometer; and a geopositioning sensor. It will be appreciated that the sensor unit may comprise any combination of sensor types, and may comprise more than one of the same type of sensor, as appropriate. The term "geopositioning sensor" should be understood to include sensors for use with GNSS systems such as GPS, Gallileo, BeiDou, etc.; however those skilled in the art will appreciate that in the context of the present invention, sensors such as cameras may act as "geopositioning sensors" in that a machine learning model may be trained / configured to determine a location (e.g. a country) based on the data / environmental signal received from said sensors (e.g. signs, buildings, and / or vehicles).

[0024] The information about the surroundings of the autonomous vehicle (of which, the environmental signal is indicative) may be used to determine how the vehicle is to be controlled. There are many environmental variables that may be important for the autonomous driving system to consider when determining how to control the autonomous vehicle. In some embodiments, the environmental signal comprises information on: a number of detected vehicles; respective distances between detected vehicles and the autonomous vehicle; respective locations of detected vehicles; respective orientations of detected vehicles; respective speeds of detected vehicles; and / or respective bounding boxes of detected vehicles.

[0025] The environmental signal may comprise information on: a number of detected pedestrians; respective distances between detected pedestrians and the autonomous vehicle; respective locations of detected pedestrians; respective orientations of detected pedestrians; respective speeds of detected pedestrians; and / or respective bounding boxes of detected pedestrians.

[0026] The environmental signal may comprise information corresponding to: a number of detected cyclists; respective distances between detected cyclists and the autonomous vehicle; respective locations of detected cyclists; respective orientations of detected cyclists; respective speeds of detected cyclists; and / or respective bounding boxes of detected cyclists.

[0027] Additionally, or alternatively, the environmental signal may include information corresponding to the state of traffic signals to which the autonomous vehicle is subject. For example, the environmental signal may comprise information corresponding to: a detected traffic signal; respective distances between a detected traffic signal and the autonomous vehicle; arespective location of a detected traffic signal; and / or a respective state of a detected traffic signal. Such traffic signals may, for example, include traffic lights where the illumination state of different coloured lights indicates required or permitted behaviour (e.g. red meaning 'stop', green meaning 'go', and so on). Other 'dynamic' traffic signals that may be represented in the environmental signal include level crossing signs, smart motorway signs, and so on. The traffic signals may, additionally or alternatively, include 'static' traffic signals such as stop signs, give way markings, speed limit signs, road signage (e.g. to indicate road layout, lane allocations, etc.), warning signs, and so on.

[0028] The environmental signal may, for example, comprise camera images captured from one or more cameras positioned around the vehicle. The camera images may comprise a time sequence of images.

[0029] The environmental signal may be a live ('online' or 'real-time') environmental signal corresponding to a live ('real-time') environment of the vehicle, for example in an embodiment of the first aspect being implemented in a live, driving environment.

[0030] The environmental signal may be a historic environmental signal corresponding to a historic environment of the vehicle, for example in an embodiment of the first aspect being implemented in a test environment (e.g. 'offline').

[0031] In a simulated environment, the environmental signal corresponding to a surrounding environment of the autonomous vehicle may be a simulated environmental signal corresponding to a simulated / synthetic environment. The simulated environmental signal may comprise simulated images (i.e. synthetically generated images). For example, the simulated environmental signal may be raw RGB data / values of a simulated environment. In such a scenario, the simulated environmental signal may not be received from a sensor unit. This is since the signal is synthetically generated rather than sensed. However, the simulated environmental signal may comprise any or all of the information about the simulated or synthetic surroundings / variables of the autonomous vehicle, as above.

[0032] The environmental signal is not limited to comprising information from only a single sensor / single sensing modality. The environmental signal may comprise information (data) from a plurality of different types of sensor within the sensor unit and / or information (data) corresponding to a plurality of simulated sensing modalities (e.g. simulated image data, simulated lidar data, etc.).The environmental signal based on which the language model unit generates the one or more driving commands and / or the one or more textual data outputs may be a raw environmental signal. The raw environmental signal may be a signal received by the language model unit / machine learning language model directly from the sensor unit / from the simulated environment without any pre-processing. The raw environmental signal may be provided 'as is', i.e. in the exact format in which it is generated by the sensor unit / by the simulated environment to the language model unit.

[0033] Alternatively, the environmental signal based on which the language model unit generates the one or more driving commands and / or the one or more textual data outputs may be a processed environmental signal (i.e., the language model unit may generate the one or more driving commands and / or the one or more textual data outputs based on a (pre-) processed environmental signal). Producing the (pre-)processed environmental signal may comprise encoding the (raw) environmental signal, e.g. with a neural network. Accordingly, the autonomous driving system (e.g. the language model unit) may comprise an encoding unit (e.g. comprising a neural network) configured to receive the environmental signal and encode the environmental signal into an embedding (suitable) for input to the machine learning language model of the language model unit. The encoding may comprise encoding the environmental signal into tokens. The processed (in this example "encoded") environmental signal may itself be received by the machine learning language model.

[0034] In other words, the processed environmental signal may be a vector embedding / encoding that comprises a structured semantic representation of the surrounding environment of the vehicle. The term "structured semantic representation" refers to information which is organized, modelled, and encoded to capture the relationships, meanings, and attributes of the data in a way that is understandable and utilisable by computational systems like neural networks, and in particular language models (such as large language models (LLMs)). Thus, those skilled in the art will readily appreciate that the term "structured semantic representation" as used herein means a vector embedding / encoding in a format usable as a direct input to the language model. The content of that vector embedding / encoding corresponds to the information contained in the natural language prompt, environmental signal, operational parameters, and / or destination information (see below), as appropriate.

[0035] In an example, the encoding unit may comprise an image encoder, the image encoder being configured to encode an environmental signal in the form of a time sequence of images (e.g. captured by one or more cameras of the sensor unit) into a sequence of tokens. These image tokens may then be received by the language model unit.The language model unit may be configured to generate the one or more driving commands and / or one or more textual data outputs based on: the environmental signal corresponding to an environment of the vehicle, and a raw natural language prompt. That is, the language model unit / the language model of the language model unit may be configured to receive the natural language prompt 'as is' from the prompt unit in the exact (textual) format in which it is generated by the prompt unit without any pre-processing.

[0036] Alternatively, the natural language prompt that the language model unit is configured to generate the one or more driving commands and / or the one or more textual data outputs based on (in addition to the environmental signal) may be a (pre-)processed natural language prompt (i.e. processed after output from the prompt unit and prior to receipt at the language model unit / the language model). Producing the processed natural language prompt may comprise encoding the (raw) natural language prompt with a neural network. Accordingly, the system may comprise an encoding unit (which may be the same as the aforementioned encoding unit or distinct therefrom) that may comprise a textual encoder (e.g. a neural network), the textual encoder being configured to encode natural language (e.g. a natural language prompt) into a sequence of tokens. Accordingly, the textual encoder may be configured to encode the raw, 'as is' natural language prompt produced by the prompt unit into a sequence of tokens. The language model unit / language model may be configured to receive the sequence of tokens and base the generation of the one or more driving commands and / or the one or more textual data outputs on the sequence of tokens and the environmental signal.

[0037] The language model unit may be configured to generate the one or more driving commands and / or the one or more textual data outputs based on (in addition to the natural language prompt and the environmental signal) one or more operational parameters corresponding to operation of the vehicle and / or information relating to a destination of the vehicle.

[0038] The operational parameter(s) may relate to states and / or variables corresponding to the autonomous vehicle itself. For example, the operational parameters may relate to: a location of the autonomous vehicle; a speed of the autonomous vehicle; an orientation of the autonomous vehicle; a trajectory of the autonomous vehicle; a bounding box of the autonomous vehicle; one or more actuator positions of the autonomous vehicle; an accelerator state of the vehicle; and / or a brake state of the vehicle.

[0039] The information relating to a destination of the vehicle may be an end-destination and / or it may be a route (i.e. a planned path from the current or initial location of the autonomousvehicle to the intended end-destination). This may be the coordinates of the destination, a list of upcoming directions (e.g. next right turn, second exit at roundabout, straight on for 5 miles, etc.), a next direction, or a route of the autonomous vehicle.

[0040] As with the environmental signal, the operational parameter(s) and / or destination information may be live ('real-time') data corresponding to a live ('real-time') state of the vehicle, for example in an embodiment of the first aspect being implemented in a live, driving environment. However, in a test / simulated environment the operational parameter(s) may be / comprise historic or synthetic operational parameter(s), and the destination information may be / comprise historic or synthetic destination information.

[0041] The language model unit may be configured to generate the one or more driving commands and / or the one or more textual data outputs based on (in addition to the natural language prompt and the environmental signal) the operational parameter(s) and / or information relating to a destination of the vehicle in a 'raw' format. For example, the operational parameter(s) received at the language model unit / language model may be received from the autonomous vehicle 'as is', without any pre-processing.

[0042] Alternatively, the operational parameter(s) and / or information relating to a destination of the vehicle that the language model unit is configured to generate the one or more driving commands and / or the one or more textual data outputs based on (in addition to the natural language prompt and the environmental signal) may be processed operational parameter(s) and / or processed information relating to a destination of the vehicle. Accordingly, the system may comprise an encoding unit (which may be the same as the aforementioned encoding unit(s) or distinct therefrom) that may comprise a parameter encoder (e.g. a neural network) configured to encode operational parameter(s) and / or information relating to a destination of the vehicle. Accordingly, the parameter encoder may be configured to encode the raw, 'as is' operational parameter(s) and / or information relating to a destination of the vehicle. The language model unit / language model may be configured to receive the encoded operational parameter(s) and / or information relating to a destination of the vehicle and base the generation of the one or more driving commands and / or the one or more textual data outputs based on the encoded operational parameter(s) and / or information relating to a destination of the vehicle (in addition to the natural language prompt and the environmental signal from the sensor unit).

[0043] The processed operational parameter(s) and / or processed information relating to a destination of the vehicle may be provided in the form of a vector embedding(s) / encoding(s) that comprise a structured semantic representation of the one or more operational parameters and / or of theprocessed information relating to a destination of the vehicle. The embedding(s) may be the same or different to the above referenced embedding that comprises a structured semantic representation of the environment of the vehicle.

[0044] Alternatively, the encoded ( / processed) operational parameter(s) and / or encoded ( / processed) information relating to a destination of the vehicle may be used to generate textual representation of the operational parameter(s) and / or information relating to a destination of the vehicle, the textual representation being received by the language model unit. Producing the textual representation(s) may comprise inputting the encoded operational parameter(s) and / or encoded information relating to a destination of the vehicle to the prompt unit to generate the textual representation.

[0045] The driving parameters that are selected for generating the natural language prompt may be based on one or more operational parameters and / or information relating to a destination of the vehicle (e.g. the one or more driving parameters may be based on or comprise a list of driving directions).

[0046] Accordingly, the one or more driving parameters may be selected in accordance with (and the natural language prompt generated based on) data derived from at least one of the sensor unit, an operational parameter unit comprised as part of the vehicle / autonomous driving system, and a destination information unit comprised as part of the vehicle / autonomous driving system. Whilst the Applicant has appreciated that the vehicle may be driven based on only the environmental signal obtained from the sensor unit, the inclusion of the operational parameters and the destination information (and the natural language prompt generated based on the user input and / or the one or more driving parameters derived from at least one of the environmental signal, operational parameters, and destination information) is beneficial for enhanced functionality, adaptability and granular control of the vehicle.

[0047] As apparent from the preceding paragraph, the vehicle / autonomous driving system may comprise a destination information unit and / or an operational parameter unit. The destination information unit may produce / sense / provide the information relating to a destination or a route of the vehicle. The operational parameter unit may produce / sense / provide the operational parameter(s) and / or information and / or data relating to an operational state of the vehicle.Those skilled in the art will appreciate that a token is a parsed segment or unit of data (e.g. text), where each token is selected from a 'vocabulary' of possible tokens - i.e. there is a predetermined, finite list of tokens available. A vector is a numerical representation often used to encapsulate information about a token, sequence of tokens, or other types of data. However, not all data needs to be 'split up' into different tokens. Those skilled in the art will appreciate that the tokenisation (or not) of data is dependent on the specific type of data and its nature - tokenisation may be technically advantageous for certain types of data.

[0048] The encoder unit(s) (e.g. the pre-processing unit(s)) may comprise any suitable components for producing embeddings suitable for the machine learning language model of the language model unit. For example, the (image encoder of the) encoder unit(s) may comprise a transformer (network) configured to encode a time sequence of images captured by one or more cameras (i.e. an exemplary environmental signal) into embeddings, which then may be projected into the language model latent space, e.g. using a linear projection layer.

[0049] Those skilled in the art of deep learning will appreciate that a latent space is, in effect, a multidimensional space in which meaningful representations of 'real world' events or entities (or simulated world events or entities) are encoded. These encoded latent embeddings are typically a compressed representation of the real (or simulated) world, where the compression provides for a lower dimension representation of the real (or simulated) world (which typically exists at higher dimensionality).

[0050] There are several language model / language model unit architectures that could be used in accordance with embodiments of the present invention. For example, the language model / language model unit may be / comprise: a language model transformer. The language model transformer may be a multi-modal large language model (MMLLM). The MMLLM may be, for example, a vision-language-action-model (VLAM) adept at processing both visual and textual inputs to generate coherent textual data outputs and actionable driving commands. The language model transformer may be configured to use a respective attention mechanism to generate the one or more driving commands and / or textual data outputs.

[0051] The language model may be frozen. Those skilled in the art will appreciate that the term 'frozen' should be understood to mean that the weights used by the language model are not updated during any 'fine-tuning' (a step that conventionally takes place downstream / after initial training), i.e. the internal weights of the language model are fixed at particular values after the initial training of the language model, but are not changed during any later fine-tuning process.The attention mechanism that may be used by the transformer may be dependent on a query matrix (Q), a key matrix (K), and a value matrix (V). In some further such embodiments, the language model unit further comprises a Low-Rank Adaptation (LoRA) module configured to supply the query matrix (Q), the key matrix (K), and the value matrix (V). In a set of such embodiments, one or more of said matrices are trainable. Such an arrangement may advantageously allow a frozen LLM to be 'fine-tuned' by the LoRA.

[0052] Language models such as LLMs can learn knowledge on their own, however applying such knowledge to real world problems typically requires a fine-tuning process. Fine-tuning involves, following pre-training the language model using a sufficiently large training set, then adapting that language model to some real-world task. Language models may be fine-tuned by providing the model with a prompt and an answer (often with additional context) and the model is then trained to generate an answer like the answer originally provided for training.

[0053] The user input may be a driving instruction from the user. For example, the user input may be an instruction to: turn, accelerate, brake, slow down, stop, pull over, change lane, take an exit, park, reverse, manoeuvre, re-position, indicate, park, merge, yield / give way, adjust speed, use headlights, use hazard lights, change gear, start the engine, stop the engine, take an alternative route to a planned route, change destination, travel to a specified location, overtake etc. The driving instruction may comprise any combination of these individual driving instructions.

[0054] The user input may be driving feedback from the operator / user (e.g. feedback based on the driving performance of the autonomous driving system / vehicle). For example, the user input may include feedback such as "you should not have overtaken that car" or "that was not a safe place to pull over". The driving feedback may comprise any combination of individual driving feedback. Future one or more driving commands and / or one or more textual data outputs may be responsive, correspond and / or account for the driving feedback from the user (i.e. the autonomous driving system, and more specifically the machine learning language model, may 'learn' from the user feedback).

[0055] The one or more driving commands and / or the one or more textual data outputs may be responsive and / or correspond to the driving instruction from the user (i.e. the user input). For example, where the instruction is to "turn", the language model unit may be configured to generate a command to "turn the vehicle". This "turn the vehicle" command may in turn beprovided to the controller such that the controller operates the autonomous vehicle to "turn". As another example, where the instruction is to "slow down" the language model unit may be configured to generate a command to "slow down the vehicle". This "slow down the vehicle" command may in turn be provided to the controller such that the controller operates the autonomous vehicle to "slow down". This response / correspondence of the driving command relative to the driving instruction can be replicated for any driving instruction.

[0056] In addition to or instead of a responsive driving command, the one or more textual data outputs may be responsive and / or correspond to the driving instruction from the user. For example, a textual data output to confirm to the user that the driving instruction is being actioned may be generated by the language model unit, which may or may not occur in parallel to generation of the command responsive / correspondent to the driving instruction. As another example, a textual data output to advise the user that the driving instruction is not being actioned may be generated by the language model unit. This may be generated where the language model unit determines it is not safe or appropriate to operate based upon the user instruction.

[0057] As can be deduced from the above, any driving command and / or textual data output that is generated responsive and / or correspondent to a driving instruction from the user may additionally be generated based on the environmental signal, the operational parameter(s) and / or information relating to a destination of the vehicle. This can ensure that the driving command and / or textual data output produced accounts for relevant environmental, operational and destination factors that might be relevant to the driving command and / or textual data output.

[0058] The user input may be a query from the user. The query may relate to driving operation or performance of the autonomous driving system / vehicle. The one or more textual data outputs configured to be generated by the language model unit may be responsive to the query from the user. The Applicant has appreciated that the use of language models to provide for a response to a user query advantageously allows for provision of reasoning used by the autonomous driving system for its decisions, providing confidence that the system not only makes the correct decision, but also that it is doing so for the correct reason. This is technically advantageous in terms of debugging and troubleshooting the autonomous driving system - a human operator (such as an engineer or technician) can probe the technical workings of the autonomous driving system with a query and obtain a response, rather than needing to analyse the system as a black box.The user input (and in some cases, as explained below, the one or more driving parameters) may be provided / obtained via a user interface unit in communication with the prompt unit. Accordingly, the autonomous driving system / vehicle may comprise a user interface unit in communication with the prompt unit. The user interface unit may comprise one or more of a graphical user interface, a touch user interface, a voice user interface (e.g., comprising a microphone), a gesture-based interface, a natural language user interface, a menu-driven interface, etc. The user interface unit is configured to convert the user input to a textual user input for providing / communicating to the prompt unit. For example, wherein the user interface unit comprises a voice user interface (e.g. comprising a microphone), the user interface unit may be configured to, or comprise a real-time speech recognition unit configured to, convert audio user input received via the voice user interface to a textual user input.

[0059] The prompt unit may be a distinct component / unit separate from the language model unit, or it may be integrated with / a component part of the language model unit. In some examples, the language model unit comprises the prompt unit (both units being configured to use the same machine learning language model).

[0060] The user interface, or a distinct / different user interface, may be in communication with the language model unit. The (distinct) user interface may therefore provide the generated textual data output to the user.

[0061] The textual data output need not be provided to the user. The textual data output may constitute an intermediate reasoning step in a 'Chain-of-Thought' prompting of the machine learning language model.

[0062] The one or more driving parameters may be based on driving laws, rules, and / or guidance (collectively termed "regulations"). For example, and with respect to the UK, the driving parameters may be based (at least in part) on the Highway Code. Alternatively, the driving parameters may comprise natural language that the machine learning language model / language model unit has been configured / trained to understand / associate with particular driving laws, rules, and / or guidance. For example, where the one or more driving parameters correspond to or comprise the natural language "You are driving in the UK", the machine learning language model / language model unit inherently understands that any generated driving commands require compliance with (learnt) UK driving / Highway Code.Additionally and / or alternatively the one or more driving parameters may be based on preset driving modes (e.g. that can be characterized as "cautious", "mild" or "assertive"), said preset driving mode being selected by a user. The preset driving mode may be one that has been developed based on the user's manual driving style. The preset driving mode may include / be associated with instructions that relate to / influence the generated one or more textual data outputs (as well as the one or more driving commands). For example, the "cautious" driving mode may comprise or be associated with increased frequency of commentary / textual outputs, so that the user is reassured / informed of the autonomous driving system's (intended) actions.

[0063] Thus, the natural language prompt may, in addition to or instead of a user input, be generated based on driving regulations and / or a preset driving mode. This can ensure any resultant driving command / textual data output can be generated in the context of driving regulations and / or a preset driving mode. Hence, the resultant operation of the vehicle can be carried out in accordance with appropriate regulations and equally in a manner that is in accordance with general desired driving performance.

[0064] Such adherence to regulation / desired performance may be achieved in conjunction with operating according to user input. For example, where the user provides a driving instruction (e.g. to "overtake") the language model unit can generate a driving command which, when executed, satisfies both the user's selected driving mode and the driving regulations.

[0065] Equally, where operating according to user input would result in a contravention of driving regulations and / or a preset driving mode, the language model unit may not generate a driving command in accordance with the user input, but instead may generate a textual data output that (e.g. when relayed to the user via a user interface unit or the like) explains to the user that operation in accordance with the user input is prohibited due to, e.g., contravening driving regulations.

[0066] The one or more driving parameters that are selected for generating the natural language prompt may be selected in accordance with data derived from the environmental signal, operational parameter(s), and / or destination information. For example, the one or more driving parameters (e.g. driving regulations) selected may be selected based on a location determined from the environmental signal (received from, for example, the geopositioning sensor and / or one or more video cameras of the sensor unit). This can ensure appropriate local driving regulations are being adhered to.Similarly, the preset driving mode (e.g. "night", "light rain", "snow", "fog", "filtering", "lane closure") may be selected in accordance with at least one of a time of day, weather condition, and traffic condition determined based on the environmental signal (received from, for example, one or more of a camera; a radar sensor; an ultra-wideband radio sensor; a lidar sensor; an optical flow sensor; a magnetometer; a thermometer; a barometer; a moisture sensor; and a humidity sensor of the sensor unit).

[0067] Thus, the prompt unit may be configured to generate a natural language prompt based on a combination (e.g. aggregation) of one or more user prompts (user input) and one or more system prompts (driving parameters), wherein the one or more driving parameters / system prompts may be generated based on at least one of the environmental signal, the operational parameter(s), and the destination information.

[0068] For example, the prompt unit may generate a natural language prompt that describes the present scenario such as:

[0069] "Currently driving in the UK and approaching densely populated urban area; reducing speed to comply with local 30 km / h speed limit. Weather conditions are rainy, activating increased traction control and adjusting headlights, driving carefully and mild. Preparing to execute a gentle right turn at upcoming intersection as user requested. "

[0070] based on the user prompt "Slow down and turn right at the next intersection" and system prompts "In the UK, in a 30 mph speed limit zone" and "Light rain" based on the environmental signals received from a geopositioning sensor and / or one or more video cameras of the sensor unit (or one or more of: a radar sensor; an ultra-wideband radio sensor; a lidar sensor; an optical flow sensor; a magnetometer; a thermometer; a barometer; a moisture sensor; and a humidity sensor).

[0071] In accordance with a second aspect of the invention, there is provided a vehicle (e.g. car, lorry, van, truck, bus, etc.) comprising an autonomous driving system in accordance with the first aspect of the invention. Optionally, the autonomous driving system is in accordance with any optional form of the first aspect of the invention as defined above.

[0072] In accordance with a third aspect of the invention, there is provided an autonomous driving system in accordance with the first aspect of the invention in combination with a vehicle (e.g.car, lorry, van, truck, bus, etc.). The autonomous driving system may be in accordance with any optional form of the first aspect of the invention as set out above.

[0073] In the third aspect of the invention, the autonomous driving system need not be comprised as part of the autonomous vehicle. For example, some or all component parts of the autonomous driving system (e.g. the language model unit, the prompt unit, the encoding unit, etc.) may be situated at a remote location from the vehicle but in communication therewith to thereby provide necessary functionality for operation of the vehicle. Some or all of the component parts of the autonomous driving system (e.g. the sensor unit, the controller, etc.) may be provided in combination with / situated aboard the vehicle.

[0074] In a fourth aspect of the invention, there is provided a method of operating an autonomous driving system for a vehicle (e.g. an autonomous vehicle), the autonomous driving system comprising a language model unit configured to use a machine learning language model to generate one or more driving commands and / or one or more textual data outputs. The method comprises generating, with the machine learning model of the language model unit, one or more driving commands and / or one or more textual data outputs based on (input of) an environmental signal corresponding to an environment of the vehicle and a natural language prompt (to the language model), the natural language prompt being based on one of: a user input and one or more driving parameters.

[0075] The method of the fourth aspect may comprise generating the natural language prompt based on at least one of a user input or one or more driving parameters. This may comprise using a prompt unit (comprised as part of the autonomous driving system) to generate the natural language prompt based on the one of a user input and one or more driving parameters.

[0076] It will be appreciated that the method of the fourth aspect corresponds to a method of operating the autonomous driving system for an (autonomous) vehicle in accordance with the broadest aspect of the invention. The fourth aspect of the invention may further extend to a method of operating an autonomous driving system for an (autonomous) vehicle in accordance with any optional form of the broadest aspect of the invention, which includes a method of operating an autonomous driving system for an (autonomous) vehicle in accordance with the first aspect of the invention (and, optionally, any optional forms thereof).

[0077] In accordance with a fifth aspect of the invention, there is provided a computer-implemented method of operating (and / or controlling) a vehicle using the autonomous driving system ofthe broadest aspect of the invention. The method of the fifth aspect may comprise the method of the fourth aspect (optionally in any optional forms thereof). In other words, the computer-implemented method (of the fifth aspect) of operating a vehicle may comprise using an autonomous driving system according to the broadest aspect (optionally in any optional forms thereof, such as in the form of the first aspect) to operate the vehicle.

[0078] In the fourth or fifth aspects of the invention, the method may comprise: operating the vehicle based on the generated one or more driving commands, e.g. by / using a controller of the autonomous driving system / vehicle. That is, the method of the fourth or fifth aspects may comprise communicating / transmitting / inputting the one or more driving commands to a controller, the controller being configured to receive the one or more driving commands and control the autonomous vehicle accordingly.

[0079] In the fourth or fifth aspects of the invention, the method may comprise: communicating, via a user interface unit, the one or more textual outputs to a user. That is, the method of the fourth or fifth aspects may comprise communicating / transmitting / inputting the one or more textual outputs to a user interface unit (as described above) configured to provide / communicate the one or more textual outputs to a user.

[0080] In the fourth or fifth aspects of the invention, the method may comprise: obtaining / receiving the environmental signal from a sensor unit of the autonomous driving system / vehicle, e.g. in a live, driving environment. Alternatively, the method may comprise: obtaining / receiving a historic environmental signal or a simulated environmental signal, e.g. in a test and / or simulated / synthetic environment.

[0081] In the fourth or fifth aspects of the invention, the method may comprise: obtaining / receiving the user input by / at the prompt unit, e.g. via a user interface unit (of the autonomous driving system / vehicle). That is, the method may comprise obtaining / receiving the user input at the user interface unit, and communicating (providing / inputting) the user input to the prompt unit.

[0082] In the fourth or fifth aspects of the invention, the method may comprise: obtaining / receiving the one or more driving parameters by / at the prompt unit. The prompt unit may obtain / derive the one or more driving parameters based on input including at least one of: user input (e.g. received from the user interface unit, such as a selection / indication of a preset driving mode), the environmental signal (e.g. received from the sensor unit, such as information relating to / indicative of a weather condition), one or more operational parameters (e.g. received from the operational parameter unit, such as a current speed of the autonomousvehicle), and destination information (e.g. received from the destination information unit, such as a direction of the planned route of the autonomous vehicle).

[0083] In accordance with a sixth aspect of the invention, there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of the fourth or fifth aspects of the invention, optionally in any other optional forms thereof.

[0084] In accordance with a seventh aspect of the invention, there is provided a transitory, or non-transitory, computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of the fourth or fifth aspects of the invention, optionally in any other optional forms thereof.

[0085] Thus, it will be appreciated that the aspects of the invention provide advantageous methods and systems that can be used for controlling, and dynamically tailoring / adapting the control of, autonomous vehicles by using a machine learning language model (e.g. via the utilisation of natural language prompts). The aspects of the invention address the pressing challenge of creating a highly adaptable, deeply personalized autonomous driving experience, one that can seamlessly adjust (optionally in real-time) to the unique preferences of each user, the specific demands of varying situations, and the local driving regulations and laws.

[0086] Current systems are frequently constrained by their Operational Design Domains (ODDs), exhibiting a lack of flexibility that renders them overly rigid and often not user-friendly outside of their initial programming parameters. These systems struggle to accommodate the nuanced differences encountered across geographic locations or to adapt to the evolving needs and expectations of their users. Moreover, existing solutions fail to offer a truly interactive platform that enables users to communicate their preferences and requirements in an intuitive and effective manner. The Applicant has appreciated that language models - particularly Large Language Models (LLMs) - show remarkable capabilities in terms of reasoning and generic situation understanding. The remarkable abilities of large language models (LLMs) demonstrate early signs of artificial general intelligence (AGI), exhibiting capabilities such as out-of-distribution (OOD) reasoning, common sense understanding, knowledge retrieval, and the ability to naturally communicate these aspects with humans. These capabilities align well with the focus areas of autonomous driving and robotics in general.

[0087] Aspects of the present invention offer an innovative approach to dynamically process and react to real-world driving scenarios (optionally in real-time), including rare and complexsituations, by leveraging the general-purpose reasoning capability learned by language models such as LLMs, significantly enhancing the driving system's capability.

[0088] As above, the obtained / received / generated environmental signal may be 'live' (real-time) sensor data, i.e. current sensor data of an active vehicle - sensor data representative or indicative of the instantaneous state of the environment of the vehicle at the point at which it is sensed (i.e. a 'snapshot' of the instantaneous environment of the vehicle, absent any information of the future development of the environment). In this context, the environmental signal(s) may be "live" sensor data points corresponding to the immediate / instantaneous present (e.g., t=Os) but may also include some past sensor data points (e.g., t<Os). In one example, "live" sensor data may be (the) sensor data points in the range t=-1 ,0s to t=Os. In another example, "live" sensor data may be (the) sensor data points in the range t=-0.5s to t=Os. However, the obtained / received / generated environmental signal may be historical (e.g. corresponding to a real-world driving environment or scenario that has previously been experienced / encountered by an active vehicle) or simulated (e.g. synthetically / artificially generated environmental data / signals indicative of a simulated / synthetic driving environment or scenario).

[0089] Thus, it will be appreciated that aspects of the invention as described herein relate to autonomous driving systems (and methods of operating them) that comprise / leverage machine learning language models, such as LLMs. These language models may be capable of operating sufficiently quickly to be able to respond to real-world situations in realtime, thereby meeting the latency requirements for driving on public roads, though in certain use cases of the invention (e.g., in simulated / test environments and where the aspects of the invention are not being utilised in a live driving environment), such latency requirements are not necessitated.

[0090] It will further be appreciated that the optional features described hereinabove in respect of embodiments of the first aspect of the invention apply equally, where technically appropriate, to the other aspects of the invention outlined herein.

[0091] Where technically appropriate, embodiments of the invention may be combined. Embodiments are described herein as comprising certain features / elements. The disclosure also extends to separate embodiments consisting or consisting essentially of said features / elements.

[0092] Technical references such as patents and applications are incorporated herein by reference.Any embodiments specifically and explicitly recited herein may form the basis of a disclaimer either alone or in combination with one or more further embodiments.

[0093] In the context of this specification "comprising" is to be interpreted as "including". Aspects of the invention comprising certain elements are also intended to extend to alternative embodiments "consisting of' or "consisting essentially" of the relevant elements.

[0094] The term "vehicle" as used herein should be understood to mean any kind of vehicle intended or able to travel on roads (or equivalent) where some mechanical and / or electrical propulsion is used to drive the vehicle, whether operated autonomously or not. This includes, but is not limited to: cars, motorbikes, trucks, buses, coaches, vans, lorries, campervans, motor caravans, minibuses, limousines, all-terrain vehicles (ATVs), tractors, and other such vehicles that are mechanically or electrically driven.

[0095] Where context allows (e.g. in respect of other vehicles detected by the autonomous vehicle), the term "vehicle" further extends to non-driven vehicles, i.e. those without mechanical or electrical propulsion. This includes, but is not limited to: bicycles, unicycles, tricycles, quadracycles, rickshaws, carts, wagons, horse-drawn carts or carriages, and other such vehicles that are not mechanically or electrically driven.

[0096] The terms “data” is used in different contexts herein to refer to digital information, such as that represented by known bit structures within one or more programming languages. In use, data may refer to digital information that is stored as bit sequences within computer memory.

[0097] Certain machine learning models may operate on structured arrays of data of a predefined bit format. Using terms of the art, these may be referred to as "vectors", as used herein. Those skilled in the art will appreciate that a vector is a type of tensor and is a term often used for a first rank or 1 D tensor. Conversely, a tensor may, in some cases, be understood to be a collection of vectors. For the avoidance of doubt, the terms "vector" and "tensor" can be used interchangeably where technical context allows.

[0098] It should be noted that for machine learning methods multidimensional arrays, e.g. with a defined extent in multiple dimensions, may be converted to different forms for representation (e.g., within memory) as a sequence or vector of values stored according to the predefined format (e.g., n-bit integer or floating point number, signed or unsigned). For example, images taken from a camera are typically a 3D tensor (typically containing RGB values), which can be converted into a vector, and potentially tokenised as outlined above. Large arrays such asimages may also be split up into multiple 'patches', which may be operated on separately to ease processing and / or memory constraints (e.g. an image of size 1000 pixels by 1000 pixels could be split into patches of size 100 pixels by 100 pixels). Such patches can then be merged back together after processing to combine the results, if necessary.

[0099] Brief Description of the Drawings

[0100] Certain embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0101] Fig. 1 is a block diagram illustrating the architecture for driving with LLMs;

[0102] Fig. 2 is a further block diagram illustrating the architecture for driving with LLMs; Fig. 3 is a block diagram of the architecture of the driving LLM agent model;

[0103] Fig. 4 is a block diagram that provides an illustration of the Driving QA Dataset automatic labelling process;

[0104] Fig. 5 is a schematic diagram illustrating a demonstration of the interaction with the pretrained driving LLM agent;

[0105] Fig. 6 is a block diagram illustrating the architecture of an autonomous driving system for a vehicle;

[0106] Fig. 7 is a block diagram illustrating a method for operating a vehicle; and

[0107] Figs. 8A and 8B illustrate a specific embodiment of an autonomous driving system for a vehicle.

[0108] Detailed Description

[0109] Certain exemplary embodiments are described herein which relate to an autonomous driving system for a vehicle, an autonomous vehicle, and computer-implemented methods of operating vehicles. It will be appreciated that the autonomous vehicle and its various components - as well as systems and components it interacts with - are complex technical systems, and so the illustrations and descriptions provided herein are simplified for ease of reference.

[0110] Interpretability of Autonomous Driving Systems

[0111] A variety of explainability methods have been introduced to understand the underlying decision process of deep neural networks. For example, there are well-established modelagnostic interpretability methods that generate explanations for individual predictions. Other methods such as gradient based, saliency maps, and attention maps target the inner operations of models to explain the decision-making process.In the field of autonomous vehicles, visual attention maps which highlight causally influential regions in driving images have been proposed which combine attention-based methods with natural language into an attention-based vehicle controller to provide natural language action descriptions and explanations based on a series of image frames. Such an architecture may be improved by integrating part of speech prediction and special token penalties.

[0112] Efforts have also been made to combine this with other explanatory methods. For example, attempts have been made to explain transformers by leveraging attentive class activation tokens, encoded features, their gradients, and their attention weights all together.

[0113] Embodiments of the present invention use text modality for improved control and explainability in autonomous driving.

[0114] Multi-modal LLMs in Driving Tasks

[0115] In recent times, there has been a notable trend towards integrating multiple modalities into unified large-scale models. Notable examples include VLMs which effectively combine language and images to accomplish tasks like image captioning, visual question answering, and image-text similarity. Another noteworthy advancement involves the fusion of information from six distinct modalities: text, image / video, audio, depth, thermal, and inertial measurements. This exciting development expands the possibilities for generating content using diverse data input and output types, while also enabling broader multi-modal search capabilities. With camera sensors being one of the most common sensors used in autonomous driving, a natural step to incorporate language has been through VLMs. For example, some approaches use images and language directions to train a driving policy.

[0116] The system learns to summarize its visual observations in natural language, generate an appropriate action response (e.g. “I see a pedestrian crossing, so I stop”), and generate the controls (i.e. driving commands), accordingly. The generation of the appropriate (textual) action response may or may not occur in parallel to the generation of the driving command.

[0117] The Applicant has appreciated a novel approach in the domain of autonomous vehicles in which numeric vector modality is fused with language.

[0118] Architecture OverviewFig. 1 is a block diagram illustrating the architecture for driving with LLMs in accordance with an embodiment of the present invention. Fig. 1 shows various components of an autonomous vehicle 100. These include: a sensor unit 102; an operational parameter unit 104; a destination information unit 106; a vector generator unit 108; a pre-processing unit 110; a language model unit 112; a controller 114; and a user interface unit 116. As outlined above, this is necessarily a simplification of a practical autonomous vehicle, which may have many other components not shown or described here for brevity.

[0119] The vector generator unit 108, pre-processing unit 110, language model unit 112, and controller 114 are referred to collectively as a control system 118. Some or all the components of the control system 118 may be embodied within a single device or processor. Alternatively, some or all the components of the control system 118 may be separate, discrete devices or processors that are physically distinct. However, for ease of reference, these are grouped logically as a 'control system' 118.

[0120] The sensor unit 102 is configured to generate an environmental signal 120 corresponding to a surrounding environment of the autonomous vehicle 100. There are many different types of sensor, known in the art perse, that may be used within the sensor unit 102, as appropriate. It will be appreciated that where multiple sensors are used, some of all of these may be physically distinct from one another, however for ease of reference they are referred to collectively as a 'unit' herein, even where multiple separate sensor devices are used. The sensor unit 102 may include one or more of: a camera; a microphone; a sonar sensor; an ultrasonic sensor; a radar sensor; an ultra-wideband radio sensor; a lidar sensor; an optical flow sensor; a magnetometer; a thermometer; a barometer; a moisture sensor; a humidity sensor; an inertial measurement unit; a gyroscope; an accelerometer; a touch sensor; a laser rangefinder; an infrared sensor; a vehicle speed sensor; a yaw rate sensor; a wheel speed sensor; a current sensor; a fuel level sensor; a voltage sensor; an altitude sensor; a tilt sensor; a Hall effect sensor; an odometer; and a geopositioning sensor.

[0121] The sensor unit 102 is used to determine the state of the world around the autonomous vehicle 100. This may include detecting entities and objects such as other vehicles, pedestrians, cyclists, traffic signals, road obstructions, and so forth, as well as the state of such detected entities and objects.

[0122] The environmental signals generated by the sensor unit 102 may include information relation to, by way of non- limiting example:

[0123] • a number of detected vehicles• respective distances between detected vehicles and the autonomous vehicle

[0124] • respective locations of detected vehicles

[0125] • respective orientations of detected vehicles

[0126] • respective speeds of detected vehicles

[0127] • respective bounding boxes of detected vehicles

[0128] • a number of detected pedestrians

[0129] • respective distances between detected pedestrians and the autonomous vehicle • respective locations of detected pedestrians

[0130] • respective orientations of detected pedestrians

[0131] • respective speeds of detected pedestrians

[0132] • respective bounding boxes of detected pedestrians

[0133] • a number of detected cyclists

[0134] • respective distances between detected cyclists and the autonomous vehicle

[0135] • respective locations of detected cyclists

[0136] • respective orientations of detected cyclists

[0137] • respective speeds of detected cyclists

[0138] • respective bounding boxes of detected cyclists

[0139] • a detected traffic signal

[0140] • respective distances between a detected traffic signal and the autonomous vehicle • a respective location of a detected traffic signal

[0141] • a respective state of a detected traffic signal

[0142] Such traffic signals may, for example, include traffic lights where the illumination state of different coloured lights indicates required or permitted behaviour (e.g. red meaning 'stop', green meaning 'go', and so on). These may also include 'static' traffic signals such as stop signs, give way markings, speed limit signs, road signage (e.g. to indicate road layout, lane allocations, etc.), warning signs, level crossing signs, smart motorway signs, and so forth.

[0143] The operational parameter unit 104 provides information or data relating to the operational state of the autonomous vehicle (i.e. the 'ego' vehicle). The operational parameters provided by the operational parameter unit 104 may include information relation to, by way of nonlimiting example:

[0144] • a location of the autonomous vehicle

[0145] • a speed of the autonomous vehicle

[0146] • an orientation of the autonomous vehicle

[0147] • a trajectory of the autonomous vehicle

[0148] • a bounding box of the autonomous vehicleone or more actuator positions of the autonomous vehicle

[0149] an accelerator state of the vehicle

[0150] a brake state of the vehicle

[0151] The destination information unit 106 provides information relating to the destination or route of the vehicle. This may be the coordinates of the destination, a list of upcoming directions (e.g. next right turn, second exit at roundabout, straight on for 5 miles, etc.), a next direction, or a route of the autonomous vehicle.

[0152] While in this embodiment, the control system 118 of the autonomous vehicle 100 makes use of the operational parameters and destination information, these may be omitted. The Applicant has appreciated that the autonomous vehicle may drive based on only the environmental signal obtained from the sensor unit 102, though the inclusion of the operational parameters and destination information is beneficial for enhanced functionality.

[0153] The vector generator unit 108 is configured to generate a numeric vector 122 from: the environmental signal 120 received from the sensor unit 102, the operational parameters received from the operational parameter unit 104, and the destination information received from the destination information unit 106. Each numeric vector is an array of numerical values, which correspond to the information of the environmental signal, operational parameters, and destination information, respectively. There may be more than one vector created from any one of the environmental signal, operational parameters, and / or destination information. For example, separate vectors may be generated from the environmental signal corresponding to detected vehicles (as one vector) and detected pedestrians (as another vector).

[0154] The pre-processing unit 110 receives as an input the numeric vector 122 generated by the vector generator unit 108. The pre-processing unit 110 generates a vector embedding 124 from the numeric vector 122. The vector embeddings 124 comprises a structured semantic representation of the surrounding environment of the autonomous vehicle 100.

[0155] The language model unit configured to use a machine learning language model to generate output 126, which may include one or more textual outputs and / or one or more driving commands, based on the vector embedding. These output 126 may include a semantic description of what the control system 118 is observing, what actions it has planned, and the reasoning behind its decisions.The controller 114 is configured to receive the one or more driving commands within the output 126 of the LMU 112 and operate the autonomous vehicle based on said one or more driving commands, by supplying suitable signals 128 to the drive system 130 of the autonomous vehicle. This drive system 130 may be a suitable drive-by-wire system, the details of which are not outlined herein for sake of brevity. Those skilled in the art will appreciate that a drive-by-wire system is an arrangement in which driving functions such as acceleration, braking, and steering are controlled electronically.

[0156] Fig. 2 is a further block diagram illustrating the architecture for driving with LLMs. As can be seen in Fig. 2, the model is built on three key components: the vector encoder 202, vector former 204, and a frozen LLM 206 with a Low-Rank Adaptation (LoRA) module 208. The internal architecture of these components can be seen in more detail Fig. 3. This architecture is described below with reference to Figs. 1 through 3. For ease of illustration, some of the higher-level components described above with reference to Fig. 1 are omitted from Figs. 2 and 3.

[0157] The first component is the vector encoder 202. The four input vectors 122 initially pass through the Multilayer Perceptron (MLP) layers 302 (there are MLPs for each vector in this embodiment). A cross-attention layer 304 is employed to encode the vector feature into a latent space, producing a vector latent 306. At this stage, the ego feature is added to each learned input latent to accentuate the ego states.

[0158] The vector encoder 202 receives a number of latents 313 - in this particular embodiment there are thirty-two latents input to the cross-attention layer 304. Each of these latents 313 and the outputs of the MLPs 302 are arguments to the cross-attention layer 304. In other words, they are multiplied via a cross-attention operation, in a manner known in the art per se.

[0159] The second component is the vector former 204. This module comprises self-attention layers 308 and a cross-attention layer 310. There may, by way of example only, be multiple selfattention layers 308, for example seven such layers. Their efficient learning mechanisms enable the transformation of the vector latent 306 into an embedding 312 that the LLM 206 can decode.

[0160] The vector former 204 also receives a number of queries 314 - in this particular embodiment there are sixty-four queries input to the cross-attention layer 310 of the vector former 204.Thus the outputs of the self-attention layer 308 and the queries 314 are arguments to the cross-attention layer 310, and thus are multiplied via a cross-attention operation.

[0161] The third component is the frozen LLM 206, which receives the vector embedding 312 from the vector former 204, as well as a prompt embedding 316. The prompt embedding 316 is an embedding of the prompt given to the system, which might be a request for driving commands or a request for an explanation of what the system will do and why, for a given world state observed by the system from the environmental signals. The prompt and its embedding may be generated automatically and / or the prompt and its embedding may be derived from a user input given via the user interface unit 116 shown on Fig. 1.

[0162] The LLM 206 of this embodiment has an architecture where each block starts with a layer normalization followed by a multi-head self-attention mechanism. After this, there is a residual connection and another layer normalization. This is then followed by a feed-forward layer and another residual connection.

[0163] The pretrained LLM typically includes query (Q), key (K), and value (V) linear layers within the transformer model. As shown in Fig. 3, trainable rank decomposition matrices 318 - the trainable query (Q), key (K), and value (V) matrices - are injected into each layer of the transformer of the LLM 206 (alongside the original Q, K, and linear layers) for efficient finetuning of the pre-trained LLM. This equips the LLM 206 to generate driving-related answers based on the prompt and vector inputs. For example, embodiments of the present invention may utilize LLaMA-7b as the language model, though other models may readily be used, as appropriate. The Q, K, and matrices 318 are generated by the LoRA module 208.

[0164] As can be seen in Fig. 2, the LLM 206 generates output 126, which includes one or more driving commands, based on the vector embedding 312 and the prompt embedding 316. This output 126 also includes a semantic description of what the control system 118 is observing, what actions it has planned, and the reasoning behind its decisions.

[0165] By way of example only, Fig. 2 shows a particular scene 200 in which the autonomous vehicle 100 is driving behind another vehicle 201, which it is to overtake. The LLM 206 may produce a output 126 comprising textual data outputs that says:

[0166] "Here are my actions:

[0167] - Accelerator pedal 0%

[0168] - Brake pedal 81%Steering 31% to the right

[0169] Reason:

[0170] I need to slow down and overtake the vehicle in front of me"

[0171] The controller 114 receives this output 126 as input and extracts suitable signals 128 which are supplied in a drive-by-wire manner to the drive system 130. This will result in the accelerator pedal being set to 0%, the brake pedal being set to 81%, and the steering being set 31% to the right. The reasoning behind these is captured in the text and can be supplied as a separate output, either in response to a user query during driving operations, or as part of a debugging process, for example. This may be presented to a user via the user interface unit 116 or via some other means.

[0172] Structured Language Generation for Pseudo Vector Captioning

[0173] It can be seen, therefore, that a vector representation of the environment is mapped to an action for the vehicle dynamics with an attention-based neural network architecture. Vector representations are converted into language using a structured language generator. The object-level vectors contain semantically significant attributes such as the number of cars and pedestrians, their respective locations, orientations, speeds, bounding boxes, and other attributes.

[0174] Fig. 4 is a block diagram that provides an illustration of the Driving QA Dataset automatic labelling process. The pre-processing provided by the vector encoders 202 and vector former 204 may be employed to craft language labels - automatically generated captions -which are derived from the vector space representation of a given scenario 400, as illustrated below.

[0175] In other words, the vector encoders 202, vector former 204, and LLM 206 collectively provide a language generator (lanGen) function 402. For example, this lanGen function 402 may provide an exemplary output 404 such as:

[0176] lanGen(yCar, vped, Vego, Vroute, [Or / ]) - >

[0177] “I’m observing 1 car and 1 pedestrian

[0178] A moving car; Angle in degrees: 1.19; Distance: 9.98m; [My attention: 78%] A pedestrian; Angle in degrees: -41.90; Distance: 11 ,94m; [My attention: 22%]

[0179] My current speed is 11.96 mph.There is a traffic light and it is red. It is 12.63m ahead.

[0180] The next turn is 58 degrees right in 14.51m.

[0181] [Here are my actions:]

[0182] [ - Accelerator pedal 0%]

[0183] [ - Brake pedal 80%]

[0184] [ - Steering straight]”

[0185] In this function 402, the variables vcar, vped, vego, andrOute denote the vector information corresponding to cars, pedestrians, ego vehicle, and route, respectively - as outlined previously.

[0186] The term oricorresponds to the output from the RL agent, consisting of additional attention and action labels for guiding the action reasoning process. Attention labels are collected from RL policy attention layers.

[0187] This structured language generator function 402 enables the transformation of vector representations into understandable language captions 404. It crafts a comprehensive narrative of the current driving scenario, inclusive of the agent’s observations, the agent’s current state, and its planned actions.

[0188] This comprehensive contextual foundation enables the LLM 206 to conduct reasoning and construct appropriate responses 406 in a manner that humans can interpret and understand.

[0189] The inclusion of oriis optional, and so two different versions of pseudo labels may be generated to cater to different reguirements:

[0190] Without Attention / Action: Employed during the representation pre-training stage where the inference of attentions and actions is not reguired.

[0191] With Attention / Action: Utilized for QA labelling with GPT during the fine-tuning stage. This eguips GPT with the capacity to ask specific guestions about attentions and actions, thereby empowering the driving LLM agent with the ability to reason about attentions and actions.

[0192] The QA dataset labelling and training processes are outlined in further detail below.

[0193] Driving QA Dataset LabellingThe Applicant has appreciated that having a large amount of data is typically the key to enabling the question answering ability of language models. This becomes particularly crucial when a new modality is introduced to the LLMs; in such circumstances, it may be highly beneficial to have a high-quality question-answering dataset relevant to the modality input.

[0194] Data labelled using ChatGPT has previously been shown to surpass crowd-sourced workers in terms of performance for text-annotation tasks. Embodiments of the present invention may utilize GPT-3.5 to generate a Driving QA dataset. This dataset is conditioned on the structured language input, serving as a representative for the vectors. To be specific, GPT-3.5 may, in some embodiments, be tasked to emulate the role of a certified professional driving instructor. It will of course be appreciated that different language models, e.g. other LLMs, may be used.

[0195] Basic driving rules are supplied as pre-prompts, covering elements like traffic signal transitions and the driving side (e.g. if operating in a left-handed driving country). GPT-3.5 is tasked with creating sixteen sets of random questions and answers per scenario, considering the current driving scenario provided by language description generated by the lanGen 402. These question-answer pairs were generated based on two criteria: the answers should be derived from the given input observations and the model’s reasoning, and the responses may be formatted in JSON line format for easier parsing. Those skilled in the art will appreciate that 'JSON' refers to JavaScript Object Notation (JSON) where a series of fields are defined and values forthose fields specified using a machine-readable syntax. For example, the JSON data {“firstName”: “John”, “secondName”: “Smith”} allows values for the firstName and secondName defined fields to be extracted.

[0196] To incentivize the self-generation of question-and-answer pairs that contain information from the input vector, some preset questions are provided. Simple guardrail rules are established too - GPT-3.5 is instructed to include a driving-related question that could not be inferred from the observation input. This way, the model’s response would be something akin to "I’m unable to answer this question based on the observations I have," reducing the chance of hallucination.

[0197] Additionally, a random question unrelated to driving is included, instructing the model to respond with "As an Al Driver, the question you asked is out of my scope, but I can attempt to answer it." The model would then proceed to answer the question normally, thereby preserving the versatility of the language model as a general-purpose assistant.Below are some examples of the question-answer pairs 406 generated by GPT-3.5:

[0198] Question: "What objects are you observing?"

[0199] Answer: "I’m observing a moving car, and two pedestrians. The car is coming from the opposite direction, and the pedestrians are crossing the street from both the right and left sides."

[0200] Question: "How are you going to drive in this situation and why?"

[0201] Answer: "Here are my actions: - Accelerator pedal 0% - Brake pedal 7% - Steering straight. Reason: Because I’m maintaining a slow speed due to the pedestrians crossing, and I’m steering straight to keep our vehicle properly aligned with the road."

[0202] Question: "What is the protocol if the pedestrian crossing the road suddenly stops?" Answer: "If a pedestrian suddenly stops, we should further decrease our speed, and if necessary, come to a complete stop to ensure their safety and avoid a collision. It’s crucial to always be prepared for unpredictable pedestrian behaviour."

[0203] Question: "Can you check the tyre pressure?"

[0204] Answer: "I’m unable to answer this question based on the observation I have. My current input observations include objects, pedestrians, traffic lights, and driving actions."

[0205] Question: "What are the best tourist spots in London?"

[0206] Answer: "As an Al Driver, the question you asked is out of my scope, but I can try to answer it. Some famous tourist spots in London include the Tower of London, Buckingham Palace, The British Museum, The Shard, and the London Eye."

[0207] Training the Driving LLM Agent

[0208] Training the driving LLM agent involves formulating it as a Driving Question Answering (DQA) problem within the context of a language model. The key to this formulation is the integration of object-level vector modality with the pre-trained LLMs, creating a multi-modal system capable of interpreting and interacting with both language and vector inputs.

[0209] A two-stage process is used to train the model for effectively fusing the object-level vector modality into the driving LLM agent.In the first stage, the vector representation is grounded into an embedding that can be decoded by the LLM. This is done by freezing the language model and optimizing the weights of the vector encoders and the vector transformer.

[0210] In the second stage, the model is fine-tuned to the driving task, training it to answer driving-related questions and take appropriate actions based on its current understanding of the environment.

[0211] Fig. 5 illustrates a demonstration of the interaction with the pretrained driving LLM agent. Notably, the driving LLM agent possesses the ability to anticipate future actions when prompted. Fig. 5 shows a scenario 500 in which the autonomous vehicle 502 (the ego vehicle) is approaching a crossroads, i.e. a four-way intersection. This intersection is governed via a series of traffic lights 504a-d.

[0212] There are several pedestrians 506 walking alongside the road, and several other vehicles 508, some of which are actively turning across the intersection in front of the autonomous vehicle 502. Note that not all of the pedestrians 506 and other vehicles 508 are labelled, for ease of illustration.

[0213] In this particular scenario 500, the traffic light 504a ahead of the autonomous vehicle 502 is red, i.e. the vehicle 502 should stop and wait rather than driving into the intersection.

[0214] As shown in the LLM output 510 of Fig. 5, a user prompt of “What’s your action and why?” is met with the response:

[0215] “Here’s my actions: Accelerator pedal 0% Brake pedal 35% Steering straight. Reason: I am applying the brake pedal to slow down the car as the traffic light ahead

[0216]

[0217] is red. ”

[0218] A follow up user prompt of “What will you do if the traffic light turns green?” is met with the response:

[0219] “ / would release the brake pedal and accelerate the accelerator pedal to continue driving. ”The LLM is able to produce not only an output suitable for conversion to driving commands and its reasoning, but it is also capable of providing an output outlining what it will do under a future change in condition (i.e. the light subsequently turning green).

[0220] Training using a Simulator

[0221] To generate language-based grounded driving datasets, a realistic 2D simulator can be used with procedural generation of driving scenarios. In such arrangements, an RL agent is used to solved simulated scenarios using an object-level ground-truth representation of the driving scene.

[0222] Numeric Upweighting Loss for Vector Representation Pre-training

[0223] Integrating a new modality into pre-trained Large Language Models (LLMs) poses significant challenges due to the need for extensive data and computational resources. Embodiments of the present invention use a novel approach that uses structured language to bridge the vector space with the language embedding, with particular focus on numerical tokens.

[0224] In the pretraining phase, a strategy is employed in which the language model is frozen while the entire framework is trained end-to-end. This approach is designed to optimize the weights of the vector encoders and the vector former. Such an optimization process enables effective grounding of the vector representation into an embedding that can be directly decoded by the LLMs.

[0225] It is important to note that during this pretraining phase, attention and action elements are deliberately excluded from the structured language. This is because the focus at this stage is solely on representation training, and it is desirable to avoid prematurely integrating any reasoning components into the V-former.

[0226] To enhance the accuracy of number decoding, it may be particularly advantageous to use weighted cross-entropy on number and symbol tokens (‘0-9’,

[0227]

[0228] and In other words, number and symbol tokens are made more important than alphabetical tokens ('A-Z', and 'a-z') or other tokens. This approach adds further penalization on the decoded numbers, hence increasing the performance of the learned representation embeddings.

[0229] The pretraining was conducted on 100k simulator-collected data points with additional 200k uniform randomly generated data points over the course of 5 epochs. This totals to an equivalent of 1.5 million vector sample with vector captioning labels, providing a robusttraining set for the novel methodology. Through this approach, embodiments of the invention may effectively incorporate object-level vector modality into pre-trained LLMs, which provides a very good starting point for the fine-tuning stage.

[0230] Driving QA Fine-tuning

[0231] After the pre-training stage, the trainable LoRA module 208 is ingested to the LLM 206, and optimization of the weights of the vector encoder 202, vector former 204, and LoRA module 208 is performed in an end-to-end fashion on the Driving QA data collected earlier.

[0232] To train the driving LLM agent to output accurate action for driving, certain action-triggering guestions are added to the VQA dataset. These are guestions that, when asked, reguire the agent to generate actions in specific format. For example, a guestion like "How are you going to drive in this case and why?" would reguire the agent to infer the action based on the vector input. A simple language-action grounding strategy with regular expression matching is used to extract the action reguired to control the car in a simulator.

[0233] To ensure the model pays sufficient attention to these important guestions, the actiontriggering guestions are upsampled with different expressions such as:

[0234] • "How are you going to drive in this situation?"

[0235] • "What actions are you taking?"

[0236] • "How are you driving in this situation?"

[0237] • "What are your planned actions for this situation?"

[0238] The pre-existing model is refined on 8,000 scenarios, each incorporating 16 driving QA entities, along with an additional vector captioning data and an action prediction QA pair. This results in an aggregate of 144,000 QA pairs. Training is conducted across 5 epochs until the model achieves convergence. The fine-tuning process optimizes the weights of the vector encoders and the Vector Former in an end-to-end manner.

[0239] The outcome is a driving LLM agent that is capable of reasoning about its actions based on the current driving scenario. This demonstrates the power and potential of integrating objectlevel vector modality with pre-trained LLMs, particularly in the context of autonomous driving.

[0240] Evaluation and Results

[0241] This section aims to thoroughly assess the model’s performance using a reserved evaluation set comprising 1 ,000 distinct driving scenarios. The vector captioning and vector QA data for these scenarios were labelled following the methodologies discussed herein.Evaluation of Vector Captioning and Action Prediction

[0242] In this section, the results of the driving QA models on the vector captioning and action tasks are presented. The results on the model trained using the two-stage approach (with pretraining), and the model training only on driving QA dataset (without pretraining), are reported herein.

[0243] The Mean Absolute Error (MAE) is calculated for the predictions of the number of cars and pedestrians, denoted as E^and Epecj respectively. Additionally, the accuracy of traffic light detection is measured, as well as the mean absolute distance error in meters for traffic light distance prediction, represented as ACCTL and DTL-

[0244] The MAE for ego states, including speed in mph and steering angle in percentage, are presented as Espeed and Esteer. Furthermore, the MAE is computed for the normalized acceleration and brake pressure, denoted as Eaction-

[0245] Finally, the weighted cross entropy loss is reported for the token prediction on the evaluation set, indicated as Ltoken-

[0246] Agent Traffic Light Ego States Actio Loss

[0247] n

[0248] Ecar -I Eped -I ACC-TL DTL J. Espeed Esteer Eaction Ltoken

[0249] T

[0250] LLM 0.373 1.028 0.737 3.669 0.693 3.921 0.110 0.737 Agent

[0251] without

[0252] pretrain

[0253] LLM 0.169 0.305 0.748 3.562 1.034 3.770 0.116 0.685 Agent

[0254] with

[0255] pretrain

[0256]

[0257] Table 1: The evaluation result of vector captioning and action predictionThe results clearly demonstrate that pretraining with the novel numeric upsampling significantly enhances the agent’s detection capabilities. There is a significant decrease in the MAE for both the car and pedestrian prediction tasks. This suggests that the pretrained model exhibits a higher level of accuracy in perceiving and quantifying the number of cars and pedestrians in its environment.

[0258] The pretrained model also shows a lower loss value, Ltoken, which indicates an improvement in the overall effectiveness of the model’s token predictions. Interestingly, the pretraining stage did not yield significant improvements in ego states decoding and action prediction. The pretrained model’s performance in terms of traffic light detection accuracy, ACCTL, is similar to the model without pretraining. The distance error for traffic light detection, DTL, also remains relatively unchanged. This outcome could be attributed to the relatively small size of the driving QA dataset. The action prediction task is primarily learned during the finetuning stage, and with the current limited volume of data, the model might not be able to fully capture the complexity of driving actions. However, a key advantage of the disclosed method is its high scalability. As the driving QA dataset is expanded with more samples, it is anticipated that the performance of action prediction will improve correspondingly during the fine-tuning stage.

[0259] Evaluation of Driving QA

[0260] To assess the quality of answers to open-ended questions about the driving environment, GPT-3.5 is used to grade the model’s answers. This allows for quick and cheap evaluation of the model’s capabilities on questions without a fixed answer such as summarising the surrounding, explaining the reason for taking a certain action, or describing the most important objects in the scene.

[0261] For evaluation, GPT-3.5 is prompted with the language-based observation description used during dataset generation, the question from the test set, and the answer of the model. GPT-3.5’s task is to write a one-line assessment for each answer, followed by a score from 0 to 10 where 0 is worst and 10 is best. The final score of the model is the average over all question scores.

[0262] Grading the outputs of the LLM agent using GPT-3.5

[0263] Grading score T

[0264]

[0265] LLM Agent without pretrain 7.28 ± 1.20 LLM Agent with pretrain 7.88 ± 1.20 Constant answer "I don’t know" 2.4 ± 2.3 Answers generated by GPT-3.5 9.46 ± 0.6

[0266]

[0267] Table 2: Grading the outputs of the LLM agent using GPT-3.5. Scores are in the range 0 to 10, where 0 is worst and 10 is best.

[0268] The results show that pre-training improves the grading score of the model by 8.2% over no pre- training. For reference, also provided is the score when running the evaluation procedure with the answer labels provided by GPT-3.5 and a constant answer of "I don’t know". The model, using only vectorized representations as input, can get a score of 7.88, much higher than the score of 2.4 when just answering with "I don’t know".

[0269]

[0270] ic natural

[0271] Further to the above, embodiments of the present invention relate to methods and systems for operating autonomous vehicles by generating driving commands and / or textual data outputs based on natural language prompts (in addition to or instead of controlling autonomous vehicles based on sensor input / vector embeddings as described above).

[0272] Overview

[0273] Figs. 6 and 7 demonstrate an autonomous driving system 600 and method 700 for operating a vehicle 100.

[0274] The autonomous driving system 600 of Fig. 6 comprises a sensor unit 602, an operational parameter unit 604, a destination information unit 606, a prompt unit 608, an encoder unit 610, a language model unit 612, a controller 614, and a user interface unit 616.

[0275] The sensor unit 602 is configured to generate an environmental signal corresponding to a (surrounding) environment of the vehicle 100. There are many different types of sensor, known in the art per se, that may be used within the sensor unit 602, as appropriate. It will be appreciated that where multiple sensors are used, some of all of these may be physically distinct from one another, however for ease of reference they are referred to collectively as a 'unit' herein, even where multiple separate sensor devices are used. The sensor unit 602may include one or more of: a camera; a microphone; a sonar sensor; an ultrasonic sensor; a radar sensor; an ultra-wideband radio sensor; a lidar sensor; an optical flow sensor; a magnetometer; a thermometer; a barometer; a moisture sensor; a humidity sensor; an inertial measurement unit; a gyroscope; an accelerometer; a touch sensor; a laser rangefinder; an infrared sensor; a vehicle speed sensor; a yaw rate sensor; a wheel speed sensor; a current sensor; a fuel level sensor; a voltage sensor; an altitude sensor; a tilt sensor; a Hall effect sensor; an odometer; and a geopositioning sensor. It will be appreciated that the sensor unit 602 may comprise any combination of sensor types, and may comprise more than one of the same type of sensor, as appropriate.

[0276] The sensor unit 602 is used to determine the state of the world around the vehicle 100 (i.e. the scenario / environment in which the vehicle 100 finds itself). This may include detecting entities and objects such as other vehicles, pedestrians, cyclists, traffic signals, road obstructions, and so forth, as well as the state of such detected entities and objects.

[0277] The operational parameter unit 604 provides information or data relating to the operational state of the vehicle 100 (i.e. the 'ego' vehicle). The operational parameters provided by the operational parameter unit 604 may include information relation to, by way of non-limiting example: a location of the vehicle, a speed of the vehicle, an orientation of the vehicle, a trajectory of the vehicle, a bounding box of the vehicle, one or more actuator positions of the vehicle, an accelerator state of the vehicle, and a brake state of the vehicle.

[0278] The destination information unit 606 provides information relating to the destination or route of the vehicle 100. This may be the coordinates of the destination, a list of upcoming directions (e.g. next right turn, second exit at roundabout, straight on for 5 miles, etc.), a next direction, or a route of the vehicle 100.

[0279] While in the embodiment shown in Fig. 6, the autonomous driving system 600 makes use of the operational parameters and destination information, these may be omitted. The Applicant has appreciated that the vehicle 100 may be driven based on only the environmental signal obtained from the sensor unit 602, though the inclusion of the operational parameters and destination information is beneficial for enhanced functionality. In other embodiments, e.g. where the autonomous driving system 600 has been implemented / deployed in a test or simulated environment, the sensor unit 602 may be replaced by a test (sensor) unit configured to provide test environmental signals (e.g. comprising historical and / or synthetic environmental signals). Similarly, the operational parameter unit 604 and the destinationinformation unit 606 may be replaced by test units configured to provide test (e.g. comprising historical and / or synthetic) operational parameter(s) and / or destination information.

[0280] The prompt unit 608 is configured to generate a natural language prompt based on at least one of: a user input and driving parameters.

[0281] The prompt unit 608 may be a separate component / unit as shown in Fig. 6, or it may be integrated with / a component part of the language model unit 612. The prompt unit 608 is configured to collect / aggregate data of a textual nature, and to generate a concise natural language prompt based on the aggregated textual data. This natural language prompt is formulated in natural language, allowing it to summarise and encapsulate desired driving behaviour in a manner that is easily comprehensible for both the (machine learning language model of the) autonomous driving system and the human operators.

[0282] Accordingly, the prompt unit 608 is configured to receive at least one of: a user input (such as ad-hoc user prompts, e.g. queries or instructions received via the user interface unit 616) and driving parameters. The driving parameters may be based on local driving rules and regulations, which may be selected in accordance with a determined location of the autonomous vehicle. The driving parameters may be based on driving modalities / behaviours selected in accordance with at least one of a time of day, a weather condition, a traffic condition, and a user input (driving preference). The driving parameters may be based on operational parameters corresponding to operation of the vehicle (e.g. a top speed or acceleration of the vehicle) and / or information relating to a destination of the vehicle (e.g. an end-destination or next direction).

[0283] For example, the prompt unit 608 may be configured to receive a command, instruction, query, or request from the user interface unit 616 (e.g. "pull over", "turn right instead of left at the next intersection", "overtake the vehicle in front", "slow down"). For this purpose, the user interface unit 616 may comprise one or more of a graphical user interface, a touch user interface, a voice user interface, a gesture-based interface, a natural language user interface, a menu-driven interface, etc. The user interface unit 616 is configured to convert the user input to a textual user input for providing / communicating to the prompt unit 608. For example, wherein the user interface unit 616 comprises a voice user interface (e.g. comprising a microphone), the user interface unit 616 may be configured to, or comprise a real-time speech recognition unit configured to, convert audio user input received via the voice user interface to a textual user input.The prompt unit 608 may be configured to generate the natural language prompt based on preset driving modes (e.g. that can be characterized as "cautious", "mild" or "assertive"), wherein said preset driving mode may be selected by a user (e.g., to suit their general driving preferences). In one example, custom driving modes may be developed by learning from the user’s manual driving style, allowing the autonomous driving system to personalize driving behaviour based on past inputs.

[0284] The prompt unit 608 may be configured to generate the natural language prompt based on driving laws, rules and / or guidance (collectively termed "regulations") that can be characterized by location (e.g. "UK" or "Japan"). For example, and with respect to the UK, the driving parameters may be based (at least in part) on the Highway Code. In one example, the prompt unit 608 may be configured to retrieve driving regulations from a driving handbook / database for a particular country / region selected in accordance with a determined location based on the environmental signal received from the sensor unit 102 (e.g. based on video camera sensor data or geopositional sensor data). Alternatively, the prompt unit 608 may be configured to retrieve driving regulations from a driving handbook / database for a particular country / region selected by a user. Alternatively, the language model unit 612 is pre-trained to associate / know the relevant driving regulations for a given location name.

[0285] The prompt unit 608 may be configured to generate a natural language prompt comprising or based on driving parameters corresponding to at least one of a determined time of day, weather condition, traffic condition, etc. based on the environmental signal. For example, the prompt unit 608 may be configured to generate a natural language prompt comprising at least one preset (e.g. known) driving parameter (e.g. "night", "light rain", "snow", "fog", "filtering", "lane closure") that has been selected according to a time of day, weather condition, traffic condition, etc. that can be determined from an environmental signal received from the sensor unit 602.

[0286] The prompt unit 608 may be configured to generate a natural language prompt comprising or based on driving parameters corresponding to operational parameters of the vehicle, e.g. received from operational parameter unit 604. For example, the prompt unit 608 may be configured to generate a natural language prompt comprising a driving parameter relating to the operation vehicle (e.g. a current speed, a top speed, acceleration, braking distance(s)) that can be derived from data received from the operational parameter unit 604.

[0287] The prompt unit 608 may be configured to generate a natural language prompt comprising or based on driving parameters corresponding to destination information of the vehicle, e.g.received from the destination information unit 606. For example, the prompt unit may be configured to generate a natural language prompt comprising a driving parameter relating to the next direction in a list of directions (e.g. next right turn, straight on for 10 miles, second exit at roundabout, straight on for 5 miles, etc.).

[0288] Thus, the natural language prompt may be / comprise an amalgam / aggregation of (all) the input information received by the prompt unit 608, in the form of natural language.

[0289] For example, the prompt unit 608 may receive a user input / prompt: "Slow down and turn right at the next intersection". At the same time, the prompt unit 608 may obtain system prompts (e.g. driving parameters having a higher priority than user input): "In the UK; in a 30 km / h speed limit zone" and "Light rain" based on environmental signals / sensor data received from sensor unit 602.

[0290] Based on the received user and system prompts, the prompt unit 608 may generated a natural language prompt such as:

[0291] “Currently driving in the UK and approaching densely populated urban area; reducing speed to comply with local 30 km / h speed limit.

[0292] Weather conditions are rainy, activating increased traction control and adjusting headlights, driving carefully and mild.

[0293] Preparing to execute a gentle right turn at upcoming intersection as user requested. "

[0294] In the example shown in Fig. 6, the natural language prompt generated by the prompt unit 608 and the data from sensor unit 602 (and, optionally, data from operational parameter unit 604 and destination information unit 606) are passed to the encoding unit 610.

[0295] The encoding unit 610 comprises the necessary components to encode / tokenise the received inputs into a structured semantic representation / vector embedding that can be understood / interpreted by the language model of the language model unit 612. For example, as explained in more detail below, where the sensor unit 602 comprises one or more video cameras, the encoding unit 610 may comprise a video encoder configured to convert the video data into feature embeddings (in the language latent space). Additionally, the encoding unit 610 may comprise a tokeniser (textual encoder) configured to encode / embed the natural language prompt generated by the prompt unit 608.The encoding unit 610 may be a separate component / unit as shown in Fig. 6, or it may be integrated with / a component part of the language model unit 612. In the latter case, the environmental signal / sensor data and natural language prompt may be provided directly to the language model unit 612, i.e. in its 'raw' form.

[0296] Based on at least one of the natural language prompt and the sensor data, the language model unit 612 may be configured to generate actionable driving commands as well as a detailed language description of the intended driving manoeuvres (or an explanation of why the intended driving manoeuvre cannot be actioned), which align with both the system's assessments and user expectations.

[0297] This output of the language model unit 612, which comprises one or more driving commands and / or one or more textual data output, and is based on at least one of the (embedding of the) environmental data and the (embedding of the) natural language prompt, is then relayed to the controller 614, which controls the drive system 130 of the vehicle 100 as described earlier.

[0298] In accordance with the earlier described systems, as one or more driving commands in the output are being executed, the autonomous driving system 600 is able to convey the vehicle’s intentions to both internal occupants and external road users. This is achieved by also (in parallel or sequentially to providing the output to the controller 614) providing the output to the user interface unit 616. The user interface unit 616 may communicate with the user through a variety of communication methods, including external visual signals, auditory alerts, and possibly haptic feedback mechanisms, as well as internal displays and auditory feedback. This comprehensive communication strategy ensures that the vehicle’s manoeuvres are transparent and predictable to the user, enhancing safety and trust.

[0299] Throughout operation, the autonomous driving system 600 continuously monitors the effectiveness of the one or more driving commands and the vehicle’s adherence to the set parameters. This real-time feedback is crucial in live driving situations for dynamic adjustments and ensures that the vehicle 100 responds appropriately to changing conditions and potential unforeseen events. Adjustments are made in real-time to optimize driving performance, enhance safety, and maintain compliance with local traffic laws. By being able to respond / adjust the one or more driving commands based on ad-hoc user input, the system 600 is capable of incorporating feedback from the human user, refining, and personalizing the driving experience further.Fig. 7 demonstrates a computer-implemented method 700 of operating a vehicle 100 (e.g. using the autonomous driving system 600).

[0300] The method comprises receiving 702 an environmental signal corresponding to an environment of the vehicle 100. The environmental signal may be provided / generated by the sensor unit 602; and may be received at / by the encoding unit 610 / language model unit 612 and / or the prompt unit 608.

[0301] Concurrently or consecutively, the method comprises receiving 704 at least one of: a user input and driving parameters. The user input may be received at / by the prompt unit 608 via the user interface unit 616.

[0302] The driving parameters may be selected / obtained by the prompt unit 608 based on factors such as a determined location of the vehicle 100, a time of day, a weather condition, a traffic condition, a driving mode chosen by a user, etc.

[0303] The term "driving parameters" is used herein to refer to (textual) instructions, criteria, or guidelines that define or set particular requirements as to how the vehicle should drive. Some driving parameters (e.g. driving instructions based on local driving rules and regulations, ordriving instructions regarding a particular weather condition) may be considered 'system prompts', in that they have a higher priority than / cannot be overridden by user prompts or input. However, some driving parameters (e.g. driving instructions based on a preset driving mode, such as "cautious", "mild", or "assertive") may be selected / retrieved based on a user input / prompt, and therefore can be overridden by subsequent user prompts / input.

[0304] That being said, the system is configured to attempt to generally comply (where possible) with all the received user input and all the received driving parameters. For example, where a user provides an ad-hoc driving instruction (e.g. to "overtake"), the language model unit 612 is configured to generate one or more driving commands which, when executed, is compliant with the user's (previously) selected driving mode (e.g. "mild") and the local driving regulations (e.g. "Driving on a dual carriageway in the United Kingdom").

[0305] The method comprises generating 706 (by the prompt unit 608) a natural language prompt based on the user input and / or the driving parameters. Thus, the prompt unit 608 may be considered an "Information Aggregation Unit" in that it aggregates and assembles a natural language prompt from on the received / obtained various textual / natural language inputs.For example, where the prompt unit 608 receives a user input that selects or identifies a preset driving mode, the prompt unit 608 generates a natural language prompt that comprises (instructions that describe, correspond to, or relate to) the selected preset driving behaviour.

[0306] Additionally or alternatively, where the prompt unit 608 receives or obtains a location (determined based on the environmental signal, e.g. by the machine learning model unit 612), the prompt unit 608 generates a natural language prompt that comprises instructions that describe, correspond to, or relate to the location (e.g. local driving rules and regulations).

[0307] Additionally or alternatively, where the prompt unit 608 receives or obtains a time of day, weather condition, traffic condition, etc. (determined based on the environmental signal, e.g. by the machine learning model unit 612), the prompt unit 608 generates a natural language prompt that comprises (instructions that describe, correspond to or relate to) a preset driving mode for the time of day, weather condition, or traffic condition (e.g. "night", "light rain", "snow", "fog", "filtering", "lane closure").

[0308] The method comprises generating 708a (by the machine learning language unit 612) one or more driving commands based on at least one of the environmental signal and the natural language prompt; and / or generating 708b (by the machine learning language unit 612) one or more textual data outputs based on at least one of the environmental signal and the natural language prompt.

[0309] The step of generating 708a and / or the step of generating 708b may comprise preprocessing of the environmental signal and the natural language prompt, that is, the environmental signal and the natural language prompt may be encoded / tokenised by the encoding unit 610 prior to processing by the machine learning language unit 612.

[0310] In some examples / scenarios, one or more driving commands and one or more textual outputs may be generated in parallel based on the (same) environmental signal and / or natural language prompt.

[0311] In other examples / scenarios, the method may comprise generating 708a (by the machine learning language unit 612) one or more driving commands based on the at least one of theenvironmental signal and the natural language prompt; and then (i.e. subsequently, not in parallel) generating 708b (by the machine learning language unit 612) one or more textual data outputs, based on the at least one of the environmental signal and the natural language prompt, and also based on (e.g. guided or conditioned by) the generated one or more driving commands. For example, a generated driving command (e.g. accelerate) may be followed by a textual data output that is conditioned / influenced by the generated driving command (e.g. "We are accelerating to safely overtake the car in front, as requested.").

[0312] In other examples / scenarios, the method may comprise generating 708b (by the machine learning language unit 612) one or more textual data outputs based on the at least one of the environmental signal and the natural language prompt; and then (i.e. subsequently, not in parallel) generating 708a (by the machine learning language unit 612) one or more driving commands, based on the at least one of the environmental signal and the natural language prompt, and also based on (e.g. guided or conditioned by) the generated one or more textual data outputs. For example, a generated textual data output (e.g. "It looks like it is starting to rain.") may be provided back to the prompt unit 608 and / or used to condition the generation of (future / following) one or more driving commands.

[0313] In other examples / scenarios, the method may comprise only generating 708a (by the machine learning language unit 612) one or more driving commands based on the at least one of the environmental signal and the natural language prompt; or only generating 708b (by the machine learning language unit 612) one or more textual data outputs based on the at least one of the environmental signal and the natural language prompt. For example, a generated textual data output may inform the user that a requested driving command / action cannot be generated / executed (e.g. "We cannot pullover safely here."). In the alternative, generating 708b textual data outputs can be ignored / bypassed (whilst generating 708a driving commands) if the user does not require it or the situation requires particularly quick action.

[0314] The method may comprise providing the generated one or more driving commands to the controller 614 (thereby operating the vehicle based on the one or more driving commands). Thus, the method includes flexible control over the autonomous driving system 600 via natural language inputs at varying levels: at a high-level, the system 600 is able to interpret / understand local driving regulations to ensure compliance and safety across international borders; at a mid-level, the system 600 is able to adapt driving behaviour to the requirements of a specific scenario, including (e.g.) a type of user (user preference), a timeof day, and a weather condition; and at a low-level, the system 600 is able to respond to user commands, to provide real-time control of the driving vehicle.

[0315] Additionally or alternatively, the method may comprise providing the generated one or more textual outputs to the user interface 616 and / or prompt unit 608. Thus, the method 700 includes communicating the driving intention / plan of the system 600 to the user, providing commentary / explanation as to why particular actions have been taken or not taken (or are going to be taken or not going to be taken). This functionality enhances safety, transparency, and trust, fostering greater acceptance and reliability of autonomous vehicle technology in public domains.

[0316] Specific Embodiment

[0317] Figs. 8A and 8B demonstrate an exemplary embodiment of an autonomous driving system for an autonomous vehicle comprising a language model unit 812 (the Wayve Vision-Language-Action Model) configured to generate one or more driving commands (control signals -'action') and / or one or more textual data outputs ('intention' and 'commentary') based on a natural language prompt (natural language system prompt) and an environmental signal corresponding to an environment of the vehicle (sensor / video input).

[0318] As shown in Fig. 8A, the natural language prompts (natural language system prompts) are generated (e.g. by a prompt unit / lnformation Aggregation Unit 808) based on at least one of: user input (such as user ad-hoc driving instructions received via user interface 816) and driving parameters (e.g. textual instructions, requirements, or criteria). The driving parameters may correspond to local driving rules / regulations, which may be selected / obtained from a database 808a (e.g. of the prompt unit 808) in accordance with a location of the vehicle determined from data (e.g. GPS data) from the sensor unit 802. Alternatively or additionally, the driving parameters may correspond to a preset driving mode (obtained from a database 808b (e.g. of the prompt unit 808)) selected in accordance with at least one of a time of day, a weather condition, and a traffic condition determined from data (e.g. weather sensor) from the sensor unit 802, and / or a preset driving mode selected in accordance with a user input.

[0319] As shown in Fig. 8B, the environmental signal / sensor data from the sensor unit 802 (in this embodiment, video input) is pre-processed through a video encoder 810b (of an encoding unit 810) that converts visual information into embeddings. Simultaneously, natural language prompts are encoded using a textual tokenizer / encoder 810a (of an encoding unit 810) and an embedding lookup table (not shown). The language model is further inputted with multiplelearnable queries, each finely tuned for generating outputs related to intentions, actions, and commentary.

[0320] Thus, the Wayve LAM 812 is, in real-time, capable of and configured for generating intention, action, and commentary (i.e. one or more driving commands and / or one or more textual data outputs) based on the (embedded) natural language system prompt and (embedded) video input.

[0321] This hierarchical approach to model design - sequencing intention, action, and commentary -offers dual advantages: firstly, the auto-regressive generation from a high-level plan (intention) to a low-level plan (action) facilitates more precise and context-aware vehicle operations. Secondly, this sequence ensures that the vehicle adheres to its predetermined driving intentions, providing the ability to reflect and comment on both the predicted intentions and the executed actions from a comprehensive, global perspective. This strategic framework not only improves the driving performance but also enhances the interactive experience by aligning the vehicle's operations with real-time environmental and situational demands. As above, whether the intention and commentary are provided to the user (passenger), and not just generated as an intermediate reasoning step for 'Chain-of-Thought' prompting and reasoning, may depend on their input / selected driving mode / parameters.

[0322] The training datasets

[0323] We created a collection of datasets for bringing language to autonomous driving. The total dataset size is 419.9k question-answer pairs, where a single data sample consists of a 4-second video clip at 1Hz. Our labeled autonomous driving training dataset consists of two complementary parts: the action dataset and the scenery dataset.

[0324] i) Action dataset

[0325] The action dataset was created from recorded driving sessions featuring notable events where the car’s behaviour changes, such as decelerations, accelerations, lane changes, narrow gaps, and turns. These events were labelled by driving operators with concise, high-level descriptions of the situations and behaviours (e.g., "following lane, pedestrian on a zebra crossing, should stop").

[0326] Additional metadata was included from various perception systems, covering traffic light presence, vehicle and pedestrian detection, weather conditions, and other data like speed, steering wheel position, and road type. Using this information, we developed prompt templates for (1) describing the current action and its justification and (2) generatingexample questions and hints for expected answers. We then used these prompts with GPT-3.5 to rephrase, answer, and extend the example questions based on the provided action descriptions and answer hints. Events were rebalanced by categorizing them based on actions and behavioural policies, and up to 500 events were sampled from each category, resulting in 24,577 video snippets with 167,774 question / answer pairs.

[0327] ii) Scenery dataset

[0328] The scenery dataset was designed to complement the action dataset by focusing on finegrained perception-related questions. It was created by densely and thoroughly annotating three 30-minute driving sessions using the ELAN video annotation software. During these sessions, brief captions were provided in approximately 15 different categories, including driver actions, justifications, attention, observations of vehicles, pedestrians, road elements (such as traffic lights, traffic islands, and intersections), and environmental details (like weather and buildings). Annotations were collected every second (1 fps) for every frame to build textual descriptions that included driver actions, justifications, and observations. Unlike the Action dataset, where GPT-3.5 was used only for question rephrasing, GPT-4 was employed for the Scenery dataset to generate questions and answers. GPT-4 was prompted with chain of thought specifically targeting perception questions, resulting in a diverse and high-quality dataset with about 43 QA pairs per video.

[0329] Dataset statistics

[0330] Our training dataset covers 9 different competencies: action (what the vehicle is doing), justification (why the action is taken), attention (what should be paid attention to in the current situation), identification (identifying an object given its description), localisation, description, counting, anticipation, and reasoning given counterfactuals. The questions also cover a diverse set of objects, such as pedestrians, vehicles, cyclists, buildings, road infrastructure, signs, markings.

[0331] The vision encoder

[0332] CLIP, a Vision Transformer (ViT) pre-trained contrastively on image-language pairs, may be (but not necessarily) used in the context of the invention to encode images into features. The inputs to the vision encoder are RGB images from the front camera. We squash the input images to a size of 224x224 as opposed to cropping them in order to keep the full image context. Subsequently, we pass the features through a transformer network, the Querying Transformer (Q-Former), that akin to BLIP-2 acts as a bridge between the vision and language feature spaces. The embeddings are then projected into the large language model (LLM) space using a linear projection layer. We repeat this process for T = 5 frames of theinput video and concatenate the tokens from each image.

[0333] The language model

[0334] We leverage pretrained LLMs to give the language model unit the ability to answer general guestions related to both driving scenes, as well as general knowledge. We use Vicuna v1.5 with 7B parameters built on top of Llama-2. The language model unit is autoregressive and hence can be conditioned on textual inputs, as well as image tokens. The training objective is to predict the next language token in a seguence. We mask all tokens from the training loss that belong to the text prompt, including guestion and chat history.

[0335] Referential segmentation

[0336] The language model unit may be configured to utilise referential segmentation to visually highlight areas of interest in an image, thereby linking language and vision tasks. This feature may allow description of elements like traffic lights and road markings, but also provides visual proof of its focus, increasing user confidence in its responses.

[0337] The language model unit may be configured for multi-tasking, generating / responding with one or more textual data outputs and performing referential segmentation simultaneously. This advancement opens up new possibilities for autonomous driving, such as enhanced training and adaptability, addressing hallucinations in large language models, and improving safety communication by highlighting potential hazards. By integrating these capabilities in parallel, the system is able to communicate more like a human driver. This development is crucial for building trust and confidence in self-driving technology, ultimately contributing to safer and more reliable autonomous driving systems.

Claims

Claims1. An autonomous driving system for a vehicle, comprising:a prompt unit configured to generate a natural language prompt based on at least one of: a user input or one or more driving parameters; anda language model unit configured to use a machine learning language model to generate one or more driving commands and / or one or more textual data outputs based on an environmental signal corresponding to an environment of the vehicle and the natural language prompt.

2. The autonomous driving system according to claim 1 , wherein the model is a pretrained multi-modal large language model, and the system comprises:an encoding unit configured to receive the environmental signal and encode said environmental signal into an embedding for the pre-trained multi-modal large language model.

3. The autonomous driving system according to claim 1 or claim 2, comprising:a controller configured to operate the vehicle based on the one or more driving commands.

4. The autonomous driving system according to any preceding claim, comprising: a user interface unit in communication with the prompt unit, the user interface unit configured to receive the user input and to provide the user input as textual user input to the prompt unit;optionally wherein the one or more textual data outputs are provided to the user interface unit for communication to the user.

5. The autonomous driving system according to any preceding claim, wherein the prompt unit is configured to receive the environmental signal and to generate the natural language prompt based on one or more driving parameters corresponding to at least one of a location, time of day, weather condition, and traffic condition determined based on the environmental signal.

6. The autonomous driving system according to any preceding claim, wherein the prompt unit is configured to generate the natural language prompt corresponding to a preset driving mode selected in accordance with the user input.

7. The autonomous driving system according to any preceding claim, wherein the prompt unit is configured to receive one or more operational parameters and / or information relating to a destination of the vehicle and to generate the natural language prompt based on one or more driving parameters corresponding to the one or more operational parameters and / or the destination information.

8. A vehicle comprising the autonomous driving system of any preceding claim.

9. A method of operating an autonomous driving system for a vehicle, the autonomous driving system comprising a prompt unit and a language model unit configured to use a machine learning language model, the method comprising:generating, using the prompt unit, a natural language prompt based on at least one of: a user input or one or more driving parameters; andgenerating, using the machine learning language model, one or more driving commands and / or one or more textual data outputs based on an environmental signal corresponding to an environment of the vehicle and the natural language prompt.

10. The method according to claim 9, wherein the machine learning language model is a pre-trained multi-modal large language model, and the system comprises:obtaining the environmental signal and encoding said environmental signal into an embedding for the pre-trained multi-modal large language model.

11. The method according to claim 9 or claim 10, wherein generating a natural language prompt comprises:receiving, at the prompt unit, the environmental signal; andgenerating a natural language prompt based on one or more driving parameters corresponding to at least one of a location, time of day, weather condition, and traffic condition determined based on the environmental signal.

12. The method according to any one of claims 9 to 11 , wherein generating a natural language prompt comprises generating a natural language prompt corresponding to a preset driving mode selected based on the user input.

13. The method according to any one of claims 9 to 12, wherein generating a natural language prompt comprises generating a natural language prompt based on one or more driving parameters corresponding to one or more operational parameters and / or information relating to a destination of the vehicle.

14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 9 to 13.

15. A transitory, or non-transitory, computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 9 to 13.

Images