Apparatus and method for control of vehicles
The SAI framework addresses the challenge of complex scenarios in autonomous driving by modeling vehicle behavior from multiple perspectives, improving prediction accuracy and enabling safer operation in dynamic environments.
Patent Information
- Application Number
- PCT/EP2026/063146
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2026-05-06
- Publication Date
- 2026-11-12
AI Technical Summary
Current autonomous driving systems struggle with complex, dynamic situations known as 'long-tail cases' due to their inability to predict and plan for multiple interacting agents in unstable scenarios, limiting their operation to narrow conditions and requiring human intervention.
A Symbolic Artificial Intelligence (SAI) framework models vehicle behavior from different perspectives using a plurality of agents, iteratively predicting future interactions and updating control instructions based on continuous probability distributions and behavioral constraints.
Enhances the accuracy of predicting future vehicle behavior in complex scenarios, enabling safer and more reliable autonomous driving by simulating multiple agent interactions and adapting control instructions in real-time.
Smart Images

Figure EP2026063146_12112026_PF_FP_ABST
Abstract
Description
[0001] DEVICES AND METHODS FOR CONTROL OF VEHICLES
[0002] Technical field
[0003] The invention relates to methods and apparatus for autonomous control of at least one aspect of vehicle driving. In some specific arrangements, the invention relates to methods and apparatus for determining and optionally implementing control instructions for driving a vehicle.
[0004] Background
[0005] The Society of Automotive Engineering defines levels of autonomous driving ranging from no driving automation (Level 0) through to full driving automation (Level 5). All kinds of autonomous driving, from Levels 2-3 (Advanced Driving Assistance Systems (ADAS)) all the way to Level 5 are limited by what the industry calls “long-tail cases”. These are complex, dynamic situations that occur on the road, often involving multiple parties and for which there is no easy solution for a vehicle control system.
[0006] Especially in urban driving, long tail cases can arise without notice. One moment a vehicle may be following the car in front, the next moment a pedestrian steps out from between parked vehicles, another car is backing into the road, yet another is cutting into the vehicle’s lane, and the traffic light changes to red. These events can occur in a very short space of time, or even all at once.
[0007] The solution space of these complex scenarios is very large and far too great for brute-force approaches using traditional machine learning methodologies. There is no way to see - or even to simulate - all possible outcomes, to provide enough examples of each for a machine learning algorithm to learn what to do in each case. The situations are also inherently unstable in that the parties (agents) in each situation will react to what other agents do and even to what they themselves are doing, so there is no perfect way to plan ahead how to get through the situation. In addition, another situation can begin before the current one ends.
[0008] Surveys consistently show that the mobility industry considers this issue (called “prediction and planning” in the industry) to be the hardest problem to solve for autonomous vehicles -harder than “perception” (object detection and recognition, using cameras and / or lidars plus software) or any other problem in the autonomy stack. This is a reason why currentautonomous driving systems operate only in a very narrow range of conditions, for example: highway (or motorway) only, in perfect weather only, in ideal lighting only, and with a human always ready to take the wheel if needed.
[0009] As used herein, the term “autonomous driving” encompasses control or partial control of any aspect of a vehicle’s behaviour during driving without user input. In some specific arrangements, autonomous driving may include autonomous or semi-autonomous control of one or more of the brakes of a vehicle, the throttle of a vehicle and the direction of a vehicle.
[0010] Summary
[0011] The inventors have appreciated that an alert and experienced human driver normally solves situations such as the long tail cases repeatedly during a typical drive. Humans do this by abstracting a series of “symbols” involved in the process: cars, human behaviour, traffic rules, physical constraints, peculiarities of the local driving culture, etc. A human driver has a feel for what the car opposite will do, based on their statistical knowledge of what human-driven cars normally do in similar cases.
[0012] The inventors have further appreciated that this human capability can be modelled in software and that the resulting system can drive like a human - better, in fact, because such a system is never tired or distracted or confused by “cognitive overload” (the industry term for situations when too many things happen at once). The inventors have developed methods and apparatus using a Symbolic Artificial Intelligence (SAI) framework. SAI is a branch of machine learning that solves problems by breaking them down into human-understandable components.
[0013] According to the invention in an aspect, there is provided an apparatus for determining a control instruction for a vehicle, the apparatus comprising: a receiver configured to receive a first frame of scene data representing a scene in a vicinity of the vehicle at a first time, the first scene data comprising: a plurality of agents including an ego agent representing the vehicle and a plurality of further agents; and spatial coordinates for each of the plurality of agents; a computer processor configured to execute computer program code to control the apparatus to: set the ego agent as a first point of view, POV, agent, and the further agents as first interacting agents; determine, based on the received scene data, first model data predicting future behaviour of the first interacting agents; set one of the further agents as a second POV agent, and the ego agent and remaining further agents as second interacting agents; generate, based on the received scene data, simulated scene data from the POV of thesecond POV agent; determine, based on the simulated scene data, second model data predicting future behaviour of the second interacting agents; and determine a control instruction for the vehicle, based on the first model data and the second model data.
[0014] The first model data may be determined based on the received scene data and may predict the future behaviour of the plurality of first interacting agents. In addition, simulated scene data may be generated, which may simulate the scene data that would be received were one or more of the further agents the ego agent. That is, the simulated scene data may simulate the scene from the POV of one of the further agents. The second model data may be based on the simulated scene data and may predict the future behaviour of the second interacting agents. The future behaviour may include future position, direction, velocity and / or acceleration.
[0015] Optionally, the processor may be configured to iteratively set the plurality of further agents as the POV agent, generate simulated scene data and generate corresponding model data based on the further simulated scene data. The first model data may be updated iteratively using the iteratively determined model data.
[0016] Optionally, the processor is further configured to update the first model data based on the second model data. The control instruction may be determined based on the updated first model data.
[0017] Optionally, the processor is further configured to determine the first and / or second model data based on one or more of: a physical characteristic and / or constraint associated with one or more of the first and second interacting agents respectively; a behavioural characteristic and / or constraint associated with one or more of the first and second interacting agents respectively; an environmental constraint influencing one or more of the first and second interacting agents; a characteristic associated with one or more of the first and second interacting agents; and actions taken by one or more of the first and second interacting agents respectively in one or more previous frames.
[0018] Optionally, the first model data comprises a set of behaviour distributions over time and space for the first interacting agents; and / or the second model data comprises a set of behaviour distributions overtime and space for the second interacting agents.
[0019] Optionally, the simulated scene data represents scene data that would be received by the second POV agent at the first time. This provides scene data having a different perspective(or POV) to the received scene data and therefore improves the prediction of future behaviour of the agents in the scene.
[0020] Optionally, the processor is further configured to determine an initial control instruction based on the first model data; and update the initial control instruction based on the second model data.
[0021] Optionally, the receiver is further configured to receive a second frame of scene data representing a scene in a vicinity of the vehicle at a second time, and wherein the processor is further configured to undertake a further iteration of the steps i-vi based on the second frame of scene data, for determining a further control instruction.
[0022] Optionally, the control instruction and the further control instruction each comprises a plurality of candidate control instructions, each candidate control instruction having a probability of being the optimal control instruction.
[0023] Optionally, determining the further control instruction comprises generating updated probabilities of the candidate control instructions based on the second frame of scene data, and optionally also based on the first frame of scene data.
[0024] Optionally, the receiver is configured to receive a plurality of frames of scene data over a period of time, and wherein the computer processor is further configured to: determine an intent of one or more of the plurality of agents based on the plurality of frames of scene data, the intent being indicative of the one or more agents’ expected behaviour over a plurality of subsequent frames; and determine the control instruction based on the determined intent of the one or more of the plurality of agents.
[0025] Optionally, a plurality of possible intents each comprise a statistical model corresponding to agent behaviour over a period of time, and wherein the computer processor is further configured to: determine agent behaviour based on the plurality of frames of scene data; compare the determined agent behaviour to the statistical models of the plurality of possible intents; and determine the intent based on the comparison.
[0026] Optionally, the period of time is less than or equal to 2 seconds.
[0027] Optionally, the computer processor is configured to implement a Symbolic Al algorithm, and is further configured to: define one or more of the plurality of agents as a symbol of theSymbolic Al algorithm, wherein each symbol comprises a continuous probability distribution over time; and determine the first and / or second model data based on the continuous probability distributions.
[0028] Optionally, the continuous probability distributions model behaviour of the corresponding agent, based on one or more of: an influence of other agents, a physical constraint of the agent, a behavioural constraint of the agent and / or a characteristics of the agent.
[0029] Optionally, the processor is further configured to determine one or more errors in the first frame of scene data based on the first and / or second model data.
[0030] Optionally, the processor is further configured to apply a correction to the one or more determined errors in the second frame of scene data.
[0031] Optionally, the one or more further agents comprise one or more objects in the vicinity of the vehicle capable of impacting the control instruction for the vehicle.
[0032] Optionally, the one or more further agents comprise one or more of: a further vehicle; a pedestrian and an item of road infrastructure.
[0033] Optionally, the control instruction comprises data for a actuation of a control input of the vehicle, and wherein the processor is further configured to transmit the control instruction to the control input of the vehicle.
[0034] Optionally, the processor is further configured to cause actuation of a control input of the vehicle, based on the control instruction.
[0035] Optionally, the processor may generate simulated future scene data based on the first and second model data. This may be done based on the updated first model data.
[0036] Optionally, the processor may be configured to run a further iteration of the steps identified above based on the simulated future scene data. Further future scene data may be determined based on the first and / or second model data determined in the further iteration. This process may be repeated a plurality of times to generate a virtual video clip of the scene into a simulated future. Optionally, the control instruction may be determined based on the virtual video clip.According to the invention in an aspect, there is provided a method for determining a control instruction for a vehicle, the method comprising:
[0037] receiving, by a receiver, a first frame of scene data representing a scene in a vicinity of the vehicle at a first time, the first scene data comprising: a plurality of agents including an ego agent representing the vehicle and at a plurality of further agents; and spatial coordinates for each of the plurality of agents; setting the ego agent as a first point of view, POV, agent, and the further agents as first interacting agents; determining, based on the received scene data, first model data predicting future behaviour of the first interacting agents; setting one of the further agents as a second POV agent, and the ego agent and remaining further agents as second interacting agents; generating, based on the received scene data, simulated scene data from the POV of the second POV agent; determining, based on the simulated scene data, second model data predicting future behaviour of the second interacting agents; and determining a control instruction for the vehicle, based on the first model data and the second model data.
[0038] According to the invention in an aspect, there is provided an apparatus for predicting future behaviour of one or more agents (e.g. vehicles and / or pedestrians) in a scene. The apparatus comprises: a receiver configured to receive a first frame of scene data representing a scene in a vicinity of the vehicle at a first time, the first scene data comprising: a plurality of agents including an ego agent representing the vehicle and a plurality of further agents; and spatial coordinates for each of the plurality of agents; a computer processor configured to execute computer program code to control the apparatus to: set the ego agent as a first point of view, POV, agent, and the further agents as first interacting agents; determine, based on the received scene data, first model data predicting future behaviour of the first interacting agents; set one of the further agents as a second POV agent, and the ego agent and remaining further agents as second interacting agents; generate, based on the received scene data, simulated scene data from the POV of the second POV agent; determine, based on the simulated scene data, second model data predicting future behaviour of the second interacting agents; and determine predicted future behaviour of the first interacting agents, based on the first model data and the second model data.
[0039] According to the invention in an aspect, there is provided a method for predicting future behaviour of one or more agents (e.g. vehicles and / or pedestrians) in a scene, the method comprising: receiving, by a receiver, a first frame of scene data representing a scene in a vicinity of the vehicle at a first time, the first scene data comprising: a plurality of agents including an ego agent representing the vehicle and at a plurality of further agents; and spatial coordinates for each of the plurality of agents; setting the ego agent as a first point of view,POV, agent, and the further agents as first interacting agents; determining, based on the received scene data, first model data predicting future behaviour of the first interacting agents; setting one of the further agents as a second POV agent, and the ego agent and remaining further agents as second interacting agents; generating, based on the received scene data, simulated scene data from the POV of the second POV agent; determining, based on the simulated scene data, second model data predicting future behaviour of the second interacting agents; and determining predicted future behaviour of the first interacting agents, based on the first model data and the second model data.
[0040] According to the invention in an aspect, there is provided a vehicle comprising any apparatus disclosed herein. Optionally, the processor is further configured to cause actuation of a control input of the vehicle, based on the updated control instruction.
[0041] According to the invention in an aspect, there are provided computer program products including computer program code configured, when executed on a computer processor, to control a data processor to undertake the steps of any method described herein.
[0042] Brief description of the drawings
[0043] Embodiments of the disclosed methods and apparatus will be described in detail below, with reference to the accompanying drawings, in which:
[0044] Figure 1 is a schematic representation of an apparatus for determining a control instruction for a vehicle;
[0045] Figure 2 is a schematic representation of an exemplary scene in the vicinity of a vehicle; Figure 3 is a flow chart showing a method of determining a control instruction for a vehicle; and
[0046] Figure 4 is a flow chart showing a method of determining a control instruction for a vehicle.
[0047] Detailed Description
[0048] Generally, disclosed herein are methods and apparatus for determining future predicted behaviour of agents (e.g. vehicles and / or pedestrians) in a scene, and then optionally a control instruction for a vehicle (e.g. an ego vehicle). The determined control instruction may be used for autonomous or semi-autonomous control of a vehicle’s operation during driving. In exemplary arrangements, a control instruction may include data transmitted to one or more apparatus controlling a vehicle and the apparatus may optionally cause at least partial controlof the vehicle (e.g. it’s acceleration, velocity and / or direction) based on the control instruction. In exemplary arrangements, e.g. those that utilise levels of autonomous driving lower than level 5, the control instruction may include instructions to a user of the vehicle to input a control instruction themselves. For example, the control instruction may provide a warning to the user to brake.
[0049] Exemplary methods and apparatus utilise SAI to model, from the point of view (POV) of the vehicle, behaviour of a plurality of agents in the vicinity of the vehicle that may be interacted with by the vehicle (interacting agents). Some methods and apparatus then use SAI to determine a similar model from the POV of one or more of the interacting agents. A control instruction may then be determined for the ego vehicle based on both sets of modelled data.
[0050] By modelling behaviour of agents in a scene from the point of view of the interacting agents, a more accurate model of agent behaviour can be determined. As an example, a first interacting agent that may be viewed from the POV of the vehicle might influence the modelled behaviour of a second interacting agent that may not actually be able to see the first interacting agent as, from the second interacting agent’s POV, the first interacting agent is obstructed.
[0051] Exemplary methods and apparatus are arranged to define each symbol in an SAI framework to be an agent of the scene data that is relevant in the context of driving, e.g. a car, a bike, a van or truck or a pedestrian etc. The behaviour of each agent may then be modelled as a discrete set of continuous distributions over time and space. It is noted that not all elements of a scene will need to be modelled, for example static items that do not change like road lanes.
[0052] In exemplary arrangements, the distributions may take into account the influence of other agents, physical constraints, behavioural constraints and / or characteristics of one or more agents. The set of symbols and their distributions is such that it covers all possible actions, in an optimally sparse representation that makes it possible to calculate all the actions in real time. The parameters of each distribution, and their relative frequencies, may be learnt from real world data on driving.
[0053] The methods described herein may be recursive and may be implemented sequentially on multiple frames of scene data, each frame representing scene data at a different snap shot in time. In exemplary methods and apparatus, frame data may be received at 10 frames per second, 30 frames per second or 60 frames per second.Figure 1 shows a schematic representation of an exemplary apparatus 100 for determining a control instruction for a vehicle. The apparatus 100 may comprise part of the ego vehicle, or may be external to the ego vehicle, for example at a remote computing location or forming part of a separate vehicle or as a part of road infrastructure, such as being mounted at a road intersection.
[0054] The exemplary apparatus 100 comprises a transmitter 102 and a receiver 104. The transmitter 102 and receiver 104 may be in data communication with other entities, such as the one or more sensors located on the vehicle or servers and / or functions in a telecommunications network, and are configured to transmit and receive data accordingly.
[0055] The apparatus 100 further comprises a memory 106 and a processor 108. The memory 106 may comprise a non-volatile memory and / or a volatile memory. The memory 106 may have a computer program 110 stored therein. The computer program 110 may be configured to undertake all or part of the methods disclosed herein. The computer program 110 may be loaded in the memory 106 from a computer program product such as a non-transitory computer readable medium 112, on which the computer program is stored. The apparatus 100 may also comprise one or more sensors (not shown) that are configured to record data for inclusion in scene data that describes one or more agents in the vicinity of the ego vehicle. The processor 108 is configured to undertake one or more of the functions necessary for operation of one or more of the elements of the apparatus 100.
[0056] Each of the transmitter 102 and receiver 104, memory 106 and processor 108 is in data communication with the other features of the apparatus 100. The apparatus 100 can be implemented as a combination of hardware and software. In particular, software may be configured to run on the processor 108 and undertake the function of a behaviour modeller 114 and / or a vehicle controller 116. The memory 106 stores the various programs / executable files that are implemented by the processor 108, and also provides a storage unit for any required data.
[0057] Figure 2 shows a schematic representation of an exemplary scene comprising an ego agent 200 (i.e. the ego vehicle) and a plurality of further agents 202a-e. In the example shown, further agents 202a and 202b are cars, further agent 202c is a pedestrian, further agent 202d is a pedestrian crossing and further agent 202e are the lines marking the extent of the lane or road that the ego agent is driving along. From the POV of the ego agent 200, one or more of the further agents 202a-e may be considered interacting agents.An interacting agent encompasses a feature of the scene that may influence the driving of a POV vehicle. Accordingly, an interacting agent may be a feature that will interact with the POV agent or be interacted with by the POV agent 200. This includes vehicles, pedestrians and street furniture (including road markings). It will be appreciated that other interacting agents may be alternatively or additionally present in the scene.
[0058] Scene data may comprise data describing the scene and may be provided to the apparatus 100 by a perception layer of a vehicle computer system. The perception layer may receive raw sensor data recorded by one or more sensors that may be located on or be external to the ego vehicle 200, and determine the scene data based on the raw sensor data. The scene data may include one or more of the following:
[0059] • A plurality of agents, which may include the ego agent
[0060] • A classification of each agent (e.g. car, pedestrian, bike, bus lamp post, lane markings etc.)
[0061] • A bounding box that specifies the size of the object, optionally including any parts hidden from the POV of the ego vehicle 200
[0062] • Relative spatial coordinates (e.g. a distance and direction from the ego vehicle 200)
[0063] The scene data may optionally include motion data for each agent, for example acceleration, speed / velocity, direction and / or whether a change in direction is occurring. The motion data may be determined from raw sensor data that is capable of measuring motion. The motion data may be determined based on a change in location of an agent over a plurality of frames of scene data.
[0064] Scene data may include or consist of simulated data. In arrangements, scene data may include a correction that has been applied to raw sensor data based on a previous output from the methods disclosed herein, for example in Figure 3. This is discussed in greater detail below.
[0065] Figure 3 shows a flow diagram of an exemplary method for autonomous control of a vehicle. It is noted that not all the steps shown in Figure 3 are essential.
[0066] Sensor data is recorded 300. The sensor data comprises data that describes one or more objects in the vicinity of the ego vehicle 200 (the scene) at a moment in time. The sensor data may be recorded as part of the exemplary methods described herein, or may be separatelyrecorded and passed to a computer processor of the invention that may be configured to process it to determine a control instruction for the ego vehicle 200. Accordingly, the apparatus 102 may include one or more sensors that may be under the control of the processor 108. The sensor data may comprise camera data, LIDAR data, GNSS data and other data that will be apparent to a skilled person. The sensor data is typically indicative of at least the size and spatial location of each object in the scene relative to the ego vehicle 200.
[0067] Scene data is determined 302 based on the sensor data. This may be undertaken by the apparatus 100, for example using the processor 108, although it may alternatively be undertaken, at least in part, by a further apparatus and transmitted to the apparatus 100. The scene data defines at least: a plurality of agents in the scene; and a relative spatial position of each agent. The scene data may additionally include a classification for each agent, such as vehicle, type of vehicle, pedestrian etc. Other information may also be included in the scene data, such as acceleration, speed / velocity, direction and whether an agent is engaged in a manoeuvre, such as pulling away from a junction, crossing a road or turning a corner.
[0068] It is noted that new sensor data may be recorded at different time intervals, which may be regular time intervals. Each set of sensor data may be used to determine a frame of scene data. Each frame may be a snapshot of the scene at a particular time interval. It is also noted that scene data may include information and / or data that has been determined by the apparatus based on one or more previous frames of scene data.
[0069] The scene data is received, e.g. by the receiver 104. The processor 108 sets an agent to be the point of view (POV) agent. The POV agent is the agent in the scene from whose POV the actions of other agents in the scene will be modelled.
[0070] Typically, in a first iteration of the method shown in Fig. 3, the POV agent will be the ego agent 200 and one or more of the further agents 202a-e are set to be the first interacting agents. In one or more subsequent iterations, the POV agent will be set to be one of the further agents 202a-e and the ego agent 200 becomes one of the interacting agents. That arrangement is discussed below, although it will be appreciated that the invention need not be limited to such an arrangement. It is noted that in some implementations, the term agent may include only features of the scene that have agency, such as vehicles (through their drivers) and pedestrians. Street furniture and static, inanimate items may be treated as mere features of the scene.It is determined at step 306 whether the agent that has been set to be the POV agent is the ego agent 200. In a first iteration of the method, the ego agent 200 is typically the POV agent and so the method moves to step 308 in which the modeller 114 determines first model data that predicts or is indicative of future behaviour of the POV agent and / or at least one of the interacting agents (in this case the further agents 202a-e) from the POV of the POV agent (in this case the ego agent 200). As mentioned above, the scene data defines the plurality of further agents 202a-e in the scene relative to the ego agent 200. Therefore the first model data can be determined directly from the received scene data when the POV agent is the ego agent 200.
[0071] The process of determining the model data is described in further detail below with reference to Figure 4.
[0072] At step 312, it is determined whether modelling is to be undertaken from the POV of any other agents of the plurality of agents in the received scene data.
[0073] In this exemplary case, the answer is yes. Therefore, the method returns to step 304 and one of the further agents 202a-e is set to be the POV agent and the ego agent 200 and remaining further agents are set to be the interacting agents. For illustrative purposes only, the example described sets the agent representing the car 202a as the POV agent. Accordingly, the remaining agents, which become second interacting agents from the POV of the car 202a, include the ego agent 200 (which is no longer the POV agent) and the agents 202b-e.
[0074] At step 306, it is determined that the POV agent is not the ego agent 200 so the method proceeds to step 310 and the processor 108 generates simulated scene data. The simulated scene data is a representation of scene data from the POV of the car 202a, rather than the POV of the ego agent 200. The simulated scene data is determined based on the scene data obtained at step 302. Because the received scene data includes spatial coordinates for all agents, it is possible to generate scene data from the POV of any of the further agents based on the received scene data. The simulated scene data may include the data parameters that is described above in relation to the determined or received scene data, e.g. classification, bounding box, spatial coordinates and motion data.
[0075] The method proceeds to step 308 and the modeller 114 determines second model data that is indicative of future behaviour of at least one of the interacting agents from the POV of the second POV agent 202a. On this second iteration of the method, the second set of interacting agents comprise the ego agent 200 and the agents 202b-e.The simulated scene data gives a further perspective from which to determine model data. For example, one or more agents that are visible from the POV of the ego vehicle 200 may be obscured in the simulated data from the POV of the corresponding agent, or vice-versa. Further, the interaction between the second interacting agents and / or the between the POV vehicle and the second interacting agents may lead to different predicted future behaviour. Incorporating into the method model data that has been determined based on the simulated scene data increases the accuracy of the predicted future behaviour of the agents in the scene.
[0076] Based on the first and second model data (and optionally any other determined model data from the POV of other agents), the controller 116 may determine future predicted behaviour of the agents in the scene and / or a vehicle control instruction. In arrangements, the second model data may be used to update the first model data. More accurate first model data is thereby generated. The updated first model data may be used to determine the control instruction. In arrangements an initial control instruction may be determined based on the first model data and updated based on the second model data. In other arrangements, the first model data may be updated based on the second model data.
[0077] Modelling from the POV of the further agents in the scene data may be fed back into the determination of the control instruction and allows for greater accuracy of predicted future behaviour of the further agents 202a-e.
[0078] Further iterations of the method may be undertaken for as many of the further agents 202a-e as desired.
[0079] The vehicle control instruction may be transmitted 316 to a further apparatus. The apparatus 100 and the further apparatus may each be on the vehicle. The vehicle may also be controlled 318, e.g. by actuation of one or more vehicle controls, based on the vehicle control instruction. Steps 316 and 318 need not form part of all methods and apparatus disclosed herein. It is noted that the control instruction may additionally or alternatively be output to a user of the vehicle as an instruction to the user to actuate a control input of the vehicle.
[0080] In arrangements in which the model data is updated, step 314 may comprise determining a control instruction from the updated model data.Figure 4 shows a flow diagram of a method for modelling future behaviour of one or more agents based on scene data.
[0081] Each of the plurality of agents is defined 400 as a symbol in an SAI framework. The symbols may correspond to the classifications for each agent or may be one of a subset of symbols for each classification. For example, further agents 202a & 202b may have a classification of vehicle and may have symbols of the type ‘car’. Agent 202c may have a classification and a symbol of type ‘pedestrian’. Agent 202d may have a classification of type infrastructure and a symbol of type ‘pedestrian crossing’. Agent 202e may have a classification of type infrastructure and a symbol of type ‘lane’.
[0082] For one or more of the defined symbols, future behaviour may be modelled by determining 402 a discrete set of continuous symbol distributions over time and space. In embodiments, the symbol distributions are determined based on known physical and or behavioural constraints that are predetermined for each symbol type.
[0083] Symbol distributions encompass the probability that a parameter of a symbol will change over time and to what degree. For example, a symbol distribution may be determined for the velocity of a vehicle and how that velocity may change in the next frame. Similar distributions may be determined for other parameters, such as direction, acceleration, speed, position etc.
[0084] Physical constraints are absolute and encompass physically possible or impossible scenarios relating to each symbol. For example, two different vehicles cannot overlap. However, a vehicle can overlap with a lane or a pedestrian crossing. Typically, physical constraints are known and do not need to be learnt.
[0085] Behavioural constraints (or behavioural distributions) encompass scenarios that are possible for a particular symbol and how likely they are to occur. Behavioural constraints may be preprogrammed and / or learnt using real world or simulated data. Each behavioural constraint may comprise a probability distribution of scenarios. For example, it is less likely that a pedestrian is travelling at more than around 6 km / h than at 4 km / h, and it is very unlikely that a pedestrian is travelling at 15 km / h. As another example, if there is a stop sign, there is a high probability that a car will stop at the corresponding line before it. The behavioural constraints may show how likely that is (it is not 100% as some cars would not stop) and / or how it is typically done (e.g. slow down at a constant pace over 40 m leading up to the line).Behavioural constraints refer to the way an agent is most likely to behave in certain situations and / or based on previous behaviour (e.g. in one or more previous frames of scene data). The behavioural constraints of each symbol may be learnt from historical real world driving data and represented by a behavioural statistical model.
[0086] For example, a pedestrian typically moves in a certain way, is constrained based on physical limitations (physical constraint) and conforms to statistical characteristics of human behaviour (behavioural constraint). Historical data showing aspects of a plurality of pedestrians’ actions allows the physical and / or behavioural constraints of a generalised pedestrian to be represented as a behavioural statistical model.
[0087] A car would also have known constraints, but they would be different to a pedestrian’s. For example, the most common behaviour of a car could be that it follows the lane it is currently in, aiming to remain in the middle of the lane and aiming to go as fast as the speed limit and environmental constraints such as other agents and / or scene features allow.
[0088] The concepts of “following the lane”, “aiming to remain in the middle”, “aiming to go as fast as the speed limit” are non-deterministic concepts that nevertheless conform to a statistical model (behavioural statistical model or distribution). Therefore, a car may be more likely to be 50 cm away from the middle of the lane than 2 m away. This may be represented by a spatial distribution. If a car is currently driving at 38 km / h and the speed limit is 50km / h, the car may be more likely to be driving at 40 km / h in the next frame than at 22 km / h. This may be represented by a temporal distribution.
[0089] The symbol distributions may be determined based on the behavioural distribution of the respective symbol type, the physical constraints and / or the scene data, which describes the current situation.
[0090] The physical and / or behavioural constraints may include a knowledge of the physical world, the world of human behaviour as it relates to driving, and / or the rules of the road. For example, a behavioural distribution describing how the speed of a car might change could be affected by an obstacle in the car’s way, but is also affected by human reaction times to the obstacle, and by traffic laws.
[0091] For the physical constraints of a symbol, exemplary arrangements use the concept of a “dynamic occupancy grid” to represent where physical objects lie in 2D or 3D space. Thebehavioural constraints may be modelled directly into the parameters of the behavioural distributions.
[0092] The first and / or second model data may be further determined based on one or more environmental constraints. These encompass factors external to the agents that affect the behaviour of the agents. For example, environmental constraints may include weather, the time of day, visibility, physical objects that are not agents etc.
[0093] Referring back to Figure 3, when determining the model data 308, the symbol distribution may be determined 402 for each interacting (non-POV) agent in the scene using the scene data (or simulated POV scene data), the physical constraints and / or the behavioural distributions. The scene data describes the situation of each interacting agent and the behavioural distributions describe the possible actions of each agent in the described situation and the probability of those actions.
[0094] Having determined the symbol distributions of one or more interacting agents, in exemplary arrangements, the modeller 114 may predict 404 future behaviour of the one or more interacting agents based on the determined symbol distributions. This may be done for each interacting agent independently. The future behaviour may include one or more of speed, direction, position and acceleration. The future behaviour may comprise predicted scene data. The future behaviour may be predicted one or more (typically a plurality) of frames forward in time.
[0095] The prediction for each interacting agent may be revised after every frame of scene data or after a number of frames of scene data. Therefore, exemplary arrangements may take into account actual observations (identified by a subsequent frame of scene data) where the subsequent frame of scene data differs from a previous prediction due to the statistical nature of the predictions. That is, future behaviour may be predicted, say, 20 frames forward in time in a first iteration. After receipt of a subsequent frame of scene data, the prediction is determined again. In many cases, the subsequent frame of received scene data does not correspond exactly to the predicted frame of scene data. Accordingly, the next 20 frames of predicted scene data are based on the subsequent received frame of scene data.
[0096] In exemplary arrangements, one or more “characteristics” of an agent may be defined as a finite set of attributes that map the agent’s behaviour onto the symbol distributions. These characteristics may be deduced from the predictions made at step 404, and the revisions based on new observations in subsequent frames of scene data. That is, the actions of anagent in one or more previous frames may be used to determine the first and / or second model data. Using, for example, Bayesian inference, the modeller 114 can assign implied characteristics to all agents. The determined characteristics of an agent’s behaviour may be used to correct future behaviour distributions for the agent.
[0097] To take a simple example, for a car that is consistently driving much slower than expected over a plurality of frames of scene data, the modeller 114 can deduce that it will likely continue to drive slower than expected, and we can now expect it to drive in a way that is consistent with the way it has been driving so far. It may be assigned the characteristic of being ‘slow’. Therefore, the behaviour distribution will model the speed of the car as learnt based on real world data, and this will be modified based on the characteristic that the vehicle is slower than a typical car. The characteristic of an agent may comprise an absolute value relating to the metric being predicted. In the example above this may, for example be that the car is 5 km / h slower, 10 km / h etc.
[0098] In exemplary arrangements, errors in perception data (used to determine the scene data) may be corrected based on the characteristics. For example, a car that is reported by the perception layer to be in the wrong location can be corrected to its true location by the probabilistic model that assumes its motion follows known statistical rules. The perception layer often misses objects in some frames for no reason that a human can see due to its machine learning probabilistic model. The perception layer may additionally or alternatively make attribute errors, for example saying that a car is closer than it really is, that two adjacent cars are a bus, that a bike is a pedestrian, and so on. This may happen in some frames but not in others.
[0099] Exemplary arrangements may correct errors in scene data based on the first and / or second model data.
[0100] In one example, it is highly unlikely that a car is at position X in a first frame, position X+50 m in the next frame, and position X+1 m in the third. Exemplary arrangements may determine the most likely scenario is that the car was at X, X+0.5 m and X+1 m, that the perception system had made a mistake and subsequently correct the mistake. In another example, a pedestrian moving at high speed in the middle of the road is a much less likely than to assume that perception had misclassified a bike.As before, the first model data may be updated every frame or number of frames, to correct for differences seen between predicted behaviour and subsequent scene data, or where multiple candidate models were possible and the correct model can be chosen only later.
[0101] During a first iteration of the modelling, using the ego agent 200 as the POV agent, exemplary arrangements predict the future behaviour of the ego agent and all interacting agents (in this case the further agents 202a-e).
[0102] In a second iteration of the modelling, the POV agent is changed to be one of the further agents 200a-e and the ego agent 200, in effect, becomes an interacting agent along with the remaining further agents. Exemplary arrangements then predict the future behaviour of the second set of interacting agents (in this case the ego agent 200 and the remaining further agents 202a-e). This prediction can then be used to update the prediction made during the first iteration.
[0103] By changing the POV of the modelling, a recursive technique is applied that improves the accuracy with which the future actions of each agent in the scene data can be predicted. The modelling may be undertaken from the POV of all agents in the scene whose behaviour is to be predicted. More accurate predictions of future behaviour of agents allows the determination of more accurate control instructions by the controller 116. It is noted that the determination of simulated scene data from the POV of a further agent in the scene is different to determine the likely actions or interactions of agents from the POV of the ego agent. The simulated scene data comprises spatial coordinates, speeds, directions and / or accelerations of one or more agents that have been determined from the POV of a further agent. This perspective is not possible from the POV of the ego agent and allows for greater accuracy.
[0104] In exemplary arrangements, the modeller 114 is further configured to determine an intent of one or more agents in the scene. In exemplary cases discussed, this includes one or more of the further agents 202a-e. As used herein, the term “intent” encompasses one of a finite set of possibilities that describe what an agent is trying to do overtime, as opposed to in the next frame. The intent may be determined at a higher level than the immediate future of the next frame.
[0105] For example, while in the next frame the agent could be moving 1 m ahead of its current position, its intent, as determined over a longer period of time, could be to keep driving at a constant speed, or to slow down and aim to stop before a stop sign, or to change lanes and accelerate.In exemplary arrangements, the modeller may determine an intent based on a plurality of frames of scene data, e.g. <0.5 seconds of frames of scene data, <1 second of frames of scene data, <2 seconds of frames of scene data or <3 seconds of frames of scene data.
[0106] Each of a plurality of intents may be defined by a statistical model. Based on a plurality of frames of received scene data, the behaviour of an agent may be compared to each of the statistical models to determine the intent of an agent. The intent may be that which has a statistical model most closely matching the agent’s behaviour.
[0107] Exemplary arrangements may use Bayesian inference to assign probabilities to the different intents based on the received scene data. The probability assigned to each intent will change overtime. Typically, the probability of the interacting agents’ actual intent approaches 1 and the probability of competing intents approaches 0 overtime as more frames of scene data are processed.
[0108] Exemplary arrangements use Sequential Probability Ratio Testing (SPRT) to track the probabilities of each intent and / or to determine when sufficient confidence of an agent’s intent is achieved. This may be done by determining when the probability of a particular intent exceeds a threshold value. While the probability of none of the intents has reached the SPRT threshold, it may be assumed that there are multiple possible intents for an agent. Further frames of scene data are received until the probability of one intent exceeds the threshold.
[0109] The determined intent for the interacting agents may be used to create a coherent prediction of all agents and their interactions into the near future, beyond the next frame. Exemplary arrangements may predict in a range from 1 second to 10 seconds into the future, or in a range from 2 seconds to 8 seconds into the future. Typically, these timings are relevant for driving decisions, but this can vary.
[0110] A control instruction that is determined based on the scene data, as discussed above, may be updated based on the determined intent. This may be done by using the intent to predict agent behaviour further into the future. In this way, the control instruction may be updated to take action early enough to ensure safe manoeuvres.
[0111] For example, based on the scene data, it may be determined that the vehicle 202a will slow down by a particular amount in the next frame and the determined control instruction may correspond to the ego vehicle 200 slowing down by the same amount. However, the intent ofthe vehicle 202a may be to slow down to a stop behind the vehicle 202b while the pedestrian 202c crosses at the crossing 202d. This may require the control instruction to be updated to slow the vehicle more than would be done based on the scene data alone. Knowing that the ego vehicle 200 will need to stop behind the vehicle 202a may result in the control instruction to be updated to increase deceleration now, understanding that a full stop may be required soon.
[0112] After every frame of scene data is received, it may be processed as described herein to determine agent data. The agent data may include a plurality of further agents 202a-e in the scene, the location of each further agent 202a-e, the velocity of each further agent 202a-e, the acceleration / deceleration of each further agent 202a-e, the direction of each further agent 202a-e, and / or the intent of each further agent 202a-e. The agent data may also include the same data for the ego vehicle 200, which may be determined using the same method or using data from the ego vehicle 200 itself.
[0113] The modeller 114 may determine the model data, which comprises a model of how the scene will evolve in the future based on the agent data. The modeller 114 may predict the behaviour of the agents into an imagined future, frame by frame, and simulate how the model expects them to behave in the real world.
[0114] In exemplary arrangements, the modeller 114 may predict the scene in a plurality of subsequent frames, which may encompass time into the future in a range from 1-10 seconds, 2-8 seconds or 2-5 seconds. Accordingly, a ‘virtual video clip’ of how the scene will play out may be determined. The virtual clip may include all variations based on statistical probabilities relating to each agent and .
[0115] In the example of Figure 2, as the pedestrian 202c crosses the crossing 202d, they will be in the way of the cars 202a, 202b that are ahead of the ego vehicle 200. Running the scene into the imagined future by determining the virtual clip can identify whether and when the cars 202a, 202b are likely to slow down, whether the ego vehicle 200 will be too close to them unless it takes action, and / or whether it is the pedestrian 202c who will stop, and the cars 202a, 202b (and the ego vehicle 200) can keep going.
[0116] It is noted that the behavioural and physical constraints, and / or characteristics deduced for each agent may be used to determine the virtual clip. The modeller 114 can predict the virtual clip based on the modelled behaviour of each agent based on an agent’s symbol distributions, its intent and its characteristics. Using the same simple example, if one of the cars 202a,202b has the characteristics of a slow driver, then once the pedestrian 202c is out of the way and the road is clear, the modeller 114 may determine that the car 202a, 202b will resume its slow speed, not the full speed of an average or generalised car.
[0117] The virtual video clip may be determined to incorporate the intents of a plurality of the agents in the scene. The computer processor may determine a driveable space and / or occupancy grid for the scene into the future based on the virtual video clip. The processor may determine the control instruction based on the driveable space and / or occupancy grid.
[0118] The above allows the modeller 114 to build a full prediction of all further agents 202a-e and their interactions into the relevant future. This includes further agents’ 202a-e interactions with the ego vehicle 200.
[0119] Forthe present frame, and for each of the imagined future frames, the controller 116 calculates the demands that the situation imposes on the ego vehicle 200. The calculated future demands may be difficult or even impossible to fulfil. In the above simple example, if the cars ahead 202a, 202b are expected to slow down, the ego vehicle 200 will need to slow down too, to avoid an accident and to keep within a safe distance. It may be that at the ego vehicle’s current speed and given its capabilities, it will not be able to slow down enough in the future to avoid hitting the car ahead whose predicted speed will be much slower.
[0120] For each of these future demands, the controller 116 may back-cast it into a demand at the time of the present frame. Thus, a need to stop in the future could imply determining a control instruction to slow down based on the current frame. In exemplary arrangements, the controller may select an immediate demand that accommodates all the implications of future demands. In the example, if in future the modeller 114 predicts that the ego vehicle 200 will have to stop for the car ahead 202a, which is predicted to stop for the pedestrian 202c, the controller may determine a control instruction to decelerate now at a pace that will achieve a full stop when predicted to be needed.
[0121] Some or all of the methods disclosed herein may be repeated after every frame, and from the real point of view of the ego vehicle 200 and the imagined points of view of every further agent. This may include further agents that were present in previous scene data but are no longer visible in the current frame.
[0122] A computer program may be configured to provide any of the above described methods. The computer program may be provided on a computer readable medium. The computer programmay be a computer program product. The product may comprise a non-transitory computer usable storage medium. The computer program product may have computer-readable program code embodied in the medium configured to perform the method. The computer program product may be configured to cause at least one processor to perform some or all of the method.
[0123] Various methods and apparatus are described herein with reference to block diagrams or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).
[0124] Computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks.
[0125] A tangible, non-transitory computer-readable medium may include an electronic, magnetic, optical, electromagnetic, or semiconductor data storage system, apparatus, or device. More specific examples of the computer-readable medium would include the following: a portable computer diskette, a random access memory (RAM) circuit, a read-only memory (ROM) circuit, an erasable programmable read-only memory (EPROM or Flash memory) circuit, a portable compact disc read-only memory (CD-ROM), and a portable digital video disc readonly memory (DVD / Blu-ray).
[0126] The computer program instructions may also be loaded onto a computer and / or other programmable data processing apparatus to cause a series of operational steps to beperformed on the computer and / or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions / acts specified in the block diagrams and / or flowchart block or blocks.
[0127] Accordingly, the invention may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor, which may collectively be referred to as “circuitry,” “a module” or variants thereof.
[0128] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated.
[0129] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed systems and methods. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed systems and methods. It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.
Claims
CLAIMS1. An apparatus for determining a control instruction for a vehicle, the apparatus comprising:a receiver configured to receive a first frame of scene data representing the scene in a vicinity of a vehicle at a first time, the first scene data comprising:a plurality of agents including an ego agent representing the vehicle and a plurality of further agents; andspatial coordinates for each of the plurality of agents;a computer processor configured to execute computer program code to control the apparatus to:i. set the ego agent as a first point of view, POV, agent, and the further agents as first interacting agents;ii. determine, based on the received scene data, first model data predicting future behaviour of the first interacting agents;Hi. set one of the further agents as a second POV agent, and the ego agent and remaining further agents as second interacting agents;iv. generate, based on the received scene data, simulated scene data from the POV of the second POV agent;v. determine, based on the simulated scene data, second model data predicting future behaviour of the second interacting agents; andvi. determine a control instruction for the vehicle, based on the first model data and the second model data.
2. The apparatus according to claim 1, wherein the processor is further configured to update the first model data based on the second model data, wherein the control instruction is determined based on the updated first model data.
3. The apparatus according to claim 1 or 2, wherein the processor is further configured to determine the first and / or second model data based on one or more of:a physical constraint associated with one or more of the first and second interacting agents respectively;a behavioural constraint associated with one or more of the first and second interacting agents respectively;an environmental constraint influencing one or more of the first and second interacting agents;24a characteristic associated with one or more of the first and second interacting agents; andactions taken by one or more of the first and second interacting agents respectively in one or more previous frames.
4. The apparatus according to any preceding claim, wherein the first model data comprises a set of behaviour distributions over time and space for the first interacting agents; and / orthe second model data comprises a set of behaviour distributions overtime and space for the second interacting agents.
5. The apparatus according to any preceding claim, wherein the simulated scene data represents scene data that would be received by the second POV agent at the first time.
6. The apparatus according to any preceding claim, wherein the processor is further configured to determine an initial control instruction based on the first model data; and update the initial control instruction based on the second model data.
7. The apparatus according to any preceding claim, wherein the receiver is further configured to receive a second frame of scene data representing a scene in a vicinity of the vehicle at a second time,and wherein the processor is further configured to undertake a further iteration of the steps i-vi based on the second frame of scene data, for determining a further control instruction.
8. The apparatus according to claim 7, wherein the control instruction and the further control instruction each comprises a plurality of candidate control instructions, each candidate control instruction having a probability of being the optimal control instruction.
9. The apparatus according to claim 8, wherein determining the further control instruction comprises generating updated probabilities of the candidate control instructions of the control instruction based on the second frame of scene data, and optionally also based on the first frame of scene data.
10. The apparatus according to any preceding claim, wherein the receiver is configured to receive a plurality of frames of scene data over a period of time, and wherein the computer processor is further configured to:determine an intent of one or more of the plurality of agents based on the plurality of frames of scene data, the intent being indicative of the one or more agents’ expected behaviour over a plurality of subsequent frames; anddetermine the control instruction based on the determined intent of the one or more of the plurality of agents.
11. The apparatus according to claim 10, wherein a plurality of possible intents each comprise a statistical model corresponding to agent behaviour over a period of time, and wherein the computer processor is further configured to:determine agent behaviour based on the plurality of frames of scene data; compare the determined agent behaviour to the statistical models of the plurality of possible intents; anddetermine the intent based on the comparison.
12. The apparatus according to claim 10 or 11, wherein the period of time is less than or equal to 2 seconds.
13. The apparatus according to any preceding claim, wherein the computer processor is configured to implement a Symbolic Al algorithm, and is further configured to:define one or more of the plurality of agents as a symbol of the Symbolic Al algorithm, wherein each symbol comprises a continuous probability distribution overtime; and determine the first and / or second model data based on the continuous probability distributions.
14. The apparatus according to claim 13, wherein the continuous probability distributions model behaviour of the corresponding agent, based on one or more of: an influence of other agents, a physical constraint of the agent, a behavioural constraint of the agent and / or a characteristics of the agent.
15. The apparatus according to any preceding claim, wherein the processor is further configured to determine one or more errors in the first frame of scene data based on the first and / or second model data.
16. The apparatus according to claim 15, wherein the processor is further configured to apply a correction to the one or more determined errors in the second frame of scene data.
17. The apparatus according to any preceding claim, wherein the one or more further agents comprise one or more objects in the vicinity of the vehicle capable of impacting the control instruction for the vehicle.
18. The apparatus according to any preceding claim, wherein the one or more further agents comprise one or more of: a further vehicle; a pedestrian and an item of road infrastructure.
19. The apparatus according to any preceding claim, wherein the control instruction comprises data for a actuation of a control input of the vehicle, and wherein the processor is further configured to transmit the control instruction to the control input of the vehicle.
20. The apparatus according to any preceding claim, wherein the processor is further configured to cause actuation of a control input of the vehicle, based on the control instruction.
21. A method for determining a control instruction for a vehicle, the method comprising:receiving, by a receiver, a first frame of scene data representing a scene in a vicinity of the vehicle at a first time, the first scene data comprising:a plurality of agents including an ego agent representing the vehicle and at a plurality of further agents; andspatial coordinates for each of the plurality of agents;setting the ego agent as a first point of view, POV, agent, and the further agents as first interacting agents;determining, based on the received scene data, first model data predicting future behaviour of the first interacting agents;setting one of the further agents as a second POV agent, and the ego agent and remaining further agents as second interacting agents;generating, based on the received scene data, simulated scene data from the POV of the second POV agent;determining, based on the simulated scene data, second model data predicting future behaviour of the second interacting agents; anddetermining a control instruction for the vehicle, based on the first model data and the second model data.
22. A computer program product comprising computer program code that, when executed on a computer processor, causes the computer processor to undertake the steps of claim 21.
23. A vehicle comprising an apparatus for determining a control instruction according to claim 1.28
