Information processing device, vehicle control device, and vehicle

Abstract

An information processing device according to an embodiment of the present disclosure stores a plurality of first matrices that each express, as a matrix, a first graph in which a known traffic scene is described using a graph structure, or a plurality of second matrices obtained by incorporating importance in the first matrices. The information processing device can generate, on the basis of map data and situation data, a second graph in which a traffic situation in the surroundings of a vehicle is described in a graph structure, and can generate a third matrix that expresses the generated second graph as a matrix. The information processing device can also compare each first matrix or each second matrix with a third matrix, and output, as a fourth matrix corresponding to a traffic scene that the vehicle is highly likely encountering, the first matrix or the second matrix that most closely resembles the third matrix.

Description

Information Processing Device, Vehicle Control Device, and Vehicle

[0001] The present disclosure relates to an information processing device, a vehicle control device, and a vehicle.

[0002] During the operation of a vehicle, the objects that need attention and their risks change depending on the overall picture and context of the traffic situation. Therefore, it is necessary to provide driving assistance after appropriately grasping the overall picture of the traffic situation and the elements that need attention in that traffic situation.

[0003] For example, Patent Document 1 discloses that the surrounding traffic situation obtained by observation is interpreted by collating it with a pre-prepared traffic ontology.

[0004] Japanese Patent Application Laid-Open No. 2017-174417

[0005] The information processing device according to the first aspect of the present disclosure includes one or more memories and one or more first processors. The one or more memories store a plurality of first matrices representing a first graph that describes a known traffic scene in a graph structure, or a plurality of second matrices in which importance is incorporated into each of the first matrices. The one or more first processors can perform information processing using the first matrix or the second matrix read from the one or more memories, the map data and the situation data around the vehicle. The one or more first processors can generate a second graph that describes the traffic situation around the vehicle in a graph structure based on the map data and the situation data, and generate a third matrix representing the generated second graph in a row-column representation. The one or more first processors can further collate each of the first matrices or the second matrices stored in the one or more memories with the third matrix, and output the first matrix or the second matrix that is most similar to the third matrix as a fourth matrix corresponding to a traffic scene that the vehicle is likely to face.

[0006] A vehicle control device relating to a second aspect of the present disclosure comprises an information processing device relating to a first aspect of the present disclosure and one or more second processors capable of performing at least one of hazard warning control and hazard avoidance control based on a fourth status matrix obtained from the information processing device relating to a first aspect of the present disclosure.

[0007] A vehicle relating to a third aspect of this disclosure comprises a notification device and a running gear, an information processing device relating to a first aspect of this disclosure, and one or more second processors capable of performing at least one of hazard notification control to the notification device and hazard avoidance control to the running gear based on a fourth matrix obtained from the information processing device relating to a first aspect of this disclosure.

[0008] The accompanying drawings are provided for further understanding of this disclosure and are incorporated herein and constitute part of this specification. The drawings illustrate one embodiment and, together with the specification, serve to illustrate the principles of this disclosure.

[0009] Figure 1 is a diagram showing an example of a functional block of a vehicle according to the first embodiment of this disclosure. Figure 2 is a diagram showing an example of a known traffic situation. Figure 3 is a diagram showing an example of a knowledge graph describing the known traffic situation in Figure 2 using a graph structure. Figure 4 is a diagram showing an example of a knowledge matrix representing the knowledge graph in Figure 3 using a matrix structure. Figure 5 is a diagram showing an example of a weight matrix representing the importance of edges in the knowledge graph in Figure 3 using a matrix structure. Figure 6 is a diagram showing an example of a weighted matrix obtained by the Hadamard product of the knowledge matrix in Figure 4 and the weight matrix in Figure 5. Figure 7 is a diagram showing a modified example of the functional block of the vehicle in Figure 1. Figure 8 is a diagram showing an example of a surrounding traffic situation. Figure 9 is a diagram showing an example of external recognition data obtained from sensors, etc., in the surrounding traffic situation in Figure 8. Figure 10 is a diagram showing an example of instantaneous data about a vehicle in front of the vehicle (preceding vehicle) included in the external recognition data obtained from sensors, etc., in the surrounding traffic situation in Figure 8. Figure 11 is a diagram showing an example of an observation situation graph describing the surrounding traffic situation in Figure 8 using a graph structure. Figure 12 is a diagram showing an example of an observation status matrix, which is a matrix representation of the observation status graph in Figure 11. Figure 13 is a diagram showing an example of a matrix (surrounding situation matrix) that is most similar to the observation status matrix in Figure 12, among several weighted matrices obtained from the knowledge matrix DB and weight matrix DB in Figure 1. Figure 14 is a diagram showing an example of a driving assistance procedure in the vehicle in Figures 1 and 7. Figure 15 is a diagram showing an example of a functional block in a vehicle according to a second embodiment of this disclosure. Figure 16 is a diagram showing an example of a driving assistance procedure in the vehicle in Figure 15.

[0010] Hereinafter, several exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The following description is intended to illustrate specific examples of the present disclosure and should not be construed as limiting the disclosure. For example, elements such as numerical values, shapes, materials, parts, the location of each part, and the method of connecting each part are merely examples and should not be construed as limiting the disclosure. Furthermore, in the following exemplary embodiments, components not described in separate sections based on the highest-level concepts of the present disclosure are optional and may be provided as needed. The drawings are schematic and are not intended to be to scale. Throughout this specification and the drawings, components having substantially the same function and substantially the same configuration are denoted by the same reference numerals, and redundant descriptions are omitted. Furthermore, components not directly related to an embodiment of the present disclosure are not shown in the drawings.

[0011] <1. Background> The objects to pay attention to while driving a vehicle, and the potential dangers, vary depending on the overall picture and context of the traffic situation. Therefore, driving assistance is necessary that appropriately grasps the overall picture of the traffic situation and the elements that require attention in that traffic situation. For example, suppose there is a surrounding vehicle (another vehicle) in front of the moving vehicle (your vehicle) at a relative speed of -5.0 km / h. In this case, whether the surrounding vehicle is a parallel lane or an oncoming lane depends on whether the lane in which the moving vehicle is traveling is a parallel lane or an oncoming lane. Therefore, whether the surrounding vehicle is a parallel lane or an oncoming vehicle changes the judgment of whether the surrounding vehicle is a dangerous object that the moving vehicle should urgently avoid.

[0012] For example, Patent Document 1 discloses interpreting the surrounding traffic conditions obtained through observation by comparing them with a pre-prepared traffic ontology. However, in the invention described in Patent Document 1, it is difficult to interpret the surrounding traffic conditions if the surrounding traffic conditions are undefined in the traffic ontology or if there are gaps in the information obtained from the sensor.

[0013] Therefore, after careful consideration, the inventors of this application have conceived of a technology that can interpret the overall picture of the surrounding traffic situation and identify the elements that should be focused on in that traffic situation, even when the surrounding traffic situation is undefined in knowledge data such as traffic ontology, or when there are gaps in the information obtained from sensors. The information processing device, vehicle control device, and vehicle for realizing this will be described in detail below.

[0014] <2. First Embodiment> [Configuration Example] A vehicle 100 according to the first embodiment of the present disclosure will be described. Figure 1 shows an example of the functional blocks of the vehicle 100. The vehicle 100 corresponds to one specific example of the "vehicle" according to one embodiment of the present disclosure.

[0015] The vehicle 100 is capable of moving by the drive of a prime mover 50 (engine or motor). The vehicle 100 includes, for example, a sensor unit 10, a communication unit 20, a control unit 30, a memory 40, a prime mover 50, a brake 60, an EPS (Electric Power Steering) motor 70, and a notification unit 80, as shown in Figure 1. The memory 40 corresponds to one specific example of "one or more memories" according to one embodiment of the present disclosure.

[0016] The sensor unit 10 is comprised of various sensors mounted on the vehicle 100. For example, the sensor unit 10 comprises an accelerator opening sensor, a vehicle speed sensor, an acceleration sensor, an angular velocity sensor, a steering angle sensor, a steering torque sensor, and a brake torque sensor. The sensor unit 10 may also include sensors other than those listed above.

[0017] The accelerator pedal position sensor can detect the accelerator pedal position from the amount the accelerator pedal is pressed. The accelerator pedal position sensor can output time-series data (accelerator pedal position data) of the detected accelerator pedal position to the control unit 30.

[0018] The vehicle speed sensor is capable of detecting the speed of vehicle 100. The vehicle speed sensor is capable of outputting time-series data (vehicle speed data) of the detected vehicle speed to the control unit 30. The acceleration sensor is capable of detecting the acceleration applied to vehicle 100. The acceleration sensor is capable of outputting time-series data (acceleration data) of the detected acceleration in three directions to the control unit 30. The angular velocity sensor is capable of detecting the angular velocity of vehicle 100. The angular velocity sensor is capable of outputting time-series data (angular velocity data) of the detected three angular velocities (yaw angular velocity, roll angular velocity, and pitch angular velocity) to the control unit 30.

[0019] The steering angle sensor is capable of detecting the steering angle of the steering wheel of the vehicle 100. The steering angle sensor is capable of outputting time-series data (steering angle data) of the detected steering angle to the control unit 30. The steering torque sensor is capable of detecting the steering torque generated by the driver's steering wheel operation. The steering torque sensor is capable of outputting time-series data (steering torque data) of the detected steering torque to the control unit 30. The brake torque sensor is capable of detecting the braking force (torque of the brake 60) in relation to the brake pressure of the vehicle 100. The brake torque sensor is capable of outputting time-series data (braking force data) of the detected braking force to the control unit 30.

[0020] The sensor unit 10 further includes a stereo camera mounted on the vehicle 100 and a driving environment detection unit. The stereo camera is an autonomous sensor that senses the real space around the vehicle 100. The stereo camera is positioned, for example, symmetrically on either side of the central part of the vehicle 100 in the width direction, enabling stereo imaging of the area in front of the vehicle 100 from different viewpoints. The stereo camera is capable of outputting image data Da (a pair of stereo image data) obtained by imaging to the control unit 30.

[0021] The stereo camera is capable of generating distance image data Db based on image data Da (a pair of stereo image data) obtained through imaging, which is determined from the amount of displacement of the corresponding object's position. The driving environment detection unit can, for example, determine the lane markings that demarcate the road around the vehicle 100 based on the distance image data Db. The driving environment detection unit can further determine the road curvature of the markings that demarcate the left and right sides of the driving lane on which the vehicle 100 is traveling, and the width between the left and right markings (vehicle width). The driving environment detection unit can further perform predetermined pattern matching on the distance image data Db to detect lanes and three-dimensional objects such as structures that exist around the vehicle 100. A lane refers to a driving lane.

[0022] In the driving environment detection unit, the detection of three-dimensional objects includes, for example, the type of object, the distance to the object, the speed of the object, and the relative speed between the object and the vehicle 100 (the vehicle itself). Examples of objects to be detected include traffic lights, intersections, road signs, stop lines, other vehicles, pedestrians, bicycles, and buildings. Examples of buildings include detached houses, apartment buildings, commercial facilities, factories, and signs. The driving environment detection unit is capable of outputting driving environment information around the vehicle 100, including the information on three-dimensional objects acquired in this way, to the control unit 30.

[0023] The communication unit 20 is capable of acquiring data to supplement data that cannot be obtained from the image data Da and distance image data Db, for example, through vehicle-to-vehicle communication, vehicle-to-infrastructure communication, and satellite communication. The communication unit 20 is capable of outputting the acquired data to the control unit 30.

[0024] The communication unit 20 can acquire data obtained from other vehicles (e.g., vehicle position, vehicle speed) through vehicle-to-vehicle communication, for example. The communication unit 20 can also receive positioning signals transmitted from multiple positioning satellites through satellite communication, for example.

[0025] The communication unit 20 can acquire road map data around vehicle 100 from a control device capable of sequentially integrating and updating road map data transmitted from each vehicle via vehicle-to-infrastructure communication, and transmitting the updated road map data to each vehicle. The road map data consists of, for example, high-precision road map data (dynamic map), and mainly comprises static and quasi-static information that constitutes road information, and quasi-dynamic and dynamic information that mainly constitutes traffic information.

[0026] The static information that constitutes road information consists of information that requires updates at a frequency of no more than one month, such as roads and structures on roads, structures surrounding roads, lane information, road surface information, and permanent regulatory information. "Roads" include, for example, the location and shape of roads, intersections, and road attributes (e.g., national roads, prefectural roads, municipal roads, private roads, priority roads, non-priority roads, general roads, expressways). "Structures on roads" include, for example, traffic signs, traffic lights, convex mirrors, pedestrian overpasses, bus stops, and garbage collection points. "Structures surrounding roads" include, for example, various buildings and parks.

[0027] The quasi-static information that makes up road information consists of information that needs to be updated within an hour, such as traffic restriction information due to road construction or events, wide-area weather information, and congestion forecasts.

[0028] The semi-dynamic information that makes up traffic information consists of information that needs to be updated within one minute, such as actual congestion conditions and traffic restrictions at the time of observation, temporary traffic obstruction conditions such as fallen objects and obstacles, actual accident conditions, and local weather information.

[0029] The dynamic information that constitutes traffic information consists of information that requires updates every second, such as information transmitted and exchanged between moving objects, information on currently displayed traffic signals, information on pedestrians and cyclists at intersections, and information on vehicles traveling on roads. This road map data is maintained and updated in cycles until the next information is received from each vehicle, and the updated road map data is transmitted to each vehicle as appropriate via the communication unit 20.

[0030] Memory 40 is composed of, for example, one or more non-volatile memories, such as EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, or resistive random-access memory. Memory 40 stores, for example, a road map DB 41, a knowledge graph DB 42, a knowledge matrix DB 43, and a weight matrix DB 44, as shown in Figure 1. Memory 40 also stores, for example, data obtained by calculations performed by the hazard prediction unit 31, which will be described later.

[0031] The road map DB41 is a large-capacity storage medium such as an HDD, and stores high-precision road map data (dynamic map). This high-precision road map data, for example, similar to road map data obtained via the communication unit 20, mainly consists of static and quasi-static information that constitutes road information, and quasi-dynamic and dynamic information that mainly constitutes traffic information.

[0032] Knowledge Graph DB41 is knowledge data structured in a manner that can be used for analyzing traffic conditions, including traffic rules, reasoning rules, and common sense. Traffic conditions represent the relationship between the driving environment and the behavior of each traffic participant. Traffic rules refer to the rules that traffic participants must follow in order to participate in traffic in compliance with the rules. Reasoning rules refer to rules that allow conclusions about unknown matters to be drawn based on known matters. Common sense refers to the experience and collective intelligence that people implicitly know.

[0033] Knowledge graph DB 42 has knowledge graphs 42A that describe known traffic scenes in a graph structure. Knowledge graph DB 42 has multiple knowledge graphs 42A, each of which describes a plurality of known traffic scenes in a graph structure. Knowledge graph 42A corresponds to one specific example of the "first graph" according to one embodiment of the present disclosure.

[0034] A known traffic scene refers to a anticipated traffic scene or a past traffic scene. A traffic scene is a compilation of traffic conditions at a given moment. A traffic scene includes multiple traffic elements. Traffic elements may include, for example, object type, road structure, object speed, or object position. Object types may include, for example, vehicles (passenger cars), motorcycles, bicycles, and pedestrians. Road structure may include, for example, intersections, traffic lights, and driving lanes. Object speed may include, for example, constant speed, deceleration, and acceleration. Object position may include, for example, identifiers indicating that an object is moving in a certain place, creating a blind spot in a certain place, approaching a certain place or object, stopped in a certain place, indicating a positional relationship with a certain place or object, being in a position to collide with an object, and entering a certain place.

[0035] A graph structure refers to a structure in which known traffic scenes are described using nodes (entities) and edges (relationships). An edge connects two nodes. A sentence (subject, predicate, object) is formed by two nodes and one edge connecting them. Each node and each edge is associated with a label. Each node is assigned an identifier (ID).

[0036] Each node and each edge is assigned labels such as, for example, a label indicating the object type, a label indicating the road structure, a label indicating the state of the object, and a label indicating the object's location.

[0037] Examples of labels used to indicate the type of object include the following: • A label indicating the primary vehicle (SbjCar) • Labels indicating secondary vehicles (ObjCar1, ObjCar2, ...)

[0038] Examples of labels used to indicate road structure include the following: • Intersection label • Traffic light label

[0039] Examples of labels that indicate the state of an object include the following: • A label indicating deceleration (Deceleration) • A label indicating behavior (HasBehavior) • A label indicating role (HasRole) • A label indicating ambiguity (Obscenes) • A label indicating the state of a traffic light (TurnsGreenIn)

[0040] Examples of labels that indicate the location of an object include the following: • Labels indicating the relative position of an object (Leading) • Labels indicating the relative position of a road structure (FaceTo) • Labels indicating the stopping position (StopObjRightBefore) • Labels indicating that an object is creating a blind spot (Blind) • Labels indicating that an object is visible (View) • Labels indicating that an object is approaching (ApproachTo) • Labels indicating that an object is present (Exist)

[0041] Figure 2 shows an example of a known traffic situation (hereinafter referred to as "Known Traffic Situation X"). In Known Traffic Situation X, it is assumed that a vehicle (our vehicle) 100a is traveling on a road with two lanes in each direction. Vehicle 100a corresponds to an example of vehicle 100. This two-lane road consists of a road La on which vehicle 100a is traveling, and an opposing lane Lb provided along road La via a center line. Road La consists of lane La1 on which vehicle 100a is traveling, and an overtaking lane La2 provided along lane La1. Road Lb consists of lane Lb1 and an overtaking lane Lb2 provided along lane Lb1. An intersection IS is provided in front of vehicle 100a on this two-lane road. Intersection IS is provided with a traffic light TL1 for road La and a traffic light TL2 for the opposing lane Lb. This two-lane road intersects with road Lc (intersecting road) at intersection IS. Lane La1, passing lane La2, and lanes Lb1 and Lb1 are driving lanes.

[0042] In the known traffic situation X, there are two vehicles 100b and 100c traveling in front of the vehicle 100a. The vehicle 100b is traveling in the lane La1. The vehicle 100c is traveling in the overtaking lane La2. The vehicles 100b and 100c are traveling at positions visible from the vehicle 100a. The traffic lights TL1 and TL2 are not in the blind spots of the vehicles 100b and 100c and are visible from the vehicle 100a. In the traffic light TL1, a green (permission-to-enter indication color) light is on.

[0043] FIG. 3 shows an example of a knowledge graph 42A that describes the known traffic situation X in a graph structure. The knowledge graph 42A includes, as nodes, for example, Scene1, Intersection, TrafficLight, ObjCar1, ObjCar2, Leading, and Declaration. An ID of "1" is assigned to Scene1. An ID of "2" is assigned to Intersection. An ID of "3" is assigned to TrafficLight. An ID of "4" is assigned to ObjCar1. An ID of "5" is assigned to ObjCar2. An ID of "6" is assigned to Leading. An ID of "7" is assigned to Declaration. The knowledge graph 42A includes, as edges, for example, FaceTo, Exist, TurnsGreenIn, ApproachTo, View, HasRole, HasBehavior, and StopObjRightBefore. The knowledge graph 42A is stored in the knowledge graph DB42.

[0044] The knowledge matrix DB43 has a knowledge matrix 43A that represents the knowledge graph 42A in a row-column form. The knowledge matrix DB43 has a plurality of knowledge matrices 43A that represent each of the plurality of knowledge graphs 42A stored in the knowledge graph DB42 in a row-column form. The knowledge matrix 43A corresponds to a specific example of the "first matrix" according to an embodiment of the present disclosure.

[0045] Figure 4 shows an example of a knowledge matrix 43A. As an example of a knowledge matrix 43A, Figure 4 shows knowledge matrix A, which is a matrix representation of the knowledge graph 42A in Figure 3. When the number of IDs in the knowledge graph 42A is N, knowledge matrix A is an N × N matrix. Figure 4 shows an example of knowledge matrix A with N = 7. In knowledge matrix A, "1" means that an edge exists, and "0" means that an edge does not exist. In knowledge matrix A, the row number i (1 ≤ i ≤ N) is the ID of the node corresponding to the starting point of the edge, and the column number j (1 ≤ j ≤ N) is the ID of the node corresponding to the ending point of the edge. Therefore, when a certain component aij (i is the row number, j is the column number) of knowledge matrix A is "1", this means that there exists an edge that starts at the node with ID = i and ends at the node with ID = j. Furthermore, when a certain component aij (where i is the row number and j is the column number) of the knowledge matrix A is "0", this means that there is no edge that starts at the node with ID = i and ends at the node with ID = j. The knowledge matrix 43A is stored in the knowledge matrix DB43.

[0046] The weight matrix DB 44 has a weight matrix 44A that represents the importance of the edges of the knowledge graph 42A in a matrix. The weight matrix DB 44 has a plurality of weight matrices 44A that represent the importance of each edge of a plurality of knowledge graphs 42A stored in the knowledge graph DB 42.

[0047] FIG. 5 is a diagram showing an example of a weight matrix 44A that represents the importance of edges of the knowledge graph 42A in a row-column representation. In FIG. 5, a weight matrix W that represents the importance of edges of the knowledge graph 42A in FIG. 3 in a row-column representation is shown. When the number of IDs included in the knowledge graph 42A is N, the weight matrix W is an N×N matrix. In FIG. 5, a weight matrix W with N = 7 is illustrated. For each component bij (i is the row number, j is the column number) of the weight matrix W, a value between 0 and 1 is assigned as the importance. In the weight matrix W, "1" means the highest importance, and "0" means the lowest importance. In the weight matrix W, the row number i (1 ≤ i ≤ N) is the ID of the node corresponding to the starting point of the edge, and the column number j (1 ≤ j ≤ N) is the ID of the node corresponding to the arrival point of the edge. Therefore, when a certain component bij (i is the row number, j is the column number) of the weight matrix W is "1", this means that the importance of the edge starting from the node with ID = i and arriving at the node with ID = j is the highest. Also, when a certain component bij (i is the row number, j is the column number) of the weight matrix W is "0", this means that the importance of the edge starting from the node with ID = i and arriving at the node with ID = j is the lowest. The weight matrix 44A is stored in the weight matrix DB 44.

[0048] The weight matrix W is a matrix that represents each component bij (importance) as the probability (a value between 0 and 1) that two traffic elements (the node with ID = i and the node with ID = j) occur simultaneously. For example, the probability that the node with ID = 5 (ObjCar2) and the node with ID = 7 (Declaration) occur simultaneously corresponds to the probability of encountering a scenario where vehicle 100c takes a decelerating action when vehicle 100a is traveling in front of intersection IS. This probability is obtained by statistically processing data obtained from a large number of vehicles 100a that have traveled in front of intersection IS. When this probability is large, it means that the influence of the edge (HasBehavior) connecting the node with ID = 5 (ObjCar2) and the node with ID = 7 (Declaration) is large in the known traffic situation X.

[0049] Figure 6 shows an example of a weighted matrix S obtained by performing the Hadamard product of the knowledge matrix A in Figure 4 and the weight matrix W in Figure 5 using the following equation (1). The weighted matrix S represents the overall picture of the known traffic situation X and the importance of each traffic element included in the known traffic situation X. The weighted matrix S may be calculated by the matrix generation unit 312 described later, or it may be stored in advance in the weighted matrix DB 45 (see Figure 7).

[0050] When the number of IDs in the knowledge graph 42A is N, the weighted matrix S is an N × N matrix. Figure 6 shows an example of a weighted matrix S with N = 7. Each component cij (where i is the row number and j is the column number) of the weighted matrix S is assigned a value between 0 and 1, inclusive, as its importance. In the weighted matrix W, "1" means the highest importance, and "0" means the lowest importance. In the weighted matrix S, the row number i (1 ≤ i ≤ N) is the ID of the node corresponding to the starting point of the edge, and the column number j (1 ≤ j ≤ N) is the ID of the node corresponding to the ending point of the edge. Therefore, when a component cij (where i is the row number and j is the column number) of the weighted matrix S is "1", this means that the edge starting from the node with ID = i and ending at the node with ID = j has the highest importance. Furthermore, when a certain component cij (where i is the row number and j is the column number) of the weighted matrix S is "0", this means that the edge starting from the node with ID = i and ending at the node with ID = j has the lowest importance.

[0051] The weighted matrix DB 45 has a weighted matrix 45A obtained by performing the Hadamard product of a knowledge matrix 43A and a weight matrix 44A obtained from a common known traffic situation. The weighted matrix 45A corresponds to a matrix in which the importance included in the weight matrix 44A is incorporated into the knowledge matrix 43A. The weighted matrix 45A corresponds to a specific example of the "second matrix" according to one embodiment of the present disclosure. Here, the knowledge matrix 43A and weight matrix 44A obtained from a common known traffic situation are considered as a set of knowledge matrix 43A and weight matrix 44A. The weighted matrix DB 45 has a plurality of weighted matrices 45A obtained for each set of multiple sets of knowledge matrices 43A and weight matrix 44A included in the knowledge matrix DB 43 and weight matrix DB 44. When the weighted matrix DB 45 is provided in memory 40, the knowledge matrix DB 43 and weight matrix DB 44 may be omitted from memory 40.

[0052] The control unit 30 is capable of controlling the entire vehicle 100. The control unit 30 is, for example, a so-called ECU (Electronic Control Unit) and is composed of, for example, one or more processors and one or more memories. The control unit 30 may also be composed of, for example, a CPU (Central Processing Unit). In this case, the control unit 30 is capable of controlling the entire vehicle 100 by, for example, executing a program stored in one or more memories. The control unit 30 corresponds to one specific example of a "vehicle control device" according to one embodiment of the present disclosure.

[0053] The control unit 30 includes, for example, a locator unit. The locator unit is capable of acquiring the position coordinates of the vehicle 100 based on the positioning signal received through the communication unit 20. The locator unit is capable of estimating the vehicle's position on the road map by map matching the acquired position coordinates onto route map data. Based on the acquired position coordinates of the vehicle 100, the locator unit is capable of acquiring map data for a predetermined range including the vehicle 100 from the map data stored in the road map DB (database) 41, which will be described later.

[0054] The locator unit can switch to autonomous navigation, which estimates the vehicle's position on a road map based on vehicle speed, angular velocity, and longitudinal acceleration detected by the sensor unit 10, in environments where it is not possible to receive effective positioning signals from positioning satellites due to reduced sensitivity, such as when driving in a tunnel.

[0055] As described above, the locator unit estimates the position of the vehicle 100 on the road map (vehicle position) based on the positioning signal received through the communication unit 20 or the information detected by the sensor unit 10. Based on the estimated vehicle position on the road map, it is possible to determine the type of road the vehicle 100 is traveling on.

[0056] The locator unit can update the road map data stored in the road map DB 41 to the latest state using road map data acquired through external communication (vehicle-to-infrastructure communication and vehicle-to-vehicle communication) via the communication unit 20. This information update is performed not only on static information but also on quasi-static, quasi-dynamic, and dynamic information. As a result, the road map data is composed of road information and traffic information acquired through communication with the outside of the vehicle, and information on moving objects such as vehicles traveling on the road is updated in near real time.

[0057] The locator unit verifies the road map data based on the driving environment information recognized as described above, and can update the road map data stored in the road map DB41 to the latest state. This information update is performed not only on static information, but also on quasi-static, quasi-dynamic, and dynamic information. As a result, information on moving objects such as vehicles traveling on the road, as recognized as described above, is updated in real time.

[0058] The control unit 30 includes, for example, a hazard prediction unit 31, as shown in Figure 1. The hazard prediction unit 31 is capable of performing information processing using a knowledge graph 42A or a weighted matrix 45A and map data MD and situation data SD around the vehicle 100. As part of the above information processing, the hazard prediction unit 31 is capable of estimating whether or not the vehicle 100 is in a dangerous traffic situation. If the estimation result indicates that the vehicle 100 is in a dangerous traffic situation, the hazard prediction unit 31 is capable of predicting one or more contexts (important contexts) that are important for hazard prediction in the surrounding traffic situation (or surrounding traffic scene) that the vehicle 100 is likely to face, and the importance of each important context. The hazard prediction unit 31 is capable of outputting the one or more important contexts obtained by the prediction and the importance of each important context to the driving control unit 32. The hazard prediction unit 31 corresponds to one specific example of the "information processing device" according to one embodiment of the present disclosure.

[0059] The risk prediction unit 31 includes, for example, a data acquisition unit 311, a matrix generation unit 312, a similarity determination unit 313, and a context interpretation unit 314, as shown in Figure 1. A portion of the data acquisition unit 311, the matrix generation unit 312, the similarity determination unit 313, and the context interpretation unit 314 are configured to include one or more processors, and correspond to one specific example of "one or more first processors" according to one embodiment of the present disclosure.

[0060] The data acquisition unit 311 is capable of periodically acquiring data about the status or condition of the vehicle 100. Specifically, the data acquisition unit 311 is capable of acquiring various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, and various control signals for various devices of the vehicle 100. The data acquisition unit 311 is also capable of acquiring map data MD of the area around the vehicle 100 from the road map DB 41 in the memory 40. The various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, and various control signals for various devices of the vehicle 100 include status data SD of the area around the vehicle 100. Map data MD corresponds to one specific example of "map data" according to one embodiment of the present disclosure. Status data SD corresponds to one specific example of "status data" according to one embodiment of the present disclosure.

[0061] Figure 8 shows an example of the traffic conditions around vehicle 100d (hereinafter referred to as "surrounding traffic conditions A"). In surrounding traffic conditions A, it is assumed that vehicle (the vehicle itself) 100d is traveling on a road with two lanes in each direction. Vehicle 100d corresponds to an example of vehicle 100. This two-lane road consists of road La on which vehicle 100d is traveling and an opposing lane Lb provided along road La via a center line. Road La consists of lane La1 on which vehicle 100d is traveling and an overtaking lane La2 provided along lane La1. Road Lb consists of lane Lb1 and an overtaking lane Lb2 provided along lane Lb1. An intersection IS is provided in front of vehicle 100d on this two-lane road. Intersection IS is provided with a traffic light TL1 for road La and a traffic light TL2 for the opposing lane Lb. This two-lane road intersects with road Lc (intersecting road) at intersection IS.

[0062] In surrounding traffic conditions A, two vehicles, 100e and 100f, are traveling ahead of vehicle 100d. Vehicle 100e is traveling in lane La1. Vehicle 100f is traveling in the overtaking lane La2. Vehicles 100e and 100f are in a position where they can be seen by vehicle 100d. Traffic light TL1 is in the blind spot of vehicle 100e and cannot be seen by vehicle 100d. Therefore, vehicle 100d cannot determine the illumination status of traffic light TL1.

[0063] In surrounding traffic conditions A, the data acquired by the data acquisition unit 411 includes data on the position and speed of vehicle 100d, map data MD of the area around vehicle 100d, and situational data SD of the area around vehicle 100d. The situational data SD of the area around vehicle 100d includes, for example, data on the position and speed of vehicles 100b and 100c.

[0064] The matrix generation unit 312 is capable of generating external recognition data as shown in Figure 9 based on the map data MD and situation data SD acquired by the data acquisition unit 411. The external recognition data may include, for example, target, MapSgt, and State as major traffic elements. Target may include, for example, SujCar representing vehicle 100d, ObjCar1 representing vehicle 100e, and ObjCar2 representing vehicle 100f. MapSgt may include, for example, SubjectLane representing lane La1, PassingLane representing overtaking lane La2, OppositeLane representing opposing lane Lb, CrossingRoad representing road Lc, Intersection representing intersection IS, and TrafficLight representing traffic lights TL1 and TL2. The State field includes, for example, TrafficLightState, which indicates the lighting status of traffic light TL1. In the external environment recognition data, TrafficLightState is set to Non-Obs (not observed).

[0065] Figure 10 shows an example of instantaneous data for ObjCar2 in the external environment recognition data. The instantaneous data shown in Figure 10 is obtained from data acquired by the data acquisition unit 411 in surrounding traffic conditions A. The instantaneous data shown in Figure 10 includes data on time, type of ObjCar2, relative speed, X position, Y position, and lane.

[0066] The matrix generation unit 312 is capable of generating an observation status graph 42B that describes the traffic conditions around the vehicle 100 in a graph structure based on the generated external recognition data. The observation status graph 42B corresponds to one specific example of the "second graph" according to one embodiment of the present disclosure.

[0067] Figure 11 shows an example of the observation status graph 42B. The observation status graph 42B includes nodes such as Sceneα, Intersection, TrafficLight, Blind, ObjCar2, Leading, and Declération. Sceneα is assigned the ID "1". Intersection is assigned the ID "2". TrafficLight is assigned the ID "3". Blind is assigned the ID "4". ObjCar2 is assigned the ID "5". Leading is assigned the ID "6". Declération is assigned the ID "7". Observation status graph 42B includes, for example, FaceTo, Exist, Observations, HasRole, and HasBehavior as edges.

[0068] The matrix generation unit 312 is capable of generating an observation status matrix 43B, which is a matrix representation of the generated observation status graph 42B. The observation status matrix 43B corresponds to a specific example of the "third matrix" according to one embodiment of the present disclosure.

[0069] Figure 12 shows an example of the observation matrix 43B. Figure 12 shows the observation matrix O, which is a matrix representation of the observation graph 42B in Figure 11, as an example of the observation matrix 43B. When the number of IDs included in the observation graph 43B is N, the observation matrix O is an N × N matrix. Figure 12 shows an example of the observation matrix O with N = 7. In the observation matrix O, "1" means that an edge exists, and "0" means that an edge does not exist. In the observation matrix O, the row number i (1 ≤ i ≤ N) is the ID of the node corresponding to the starting point of the edge, and the column number j (1 ≤ j ≤ N) is the ID of the node corresponding to the ending point of the edge. Therefore, when a certain component dij (i is the row number, j is the column number) of the observation matrix O is "1", this means that there exists an edge that starts at the node with ID = i and ends at the node with ID = j. Furthermore, when a component dij (where i is the row number and j is the column number) of the observation matrix O is "0", this means that there is no edge that starts at the node with ID = i and ends at the node with ID = j.

[0070] The matrix generation unit 312 can generate a weighted matrix 45A by reading a knowledge matrix 43A and a weight matrix 44A obtained from common known traffic conditions from the memory 40 and performing the Hadamard product of the read knowledge matrix 43A and the weight matrix 44A.

[0071] The similarity determination unit 313 is capable of comparing each weighted matrix 45A generated by the matrix generation unit 312 for each known traffic situation, or each weighted matrix 45A stored in memory 40, with the observed situation matrix 43B. Based on the comparison results, the similarity determination unit 313 can output the weighted matrix 45A that is most similar to the observed situation matrix 43B as a matrix (surrounding situation matrix 43C) corresponding to the traffic scene that the vehicle 100 is most likely to be facing. Figure 13 shows an example of the surrounding situation matrix 43C. Figure 13 also shows the surrounding situation matrix B as an example of the surrounding situation matrix 43C. The surrounding situation matrix 43C and the surrounding situation matrix B correspond to a specific example of the "fourth matrix" according to one embodiment of this disclosure.

[0072] When N is the number of IDs included in the observation graph 43B, the surrounding situation matrix B is an N × N matrix. Figure 13 shows an example of the surrounding situation matrix B with N = 7. Each component eij (where i is the row number and j is the column number) of the surrounding situation matrix B is assigned a value between 0 and 1, inclusive, as its importance. In the surrounding situation matrix B, "1" means the highest importance, and "0" means the lowest importance. In the surrounding situation matrix B, the row number i (1 ≤ i ≤ N) is the ID of the node corresponding to the starting point of the edge, and the column number j (1 ≤ j ≤ N) is the ID of the node corresponding to the ending point of the edge. Therefore, when a component eij (where i is the row number and j is the column number) of the surrounding situation matrix B is "1", this means that the edge starting from the node with ID = i and ending at the node with ID = j has the highest importance. Furthermore, when a certain component eij (where i is the row number and j is the column number) in the weight matrix W is "0", this means that the edge starting from the node with ID = i and ending at the node with ID = j has the lowest importance.

[0073] The similarity determination unit 313 can calculate multiple difference matrices D by taking the difference between each weighted matrix S generated by the matrix generation unit 312, or each weighted matrix S stored in the memory 40, and the observation status matrix O, for example, as shown in equation (2). The similarity determination unit 313 can calculate the sum of all components eij included in each of the calculated difference matrices D, for example, as shown in equation (3), as shown.

[0074] The similarity determination unit 313 is capable of outputting a weighted matrix S as the surrounding situation matrix B, which is the difference matrix D that minimizes the sum of all components eij included in the difference matrix D among the multiple difference matrices D calculated.

[0075] The context interpretation unit 314 is capable of outputting important contexts interpreted based on the magnitude of each component of the surrounding situation matrix 43C. For example, as shown in Figure 13, the context interpretation unit 314 can generate important context ICs consisting of four sentences corresponding to each component of the surrounding situation matrix B that is above a predetermined threshold (e.g., 0.4), using the observation situation graph 42B. Furthermore, as shown in Figure 13, the context interpretation unit 314 is capable of extracting the values ​​of each component of the surrounding situation matrix B that is above a predetermined threshold (e.g., 0.4) as importance IMs for each sentence included in the important context ICs. The context interpretation unit 314 is capable of outputting the important context ICs and importance IMs obtained in this way to the driving control unit 32.

[0076] The control unit 30 further includes a driving control unit 32, as shown in Figure 1, for example. The driving control unit 32 is capable of controlling the driving of the vehicle 100 (for example, the torque of the prime mover 50, the amount of brake depression, the steering angle of the steering wheel) and the notification of the vehicle 100 to the driver. Based on the data acquired by the data acquisition unit 311 (various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, and various control signals to various devices of the vehicle 100) and the data obtained from the context interpretation unit 314 (important context IC and importance IM), the driving control unit 32 is capable of performing at least one of hazard notification control and hazard avoidance control.

[0077] The driving control unit 32 is capable of calculating a correction torque to correct the requested torque given to the accelerator control unit 321 (described later) based on the data acquired by the data acquisition unit 311 and the data obtained from the context interpretation unit 314. The driving control unit 32 is capable of calculating a correction torque to correct the requested torque given to the brake control unit 322 (described later) based on the data acquired by the data acquisition unit 311 and the data obtained from the context interpretation unit 314. The driving control unit 32 is capable of calculating a correction torque to correct the steering assist torque generated by the steering control unit 323 (described later) based on the data acquired by the data acquisition unit 311 and the data obtained from the context interpretation unit 314. The driving control unit 32 is capable of generating notification data to notify the driver of the vehicle 100 based on the data acquired by the data acquisition unit 311 and the data obtained from the context interpretation unit 314.

[0078] The driving control unit 32 includes, for example, an accelerator control unit 321, a brake control unit 322, a steering control unit 323, and a notification control unit 324, as shown in Figure 1. The accelerator control unit 321, brake control unit 322, steering control unit 323, and notification control unit 324 are configured to include one or more processors and correspond to one specific example of "one or more second processors" according to one embodiment of the present disclosure.

[0079] The accelerator control unit 321 is capable of controlling the torque of the prime mover 50 based on the required torque corresponding to the amount the driver of the vehicle 100 depresses the accelerator pedal. Furthermore, the accelerator control unit 321 is capable of controlling the torque of the prime mover 50 based on a target torque which is the required torque plus a correction torque. The prime mover 50 is configured to drive the steering wheels of the vehicle 100 and is capable of driving the steering wheels of the vehicle 100 according to the required torque or target torque input from the accelerator control unit 321.

[0080] The brake control unit 322 is capable of controlling the torque of the brake 60 based on the required torque corresponding to the amount the driver of the vehicle 100 presses the brake pedal. The brake control unit 322 is also capable of controlling the torque of the brake 60 based on a target torque which is the required torque plus a correction torque. The brake 60 is configured to brake the steering wheels of the vehicle 100 and is capable of braking the steering wheels of the vehicle 100 according to the required torque or target torque input from the brake control unit 322.

[0081] The steering control unit 323 can derive a steering assist torque to assist the steering torque generated by the driver's steering wheel operation, and set an EPS torque corresponding to the derived steering assist torque. The steering control unit 323 can output a control signal to the EPS motor 70 so that the output torque of the EPS motor 70 becomes the set EPS torque. The steering control unit 323 can output a control signal to the EPS motor 70 so that the output torque of the EPS motor 70 becomes the EPS torque considering the correction torque. The EPS motor 70 generates an output torque based on the input control signal and can control the steering angle of the steering wheel.

[0082] The accelerator control unit 321, brake control unit 322, and steering control unit 323 may include, for example, a CPU. In this case, the accelerator control unit 321, brake control unit 322, and steering control unit 323 can perform the various driving controls described above by, for example, executing control software stored in one or more memories.

[0083] The notification control unit 324 is capable of outputting notification data to the notification unit 80 for notification to the driver of the vehicle 100. The notification control unit 324 is capable of generating a video signal including the notification data and outputting it to the notification unit 80. The notification control unit 324 is capable of generating an audio signal including the notification data and outputting it to the notification unit 80. The notification unit 80 is composed of, for example, a display panel and a speaker. When the notification unit 80 receives the video signal from the control unit 30 (notification control unit 324), it is capable of displaying an image on the display screen corresponding to the input video signal. When the notification unit 80 receives the audio signal from the control unit 30 (notification control unit 324), it is capable of outputting an audio from the speaker corresponding to the input audio signal.

[0084] (Driving Assistance Procedure) Next, the driving assistance procedure in vehicle 100 will be described with reference to Figure 14. Figure 14 shows an example of the driving assistance procedure in vehicle 100.

[0085] The vehicle 100 (hazard prediction unit 31) first acquires various data from the sensor unit 10, various data from the outside via the communication unit 20, and various control signals for various devices of the vehicle 100. Based on the acquired data and various control signals, the hazard prediction unit 31 acquires the location information of the vehicle 100 and surrounding map data including the acquired location of the vehicle 100 from the road map data DB 41 in the memory 40. The hazard prediction unit 31 further acquires stereo image Da or distance image Db from the stereo camera. The hazard prediction unit 31 acquires attribute information and location information of each structure constituting the road around the vehicle 100 from the map data and stereo image Da or distance image Db, etc. In this way, map data MD and situation data SD are acquired (step S101).

[0086] Next, the hazard prediction unit 31 generates external recognition data based on the acquired map data MD and situation data SD, and generates an observation situation graph 42B that describes the traffic situation around the vehicle 100 in a graph structure based on the generated external recognition data (step S102). The hazard prediction unit 31 generates an observation situation matrix 43B that is a matrix representation of the generated observation situation graph 42B (step S103). The hazard prediction unit 31 calculates the similarity between each weighted matrix 45A obtained by calculation, or each weighted matrix 45A stored in memory 40, and the observation situation matrix 43B (step S104). For example, the hazard prediction unit 31 compares each weighted matrix 45A with the observation situation matrix 43B and identifies the weighted matrix 45A that is most similar to the observation situation matrix 43B as the matrix (surrounding situation matrix 43C) that corresponds to the traffic scene that the vehicle 100 is most likely to face (step S105).

[0087] Next, the hazard prediction unit 31 generates important context ICs and importance IMs of the important context ICs, which are interpreted based on the magnitude of each component of the surrounding situation matrix 43C, using the observation situation graph 42B. The hazard prediction unit 31 outputs the generated important context ICs and importance IMs to the driving control unit 32 (step S106). The driving control unit 32 performs at least one of hazard notification control and hazard avoidance control based on the data acquired by the data acquisition unit 311 and the data obtained from the context interpretation unit 314 (step S107). In this way, driving assistance is performed in the vehicle 100.

[0088] [Effects] Next, the effects of the vehicle 100 according to the first embodiment will be described.

[0089] In this embodiment, based on map data MD and situation data SD, an observation situation graph 42B is generated that describes the traffic conditions around the vehicle 100 (surrounding traffic conditions) in a graph structure, and an observation situation matrix 43B is generated by representing the generated observation situation graph 42B as a matrix. As a result, the overall picture of the surrounding traffic conditions is represented by the observation situation matrix 43B. In this embodiment, further, a plurality of weighted matrices 45A, in which importance is incorporated into each knowledge matrix 43A which is a matrix representation of a knowledge graph 42A that describes known traffic scenes in a graph structure, are compared with the observation situation matrix 43B. As a result, even if the surrounding traffic conditions are not included in the plurality of traffic conditions corresponding to the plurality of weighted matrices 45A, or if there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic conditions and the observation situation matrix 43B does not accurately represent the overall picture of the surrounding traffic conditions, the weighted matrix 45A that is most similar to the observation situation matrix 43B can be considered as the matrix representing the surrounding traffic conditions, that is, the surrounding situation matrix 43C corresponding to the traffic scene that the vehicle 100 is most likely to face. As a result, even if the observation situation matrix 43B does not accurately represent the overall picture of the surrounding traffic situation because the surrounding traffic situation is not included in the multiple traffic situations corresponding to multiple weighted matrices 45A, or because there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix 43C, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0090] In this embodiment, important context IC and importance IM are generated based on the magnitude of each component of the surrounding situation matrix 43C. This allows for the interpretation of the overall surrounding traffic situation using the generated important context IC and importance IM, and enables the identification of elements that should be considered within that traffic situation. Therefore, driving assistance can be provided that appropriately identifies elements requiring attention in the surrounding traffic situation.

[0091] In this embodiment, among the multiple difference matrices D calculated, the weighted matrix S obtained in which the sum of all components eij in the difference matrix D is minimized is output as the surrounding situation matrix B. As a result, even if the observed situation matrix O does not accurately represent the overall picture of the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix B, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0092] In this embodiment, a surrounding situation matrix 43C corresponding to the traffic scene most likely to be faced by the vehicle 100 is obtained from among a plurality of weighted matrices 45A obtained by calculating the Hadamard product of the weight matrix 44A and the knowledge matrix 43A. As a result, even if the observed situation matrix 43B does not accurately represent the overall picture of the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix 43C, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0093] In this embodiment, the weight matrix 44A expresses importance as the probability of two traffic elements occurring simultaneously. This allows the driver to identify elements that should be considered in the surrounding traffic conditions from the surrounding conditions matrix 43C. Therefore, it is possible to provide driving assistance that appropriately identifies elements that require attention in the surrounding traffic conditions.

[0094] <3. Modifications of the First Embodiment> Next, a modification of the vehicle 100 according to the first embodiment will be described.

[0095] <Modification 3-1> In the first embodiment, the weight matrix 44A may be omitted from the memory 40. In this case, the similarity determination unit 313 is able to compare each knowledge matrix 43A stored in the memory 40 with the observation situation matrix 43B. As a result of the comparison, the similarity determination unit 313 is able to output the knowledge matrix 43A that is most similar to the observation situation matrix 43B as a matrix (surrounding situation matrix 43C) corresponding to the traffic scene that the vehicle 100 is most likely to be facing.

[0096] The similarity determination unit 313 can calculate multiple difference matrices D by, for example, taking the difference between each knowledge matrix 43A stored in memory 40 and the observation situation matrix 43B. The similarity determination unit 313 can calculate, for example, the sum of all components eij included in each calculated difference matrix D, called Sum. The similarity determination unit 313 can output the knowledge matrix 43A from which the sum of all components eij included in the difference matrix D is minimized, as the surrounding situation matrix 43C.

[0097] In this modified example, each knowledge matrix 43A stored in memory 40 is compared with the observation situation matrix 43B. Even in this case, if the surrounding traffic situation is not included in the multiple traffic situations corresponding to multiple knowledge matrices 43A, or if there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic situation, and the observation situation matrix 43B does not accurately represent the overall picture of the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix 43C, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0098] <Modification 3-2> In the first embodiment and modification 3-1, each component of the knowledge matrix 43A or weighted matrix 45A may be represented by a spatial vector. Even in this case, if the surrounding traffic situation is not included in the multiple traffic situations corresponding to multiple knowledge matrices 43A or multiple weighted matrices 45A, or if there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic situation, and the observation situation matrix 43B does not accurately represent the overall picture of the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix 43C, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0099] <4. Second Embodiment> [Configuration Example] A vehicle 200 according to the second embodiment of the present disclosure will now be described. Figure 15 shows an example of the functional blocks of the vehicle 200. The vehicle 200 corresponds to a specific example of the "vehicle" according to one embodiment of the present disclosure. In the following, descriptions of components common to the first embodiment will be omitted as appropriate.

[0100] The vehicle 200 is capable of moving by the drive of a prime mover 50 (engine or motor). The vehicle 200 includes, for example, a sensor unit 10, a communication unit 20, a control unit 30, a memory 40, a prime mover 50, a brake 60, an EPS motor 70, and a notification unit 80, as shown in Figure 15.

[0101] The sensor unit 10 is comprised of various sensors mounted on the vehicle 200. For example, the sensor unit 10 comprises an accelerator opening sensor, a vehicle speed sensor, an acceleration sensor, an angular velocity sensor, a steering angle sensor, a steering torque sensor, and a brake torque sensor. The sensor unit 10 further includes a gaze sensor capable of detecting the gaze of the driver of the vehicle 200. The gaze sensor comprises, for example, a camera capable of capturing images of the driver's face in the vehicle 200, and a calculation unit capable of detecting the gaze of the driver of the vehicle 200 based on the image data obtained by the camera. The gaze sensor is capable of outputting the obtained gaze data GD to the control unit 30.

[0102] Memory 40 stores, for example, a road map DB 41, a knowledge graph DB 42, a knowledge matrix DB 43, a weight matrix DB 44, and a gaze data DB 46, as shown in Figure 15. The gaze data DB 46 has gaze data 46A that represents the proportion at which the driver of vehicle 200 simultaneously gazes on any two traffic elements included in a particular surrounding traffic situation. The gaze data DB 46 has multiple gaze data 46A corresponding to each of multiple surrounding traffic situations.

[0103] The weight matrix DB44 has a weight matrix 44A that represents the importance of the edges of the knowledge graph 42A in a matrix. The weight matrix DB44 has multiple weight matrices 44A that represent the importance of each edge of multiple knowledge graphs 42A stored in the knowledge graph DB42. The weight matrix W is a matrix in which each component bij (importance) is represented by the proportion (value between 0 and 1) that a driver simultaneously pays attention to two traffic elements (a node with ID=i and a node with ID=j). For example, the proportion that a driver of vehicle 200 simultaneously pays attention to a node with ID=5 (ObjCar2) and a node with ID=7 (Declaration) corresponds to the proportion that a driver of vehicle 200 pays attention to when vehicle 200 encounters multiple situations where vehicle 200 is driving before intersection IS, and vehicle 200 takes deceleration action. When this ratio is large, it means that the influence of the edge (HasBehavior) connecting the node with ID=5 (ObjCar2) and the node with ID=7 (Declaration) is large in the known traffic situation X.

[0104] The risk prediction unit 31 includes, for example, a data acquisition unit 311, a matrix generation unit 312, a similarity determination unit 313, a context interpretation unit 314, and an importance update unit 315, as shown in Figure 15. A part of the data acquisition unit 311, the matrix generation unit 312, the similarity determination unit 313, the context interpretation unit 314, and the importance update unit 315 correspond to one specific example of "one or more first processors" according to one embodiment of the present disclosure.

[0105] The data acquisition unit 311 is capable of periodically acquiring data about the status or condition of the vehicle 200 and data about the gaze of the vehicle 200's driver. Specifically, the data acquisition unit 311 is capable of acquiring various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, and various control signals for various devices of the vehicle 200. Furthermore, the data acquisition unit 311 is capable of acquiring map data MD of the area around the vehicle 200 from the road map DB 41 in the memory 40. The various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, and various control signals for various devices of the vehicle 200 include status data SD of the area around the vehicle 200 and gaze data GD of the vehicle 200's driver.

[0106] The importance update unit 315 is capable of generating gaze data 46A corresponding to a specific surrounding traffic condition based on the gaze data GD obtained by the sensor unit 10. The importance update unit 315 stores the newly obtained gaze data 46A (hereinafter referred to as "gaze data 46x") in the gaze data DB 46 of the memory 40. If gaze data 46A (hereinafter referred to as "gaze data 46y") obtained in the same known traffic condition as the surrounding traffic condition from which gaze data 46x was obtained is stored in the memory 40, the importance update unit 315 is capable of replacing the gaze data 46y stored in the memory 40 with gaze data 46x. In other words, the importance update unit 315 is capable of updating the gaze data 46A stored in the memory 40.

[0107] The importance update unit 315 is capable of updating the weight matrix 44A corresponding to the known traffic situation most similar to the surrounding traffic situation from which the gaze data 46x was obtained, based on the newly obtained gaze data 46x. The matrix generation unit 312 is capable of generating a weighted matrix 45A by reading the knowledge matrix 43A and weight matrix 44A obtained from common known traffic situations from the memory 40 and performing the Hadamard product of the read knowledge matrix 43A and weight matrix 44A. If the weight matrix 44A has been updated, the matrix generation unit 312 is capable of generating a weighted matrix 45A by reading the knowledge matrix 43A obtained from common known traffic situations and the updated weight matrix 44A from the memory 40 and performing the Hadamard product of the read knowledge matrix 43A and weight matrix 44A.

[0108] The matrix generation unit 312 is capable of generating external recognition data as shown in Figure 9 based on the map data MD and situation data SD acquired by the data acquisition unit 411. The matrix generation unit 312 is capable of generating an observation situation matrix 43B, which is a matrix representation of the generated observation situation graph 42B. The matrix generation unit 312 is capable of generating a weighted matrix 45A by reading the knowledge matrix 43A and weight matrix 44A obtained from common known traffic situations from the memory 40 and performing the Hadamard product of the read knowledge matrix 43A and weight matrix 44A.

[0109] The similarity determination unit 313 is capable of comparing each weighted matrix 45A generated by the matrix generation unit 312 for each known traffic situation with the observed situation matrix 43B. Based on the comparison results, the similarity determination unit 313 can output the weighted matrix 45A that is most similar to the observed situation matrix 43B as the matrix (surrounding situation matrix 43C) corresponding to the traffic scene that the vehicle 200 is most likely to be facing. The similarity determination unit 313 is capable of calculating multiple difference matrices D by, for example, taking the difference between each weighted matrix S generated by the matrix generation unit 312 and the observed situation matrix O. The similarity determination unit 313 is capable of calculating the sum of all components eij included in each calculated difference matrix D, called Sum. The similarity determination unit 313 is capable of outputting the weighted matrix S from among the multiple calculated difference matrices D that yields the smallest sum of all components eij included in the difference matrix D as the surrounding situation matrix B.

[0110] The context interpretation unit 314 is capable of outputting important contexts interpreted based on the magnitude of each component of the surrounding situation matrix 43C. For example, as shown in Figure 13, the context interpretation unit 314 can generate important context ICs consisting of four sentences corresponding to each component of the surrounding situation matrix B that is above a predetermined threshold (e.g., 0.4), using the observation situation graph 42B. Furthermore, as shown in Figure 13, the context interpretation unit 314 is capable of extracting the values ​​of each component of the surrounding situation matrix B that is above a predetermined threshold (e.g., 0.4) as importance IMs for each sentence included in the important context ICs. The context interpretation unit 314 is capable of outputting the important context ICs and importance IMs obtained in this way to the driving control unit 32.

[0111] (Driving Assistance Procedure) Next, the driving assistance procedure in vehicle 200 will be described with reference to Figure 16. Figure 16 shows an example of the driving assistance procedure in vehicle 200.

[0112] The vehicle 200 (hazard prediction unit 31) first acquires various data from the sensor unit 10, various data from the outside via the communication unit 20, and various control signals for various devices of the vehicle 200. Based on the acquired data and various control signals, the hazard prediction unit 31 acquires the vehicle 200's position information and surrounding map data including the acquired vehicle 200's position from the road map data DB 41 in the memory 40. Based on the acquired data and various control signals, the hazard prediction unit 31 acquires data about the driver's line of sight of the vehicle 200. The hazard prediction unit 31 further acquires a stereo image Da or a distance image Db from the stereo camera. The hazard prediction unit 31 acquires attribute information and position information of each structure constituting the road around the vehicle 200 from the map data and the stereo image Da or distance image Db, etc. In this way, map data MD, situation data SD, and line of sight data GD are acquired (step S101).

[0113] Next, the hazard prediction unit 31 generates external recognition data based on the acquired map data MD and situation data SD, and generates an observation situation graph 42B that describes the traffic situation around the vehicle 200 in a graph structure based on the generated external recognition data (step S102). The hazard prediction unit 31 updates the weight matrix 44A based on the gaze data GD (step S108). For example, the hazard prediction unit 31 generates gaze data 46A corresponding to a specific surrounding traffic situation based on the gaze data GD, and stores the generated gaze data 46A in the gaze data DB 46 of the memory 40. For example, the hazard prediction unit 31 updates the weight matrix 44A corresponding to the known traffic situation that is most similar to the surrounding traffic situation from which the gaze data 46A was obtained, based on the newly obtained gaze data 46A.

[0114] The hazard prediction unit 31 generates an observation situation matrix 43B by representing the generated observation situation graph 42B as a matrix (step S103). The hazard prediction unit 31 calculates the similarity between each weighted matrix 45A obtained by the calculation and the observation situation matrix 43B (step S104). The hazard prediction unit 31, for example, compares each weighted matrix 45A with the observation situation matrix 43B and identifies the weighted matrix 45A that is most similar to the observation situation matrix 43B as the matrix corresponding to the traffic scene that the vehicle 100 is most likely to face (surrounding situation matrix 43C) (step S105).

[0115] Next, the hazard prediction unit 31 generates important context ICs and importance IMs of the important context ICs, which are interpreted based on the magnitude of each component of the surrounding situation matrix 43C, using the observation situation graph 42B. The hazard prediction unit 31 outputs the generated important context ICs and importance IMs to the driving control unit 32 (step S106). The driving control unit 32 performs at least one of hazard notification control and hazard avoidance control based on the data acquired by the data acquisition unit 311 and the data obtained from the context interpretation unit 314 (step S107). In this way, driving assistance is performed in the vehicle 200.

[0116] [Effects] Next, the effects of the vehicle 200 according to the second embodiment will be described.

[0117] In this embodiment, based on map data MD and situation data SD, an observation situation graph 42B is generated that describes the traffic conditions around the vehicle 200 (surrounding traffic conditions) in a graph structure, and an observation situation matrix 43B is generated by representing the generated observation situation graph 42B as a matrix. As a result, the overall picture of the surrounding traffic conditions is represented by the observation situation matrix 43B. In this embodiment, further, a plurality of weighted matrices 45A, in which importance is incorporated into each knowledge matrix 43A which is a matrix representation of a knowledge graph 42A that describes known traffic scenes in a graph structure, are compared with the observation situation matrix 43B. As a result, even if the surrounding traffic conditions are not included in the plurality of traffic conditions corresponding to the plurality of weighted matrices 45A, or if there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic conditions and the observation situation matrix 43B does not accurately represent the overall picture of the surrounding traffic conditions, the weighted matrix 45A that is most similar to the observation situation matrix 43B can be considered as the matrix representing the surrounding traffic conditions, that is, the surrounding situation matrix 43C corresponding to the traffic scene that the vehicle 200 is most likely to face. As a result, even if the observation situation matrix 43B does not accurately represent the overall picture of the surrounding traffic situation because the surrounding traffic situation is not included in the multiple traffic situations corresponding to multiple weighted matrices 45A, or because there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix 43C, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0118] In this embodiment, important context IC and importance IM are generated based on the magnitude of each component of the surrounding situation matrix 43C. This allows for the interpretation of the overall surrounding traffic situation using the generated important context IC and importance IM, and enables the identification of elements that should be considered within that traffic situation. Therefore, driving assistance can be provided that appropriately identifies elements requiring attention in the surrounding traffic situation.

[0119] In this embodiment, among the multiple difference matrices D calculated, the weighted matrix S obtained in which the sum of all components eij in the difference matrix D is minimized is output as the surrounding situation matrix B. As a result, even if the observed situation matrix O does not accurately represent the overall picture of the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix B, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0120] In this embodiment, a surrounding situation matrix 43C corresponding to the traffic scene most likely to be faced by the vehicle 200 is obtained from among a plurality of weighted matrices 45A obtained by calculating the Hadamard product of the weight matrix 44A and the knowledge matrix 43A. As a result, even if the observed situation matrix 43B does not accurately represent the overall picture of the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix 43C, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0121] In this embodiment, the weight matrix 44A expresses importance as the proportion of time the driver pays attention to each of the two traffic elements. This allows the driver to identify elements that should be considered in the surrounding traffic situation from the surrounding situation matrix 43C. Therefore, it is possible to provide driving assistance that appropriately identifies elements that require attention in the surrounding traffic situation.

[0122] In this embodiment, the weight matrix 44A is updated based on the newly obtained gaze data 46A (gaze data 46x). This makes it possible to grasp the subjective focus elements of the vehicle 200 driver. Therefore, it is possible to provide driving assistance that appropriately captures the elements that require attention in the surrounding traffic conditions.

[0123] <5. Modifications of the Second Embodiment> Next, a modification of the vehicle 200 according to the second embodiment will be described.

[0124] In the second embodiment, each component of the knowledge matrix 43A or the weighted matrix 45A may be represented by a spatial vector. Even in this case, if the surrounding traffic situation is not included in the multiple traffic situations corresponding to the multiple weighted matrices 45A, or if there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic situation, and the observation situation matrix 43B does not accurately represent the overall picture of the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the surrounding situation matrix 43C, and the elements that should be focused on in that traffic situation can be grasped. Therefore, driving assistance can be provided while appropriately grasping the elements that should be paid attention to in the surrounding traffic situation.

[0125] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0126] Furthermore, for example, this disclosure can take the following configuration: <1> An information processing device comprising: one or more memories that store a plurality of first matrices, which are matrix representations of a first graph that describes a known traffic scene in a graph structure, or a plurality of second matrices, in which importance is incorporated into each of the first matrices; and one or more first processors that can perform information processing using the first matrices or second matrices read from the one or more memories, and map data and situation data of the vehicle's surroundings, wherein the one or more first processors can: generate a second graph that describes the traffic situation around the vehicle in a graph structure based on the map data and the situation data; generate a third matrix, which is a matrix representation of the generated second graph; compare each of the first matrices or second matrices stored in the one or more memories with the third matrix, and output the first matrix or second matrix that is most similar to the third matrix as a fourth matrix corresponding to a traffic scene that the vehicle is most likely to be facing. <2> The information processing apparatus according to <1>, wherein one or more first processors are capable of outputting important contexts interpreted based on the magnitude of each component of the fourth matrix. <3> The information processing apparatus according to <1> or <2>, wherein one or more first processors are capable of calculating a plurality of difference matrices by taking the difference between each of the first matrices or each of the second matrices and the third matrix, and outputting the first matrix or the second matrix which has the smallest sum of all the components included in the difference matrix among the calculated plurality of difference matrices as the fourth matrix. <4> The information processing apparatus according to any one of <1> to <3>, wherein each component of the first matrix or the second matrix is ​​represented by a spatial vector. <5> The information processing apparatus according to any one of <1> to <4>, wherein the second matrix is ​​a matrix obtained by taking the Hadamard product of a weight matrix that expresses the importance as the probability of two traffic elements occurring simultaneously and the first matrix.<6> The information processing device according to any one of <1> to <4>, wherein the second matrix is ​​obtained by taking the Hadamard product of a weight matrix that expresses the importance as the proportion of time the driver pays attention to the two traffic elements and the first matrix. <7> The information processing device according to <6>, wherein the one or more first processors are capable of updating the weight matrix based on newly obtained driver attention data. <8> A vehicle control device comprising the information processing device according to any one of <1> to <7>, and one or more second processors capable of performing at least one of hazard warning control and hazard avoidance control based on the fourth situation matrix obtained from the information processing device. <9> A vehicle comprising a warning device and a driving device, the information processing device according to any one of <1> to <7>, and one or more second processors capable of performing at least one of hazard warning control for the warning device and hazard avoidance control for the driving device based on the fourth matrix obtained from the information processing device.

[0127] In an information processing device, vehicle control device, and vehicle according to one embodiment of the present disclosure, a second graph is generated that describes the traffic conditions around the vehicle (surrounding traffic conditions) in a graph structure based on map data and situation data, and a third matrix is ​​generated that is a matrix representation of the generated second graph. As a result, the overall picture of the surrounding traffic conditions is represented by the third matrix. In this information processing device, the third matrix is ​​further compared with a plurality of first matrices, which are matrix representations of a first graph that describes a known traffic scene in a graph structure, or a plurality of second matrices, each of which incorporates importance levels. As a result, even if the surrounding traffic conditions are not included in the plurality of traffic conditions corresponding to the plurality of first matrices and the plurality of second matrices, or if there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic conditions, and the third matrix does not accurately represent the overall picture of the surrounding traffic conditions, the first matrix or second matrix that is most similar to the third matrix can be considered as a matrix representing the surrounding traffic conditions, that is, a fourth matrix corresponding to the traffic scene that the vehicle is most likely to be facing. As a result, even if the third matrix does not accurately represent the overall picture of the surrounding traffic situation because the surrounding traffic situation is not included in the multiple traffic situations corresponding to multiple first matrices and multiple second matrices, or because there is missing data in the external recognition data obtained from sensors etc. in the surrounding traffic situation, the overall picture of the surrounding traffic situation can be interpreted from the fourth matrix, and the elements that should be focused on in that traffic situation can be grasped.

[0128] The control unit 30 shown in Figures 1, 7, and 15 can be implemented by a circuit including at least one semiconductor integrated circuit, such as at least one processor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC) and / or at least one field-programmable gate array (FPGA). The at least one processor can be configured to perform all or some of the functions of the control unit 30 shown in Figures 1, 7, and 15 by reading instructions from at least one non-transient, tangible computer-readable medium. Such a medium can take various forms, including, but is not limited to, various magnetic media such as hard disks, various optical media such as CDs or DVDs, and various semiconductor memories (i.e., semiconductor circuits) such as volatile memory or non-volatile memory. Volatile memory may include DRAM and SRAM. Non-volatile memory may include ROM and NVRAM. The ASIC is an integrated circuit (IC) specialized to perform all or some of the functions of the control unit 30 shown in Figures 1, 7, and 15. An FPGA is an integrated circuit designed to be configurable after manufacturing to perform all or some of the various functions of the control unit 30 shown in Figures 1, 7, and 15.

Claims

1. An information processing device comprising: one or more memories that store a plurality of first matrices, which are matrix representations of a first graph that describes a known traffic scene in a graph structure, or a plurality of second matrices, each of which incorporates importance levels; and one or more first processors capable of performing information processing using the first or second matrices read from the one or more memories, and map data and situation data of the vehicle's surroundings, wherein the one or more first processors are capable of: generating a second graph that describes the traffic situation around the vehicle in a graph structure based on the map data and the situation data; generating a third matrix, which is a matrix representation of the generated second graph; comparing each of the first or second matrices stored in the one or more memories with the third matrix, and outputting the first or second matrix that is most similar to the third matrix as a fourth matrix corresponding to a traffic scene that the vehicle is most likely to be facing.

2. The information processing apparatus according to claim 1, wherein the one or more first processors are capable of outputting important contexts interpreted based on the magnitude of each component of the fourth matrix.

3. The information processing apparatus according to claim 1, wherein the one or more first processors can calculate a plurality of difference matrices by taking the difference between each of the first matrices or each of the second matrices and the third matrices, and output the first matrices or second matrices among the calculated plurality of difference matrices that have the smallest sum of all the components included in the difference matrices as the fourth matrices.

4. The information processing apparatus according to claim 1, wherein each component of the first matrix or the second matrix is ​​represented by a spatial vector.

5. The information processing apparatus according to claim 1, wherein the second matrix is ​​obtained by taking the Hadamard product of a weight matrix, which expresses the importance as the probability that two traffic elements occur simultaneously, and the first matrix.

6. The information processing apparatus according to claim 1, wherein the second matrix is ​​obtained by taking the Hadamard product of the first matrix and a weight matrix that expresses the importance as the proportion of time a driver pays attention to the two traffic elements.

7. The information processing apparatus according to claim 6, wherein the one or more first processors are capable of updating the weight matrix based on newly obtained driver gaze data.

8. A vehicle control device comprising the information processing device according to any one of claims 1 to 7, and one or more second processors capable of performing at least one of hazard warning control and hazard avoidance control based on the fourth matrix obtained from the information processing device.

9. A vehicle comprising: a notification device and a running device; the information processing device according to any one of claims 1 to 7; and one or more second processors capable of performing at least one of hazard notification control for the notification device and hazard avoidance control for the running device based on the fourth matrix obtained from the information processing device.

Images