Information processing device, vehicle control device, and vehicle

Abstract

An information processing device according to one aspect of the present disclosure calculates, for each known traffic scenario contained in a storage unit, the degree of similarity with a created surrounding traffic scenario, and as a result, when the degree of similarity is less than a threshold value for any known traffic scenario, the information processing device sets image data and the surrounding traffic scenario as transmission data to be transmitted to an external device, transmits the transmission data to the external device, and is capable of receiving, from the external device and as a response to the transmission, a new surrounding traffic scenario having a prescribed relationship with the image data and surrounding traffic scenario.

Description

Information processing apparatus, vehicle control apparatus, and vehicle

[0001] The present disclosure relates to an information processing apparatus, a vehicle control apparatus, and a vehicle.

[0002] There is known a driving support method that estimates a dangerous event by combining knowledge data such as an ontology or a knowledge graph and logical inference, and warns a driver.

[0003] Japanese Unexamined Patent Application Publication No. 2016-091039

[0004] The information processing apparatus according to the first aspect of the present disclosure includes a storage unit, a processing unit, and a communication unit. The storage unit stores a plurality of known traffic scenarios each of which is composed of a plurality of integrated time-series known traffic scenes. The processing unit is capable of performing processing using the plurality of known traffic scenarios, the image data in front of the vehicle, and the map data and situation data around the vehicle. The communication unit is capable of transmitting the data obtained by the processing unit to an external device or receiving data from the external device. The processing unit is capable of executing the following two processes. (1) Interpreting a traffic context based on the map data and the situation data, thereby generating a plurality of surrounding traffic scenes in time series, and integrating the generated plurality of surrounding traffic scenes in time series to create a surrounding traffic scenario (2) For each of the known traffic scenarios included in the storage unit, calculating the degree of similarity with the created surrounding traffic scenario, and as a result, when the degree of similarity is less than the threshold value in any of the known traffic scenarios, using the image data and the surrounding traffic scenario as transmission data to be transmitted to an external device The communication unit transmits the transmission data to the external device, and as a response to this transmission, is capable of receiving a new surrounding traffic scenario having a predetermined relationship with the image data and the surrounding traffic scenario from the external device.

[0005] The information processing device relating to the second aspect of this disclosure comprises an acquisition unit and a processing unit. The acquisition unit is capable of acquiring image data of the area in front of the vehicle, a surrounding traffic scenario composed of multiple integrated time-series surrounding traffic scenes, and multiple network image data included in Internet information. The processing unit is capable of performing image similarity matching on the image data acquired by the acquisition unit and the multiple network image data acquired by the acquisition unit, acquiring metadata of the network image data with the highest similarity, and generating a new surrounding traffic scenario based on the surrounding traffic scenario acquired by the communication unit and the acquired metadata.

[0006] The information processing device relating to the third aspect of this disclosure comprises an acquisition unit and a processing unit. The acquisition unit is capable of acquiring image data of the area in front of a vehicle, surrounding traffic data describing the surrounding traffic conditions of the vehicle using at least one of numerical and character representations, and a plurality of master traffic scenarios, each composed of a plurality of master traffic scenes in an integrated time series. The processing unit is capable of inputting at least the image data from the image data and surrounding traffic data acquired by the acquisition unit to a language processing AI, obtaining a situation interpretation statement of the surrounding traffic conditions from the language processing AI, and generating a new traffic scenario based on the acquired situation interpretation statement and the plurality of master traffic scenarios.

[0007] A vehicle control device relating to the fourth aspect of this disclosure comprises a scenario acquisition unit capable of acquiring a new surrounding traffic scenario from any of the information processing devices relating to the first to third aspects of this disclosure, and a control unit capable of performing at least one of notification control and driving control based on the new surrounding traffic scenario acquired by the scenario acquisition unit.

[0008] A vehicle relating to the fifth aspect of this disclosure comprises a notification device, a running gear, and a control unit. The control unit is capable of performing at least one of notification control for the notification device and running control for the running gear based on a novel surrounding traffic scenario obtained from any of the information processing devices relating to the first to third aspects of this disclosure.

[0009] The accompanying drawings are provided for further understanding of this disclosure and are incorporated herein and constitute part of this specification. The drawings illustrate one embodiment and, together with the specification, serve to illustrate the principles of this disclosure.

[0010] Figure 1 is a diagram showing an example of the traffic situation in front of the vehicle (surrounding traffic situation Sa). Figure 2 is a diagram showing an example of the traffic situation in front of the vehicle (surrounding traffic situation Sa). Figure 3 is a diagram showing an example of external recognition data obtained from sensors, etc., in the surrounding traffic situation Sa of Figures 1 and 2. Figure 4 is a diagram showing an example of the traffic context of the surrounding traffic situation Sa of Figures 1 and 2. Figure 5 is a diagram showing an example of the traffic situation in front of the vehicle (surrounding traffic situation Sb). Figure 6 is a diagram showing an example of the traffic situation in front of the vehicle (surrounding traffic situation Sc). Figure 7 is a diagram showing an example of the schematic configuration of a driving control system including a vehicle according to one embodiment of the present disclosure. Figure 8 is a diagram showing an example of a functional block of a driving control device provided in the vehicle of Figure 7. Figure 9 is a diagram showing an example of the concept of a known traffic scenario. Figure 10 is a diagram showing an example of a knowledge graph embedded in the knowledge space. Figure 11 is a diagram showing an example of known traffic situation A. Figure 12 is a diagram showing an example of the traffic context of known traffic situation A of Figure 11. Figure 13 is a diagram showing an example of known traffic situation B. Figure 14 is a diagram showing an example of the traffic context of known traffic situation B in Figure 13. Figure 15 is a diagram showing an example of known traffic situation C. Figure 16 is a diagram showing an example of the traffic context of known traffic situation C in Figure 15. Figure 17 is a diagram showing an example of the concept of a surrounding traffic scenario. Figure 18 is a diagram showing an example of the functional block of the server device in Figure 7. Figure 19 is a diagram showing an example of a driving assistance procedure in a vehicle equipped with the driving control device in Figure 8. Figure 20 is a diagram showing an example of a driving assistance procedure following Figure 19. Figure 21 is a diagram showing an example of a driving assistance procedure following Figure 20. Figure 22 is a diagram showing an example of the functional block of a driving control device installed in a vehicle equipped with the server device functions of Figure 7. Figure 23 is a diagram showing a modified example of the functional block of the server device in Figure 7. Figure 24 is a diagram showing a modified example of the driving assistance procedure following Figure 19. Figure 25 is a diagram showing a modified example of the functional block of the driving control device in Figure 8. Figure 26(A) is a diagram showing an example of the priority list for rear-end collisions in Figure 25. Figure 26(B) is a diagram showing an example of the priority list for intersection collisions in Figure 25. Figure 26(C) is a diagram showing an example of the priority list for cutting in accidents in Figure 25. Figure 26(D) is a diagram showing an example of the priority list for right-turn collisions in Figure 25.Figure 27 is a diagram showing an example of the accident statistics frequency table in Figure 25. Figure 28 is a diagram showing an example of the write list in Figure 25. Figure 29 is a diagram showing a modified version of the driving assistance procedure following Figure 20. Figure 30 is a diagram showing a modified version of the function block of a driving control device installed in a vehicle equipped with the server device functions of Figure 7.

[0011] Hereinafter, several exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The following description is intended to illustrate specific examples of the present disclosure and should not be construed as limiting the disclosure. For example, elements such as numerical values, shapes, materials, parts, the location of each part, and the method of connecting each part are merely examples and should not be construed as limiting the disclosure. Furthermore, in the following exemplary embodiments, components not described in separate sections based on the highest-level concepts of the present disclosure are optional and may be provided as needed. The drawings are schematic and are not intended to be to scale. Throughout this specification and the drawings, components having substantially the same function and substantially the same configuration are denoted by the same reference numerals, and redundant descriptions are omitted. Furthermore, components not directly related to an embodiment of the present disclosure are not shown in the drawings.

[0012] <1. Background> Driving assistance methods are known that estimate dangerous events and warn drivers by combining knowledge data such as ontologs and knowledge graphs with logical reasoning. In such driving assistance methods, logical reasoning works when the observed surrounding traffic conditions are included in the traffic conditions described in the knowledge data. However, if the main traffic elements of the surrounding traffic conditions include unknown traffic elements that are not included in the knowledge data, logical reasoning may not work due to the unknown traffic elements included in the surrounding traffic conditions.

[0013] Therefore, after careful consideration, the inventors of this application have conceived of a technology that enables logical reasoning to operate even when unknown traffic elements are included among the main traffic elements in the surrounding traffic conditions. Below, we will give an example of a traffic condition in which unknown traffic elements are detected by a recognition sensor (hereinafter referred to as "a traffic condition including unknown traffic elements") and explain the background of the technology newly conceived.

[0014] (Traffic situation including unknown traffic elements) Figures 1 and 2 show an example of the traffic situation (surrounding traffic situation Sa) in front of vehicle 100a (the vehicle itself). Surrounding traffic situation Sa corresponds to an example of a traffic situation including unknown traffic elements. Figure 1 shows an example of the surrounding traffic situation Sa as seen from the driver of vehicle 100a, and Figure 2 shows an example of the surrounding traffic situation Sa as seen from above the road on which vehicle 100a is traveling. Assume that vehicle 100a is traveling on a road La with one lane in each direction. Road La consists of a driving lane La1 on which vehicle 100a is traveling, and an opposing lane La2 provided along the driving lane La1 via a center line. An intersection ISa is provided in front of vehicle 100a on road La. Road La intersects with road Lb at intersection ISa. Road Lb is, for example, a road with one lane in each direction. On road La, pedestrian crossings CW1 are provided both before and after intersection ISa in relation to vehicle 100a. On road La, traffic lights TL1 are provided both before and after intersection ISa in relation to vehicle 100a. On road Lb, pedestrian crossings CW2 are provided both to the left and right of intersection ISa in relation to vehicle 100a. On road Lb, traffic lights TL2 are provided both to the left and right of intersection ISa in relation to vehicle 100a.

[0015] In the opposing lane La2, a motorized bicycle 100b is traveling towards intersection ISa. Traffic light TL1 is green (permission to enter), and vehicle 100a is traveling towards intersection ISa. Both vehicle 100a and motorized bicycle 100b are slowing down and activating their right turn signals in order to turn right at intersection ISa. Motorized bicycle 100b is a vehicle belonging to the newly created category of "specific small moped" under the revised Road Traffic Act, and is a traffic element not registered in the knowledge data installed in vehicle 100a. Vehicles belonging to the category of "specific small moped" include, for example, electric kick scooters. Vehicles belonging to the category of "specific small moped" are vehicles that are required by the Road Traffic Act to make a two-stage right turn at all intersections. Therefore, vehicle 100a cannot predict the movement of motorized bicycle 100b at intersection ISa from the knowledge data installed in vehicle 100a.

[0016] Figure 3 shows an example of data (external environment recognition data) acquired by the data acquisition unit 411 (described later) of vehicle 100a in the surrounding traffic conditions Sa. The external environment recognition data may include, for example, target, MapSgt, and State as major traffic elements. In the external environment recognition data, ObjUnknown is included as an unknown traffic element (unknown element UE) that is not included in the knowledge data, within target, which is one of the major traffic elements of the surrounding traffic conditions. ObjUnknown is a label that indicates that it is an unknown target that is not included in the knowledge data.

[0017] (target) target may include, for example, the following elements: - Name, position, and speed of vehicle 100a (SbjCar) - Name of unknown element UE (ObjUnknown) - Relative position and relative speed of unknown element UE (ObjUnknown) with respect to vehicle 100a (SbjCar) (MapSgt) MapSgt may include, for example, the following elements:・Names and structure of roads La and Lb (number of lanes, presence or absence of a median strip, etc.) ・Names of intersection IS, traffic lights TL1 and TL2, and pedestrian crossings CW1 and CW2 ・Relative positions of intersection ISa, traffic lights TL1 and TL2, and pedestrian crossings CW1 and CW2 with respect to vehicle 100a (SbjCar) ・Name and type of intersection ISa (presence or absence of time difference, presence or absence of pedestrian-vehicle separation, etc.) ・Names and types of traffic lights TL1 and TL2 (presence or absence of right / left turn indicators, etc.) (State) State may include, for example, the following elements: ・State of vehicle 100a (SbjCar) (turn signals, headlights, etc.) ・State of unknown element UE (ObjUnknown) (turn signals, headlights, etc.) ・Lighting status of traffic lights TL1 and TL2 (green, yellow, and red, etc.)

[0018] Figure 4 shows an example of a traffic context obtained by interpreting the surrounding traffic conditions Sa in the scenario creation unit 412 (described later) of vehicle 100a. The traffic context of the surrounding traffic conditions Sa consists of multiple sentences (subject, predicate, object), as shown in Figure 4.

[0019] In Figure 4, "Scene1, HasTime, 0sec" means "The transition time for Scene 1 is 0 seconds." "SbjCar, isRunningOn, subjectLane" means "Vehicle 100a is traveling in lane La1." "subjectLane, nextRoadSegment, SignalIntersection" means "Lane La1 has a signalized intersection ISa as a road segment ahead of vehicle 100a."

[0020] "SbjCar, HasSpeed, Deceleration" means "Vehicle 100a is traveling at a reduced speed." "SbjCar, ApproachTo, SignalIntersection" means "Vehicle 100a is approaching intersection ISa." "SbjCar, Ilighting, RightDirectionIndicator" means "Vehicle 100a has its right turn signal activated."

[0021] "ObjUnknown, isRunningOn,OppositeDirectionLane" means "Unknown element UE is running in the opposite lane La2." "ObjUnknown, HasSpeed, Deceleration" means "Unknown element UE is running at a reduced speed." "ObjUnknown, ApproachTo, SignalIntersection" means "Unknown element UE is approaching intersection ISa." "ObjUnknown, islighting, RightDirectionIndicator" means "Unknown element UE has its right turn signal on." "SignalIntersection,Light_turn_green,Carmoving" means that at intersection ISa, traffic light La1 is lit green (indicating permission to enter) and a vehicle is moving.

[0022] As described above, the traffic context of the surrounding traffic situation Sa includes an unknown element UE. Therefore, logical reasoning using the traffic context of the surrounding traffic situation Sa may not work. For example, as shown in Figures 5 and 6, a motorized bicycle 100b enters intersection ISa and attempts to stop before the pedestrian crossing CW1 in order to make a two-stage right turn. At this time, as a result of the logical reasoning not working, vehicle 100a may enter intersection ISa and make contact with motorized bicycle 100b while making a right turn, as shown in Figures 5 and 6. Therefore, the inventors of the present invention have conceived of a technology that can enable logical reasoning even when a traffic context containing an unknown element UE is obtained. The information processing device, vehicle control device, and vehicle for realizing this will be described in detail below in the embodiments described later.

[0023] <2. Embodiments> [Configuration Example] Vehicles 100a, 100e, and 100f according to one embodiment of the present disclosure will be described. Figure 7 shows a schematic configuration example of a driving control system 1 including vehicles 100a, 100e, and 100f according to this embodiment. Vehicles 100a, 100e, and 100f correspond to one specific example of "vehicles" according to one embodiment of the present disclosure. The driving control system 1 includes, for example, a plurality of vehicles (for example, vehicles 100a, 100e, and 100f) and a server device 200, as shown in Figure 7. The server device 200 corresponds to one specific example of "external device" according to one embodiment of the present disclosure. Each vehicle (for example, vehicles 100a, 100e, and 100f) and the server device 200 are configured to communicate with each other via a network NW. The network NW is, for example, the Internet, a wireless LAN such as Wi-Fi, or a mobile phone line. Vehicles 100a, 100e, and 100f share a common configuration. Vehicle 100a will be described below.

[0024] Vehicle 100a is capable of moving by the drive of a prime mover (engine or motor). Vehicle 100a includes a driving control device 1000, for example, as shown in Figure 7. The driving control device 1000 includes a sensor unit 10, a communication unit 20, a storage unit 30, a control unit 40, a prime mover 50, a brake 60, an EPS (Electric Power Steering) motor 70, and a notification unit 80, for example, as shown in Figure 8. The communication unit 20 corresponds to one specific example of the "communication unit" according to one embodiment of the present disclosure.

[0025] The sensor unit 10 is comprised of various sensors mounted on the vehicle 100a. For example, the sensor unit 10 comprises an accelerator opening sensor, a vehicle speed sensor, an acceleration sensor, an angular velocity sensor, a steering angle sensor, a steering torque sensor, and a brake torque sensor. The sensor unit 10 may also include sensors other than those listed above.

[0026] The accelerator pedal position sensor can detect the accelerator pedal position from the amount the accelerator pedal is pressed. The accelerator pedal position sensor can output time-series data (accelerator pedal position data) about the detected accelerator pedal position to the control unit 40.

[0027] The vehicle speed sensor is capable of detecting the speed (vehicle speed) of the vehicle 100a. The vehicle speed sensor is capable of outputting time-series data (vehicle speed data) of the detected vehicle speed to the control unit 40. The acceleration sensor is capable of detecting the acceleration applied to the vehicle 100a. The acceleration sensor is capable of outputting time-series data (acceleration data) of the detected acceleration in three directions to the control unit 40. The angular velocity sensor is capable of detecting the angular velocity of the vehicle 100a. The angular velocity sensor is capable of outputting time-series data (angular velocity data) of the detected three angular velocities (yaw angular velocity, roll angular velocity, and pitch angular velocity) to the control unit 40.

[0028] The steering angle sensor is capable of detecting the steering angle of the steering wheel of the vehicle 100a. The steering angle sensor is capable of outputting time-series data (steering angle data) of the detected steering angle to the control unit 40. The steering torque sensor is capable of detecting the steering torque generated by the driver's steering wheel operation. The steering torque sensor is capable of outputting time-series data (steering torque data) of the detected steering torque to the control unit 40. The brake torque sensor is capable of detecting the braking force (torque of the brake 60) in relation to the brake pressure of the vehicle 1. The brake torque sensor is capable of outputting time-series data (braking force data) of the detected braking force to the control unit 40.

[0029] The sensor unit 10 further includes a stereo camera mounted on the vehicle 100a and a driving environment detection unit. The stereo camera is an autonomous sensor that senses the real space around the vehicle 100a. The stereo camera is positioned, for example, symmetrically on either side of the central part of the vehicle 100a in the width direction, enabling stereo imaging of the area in front of the vehicle 100a from different viewpoints. The stereo camera is capable of outputting image data Ia (a pair of stereo image data) of the area in front of the vehicle 100a obtained by imaging to the control unit 40.

[0030] The stereo camera is capable of generating distance image data Ib, which is determined from the amount of displacement of the corresponding object's position, based on image data Ia (a pair of stereo image data) obtained by imaging. The driving environment detection unit can, for example, determine the lane markings that demarcate the road around the vehicle 100a based on the distance image data Ib. The driving environment detection unit can further determine the road curvature of the markings that demarcate the left and right sides of the road (driving lane) on which the vehicle 1 travels, and the width between the left and right markings (vehicle width). The driving environment detection unit can further perform predetermined pattern matching on the distance image data Ib to detect lanes and three-dimensional objects such as structures that exist around the vehicle 100a.

[0031] In the driving environment detection unit, the detection of three-dimensional objects includes, for example, the type of object, the distance to the object, the speed of the object, and the relative speed between the object and the vehicle (the vehicle itself). Examples of objects to be detected include traffic lights, intersections, road signs, stop lines, other vehicles, pedestrians, bicycles, and buildings. Examples of buildings include detached houses, apartment buildings, commercial facilities, factories, and signs. The driving environment detection unit can output driving environment information around the vehicle 100a, including the information on three-dimensional objects acquired in this way, to the control unit 40.

[0032] The communication unit 20 can transmit data obtained by the control unit 40 to the server device 200 via the network NW, and can also receive data from the server device 200 via the network NW and output the received data to the control unit 40. Furthermore, the communication unit 20 can acquire data to supplement data that cannot be obtained from image data Ia and distance image data Ib, for example, through vehicle-to-vehicle communication, vehicle-to-infrastructure communication, and satellite communication. The communication unit 20 can output the acquired data to the control unit 40.

[0033] The communication unit 20 can acquire data obtained from other vehicles (e.g., vehicle position, vehicle speed) through vehicle-to-vehicle communication, for example. The communication unit 20 can also receive positioning signals transmitted from multiple positioning satellites through satellite communication, for example.

[0034] The communication unit 20 can acquire road map data around vehicle 1 from a control device that can sequentially integrate and update road map data transmitted from each vehicle via vehicle-to-infrastructure communication and transmit the updated road map data to each vehicle. The road map data consists of, for example, high-precision road map data (dynamic map) and mainly comprises static and quasi-static information that constitutes road information, and quasi-dynamic and dynamic information that mainly constitutes traffic information.

[0035] The static information that constitutes road information consists of information that requires updates at a frequency of no more than one month, such as roads and structures on roads, structures surrounding roads, lane information, road surface information, and permanent regulatory information. "Roads" include, for example, the location and shape of roads, intersections, and road attributes (e.g., national roads, prefectural roads, municipal roads, private roads, priority roads, non-priority roads, general roads, expressways). "Structures on roads" include, for example, traffic signs, pedestrian crossings, traffic lights, convex mirrors, pedestrian overpasses, bus stops, and garbage collection points. "Structures surrounding roads" include, for example, various buildings and parks.

[0036] The quasi-static information that makes up road information consists of information that needs to be updated within an hour, such as traffic restriction information due to road construction or events, wide-area weather information, and congestion forecasts.

[0037] The semi-dynamic information that makes up traffic information consists of information that needs to be updated within one minute, such as actual traffic congestion and driving restrictions at the time of observation, temporary driving obstructions such as fallen objects and obstacles, actual accident conditions, and local weather information.

[0038] The dynamic information that constitutes traffic information consists of information that requires updates every second, such as information transmitted and exchanged between moving objects, information on currently displayed traffic signals, information on pedestrians and cyclists at intersections, and information on vehicles traveling on roads. This road map data is maintained and updated in cycles until the next information is received from each vehicle, and the updated road map data is transmitted to each vehicle as appropriate via the communication unit 20.

[0039] The storage unit 30 is composed of, for example, non-volatile memory, such as EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, or resistive random-access memory. The storage unit 30 stores, for example, a road map DB 31, a knowledge graph DB 32, and a knowledge space DB 33, as shown in Figure 8.

[0040] The road map DB31 is a large-capacity storage medium such as an HDD, and stores high-precision road map data (dynamic map). This high-precision road map data includes, for example, static and quasi-static information that mainly constitutes road information, and quasi-dynamic and dynamic information that mainly constitutes traffic information.

[0041] Knowledge graph DB32 and knowledge space DB33 are knowledge data structured in a manner that can be used for analyzing traffic conditions, including traffic rules, reasoning rules, and common sense, respectively. Traffic conditions represent the relationship between the driving environment and the behavior of each traffic participant. Traffic rules refer to the rules that traffic participants must follow in order to participate in traffic in compliance with the rules. Reasoning rules refer to rules that allow conclusions about unknown matters to be drawn based on known matters. Common sense refers to the experience and collective intelligence that people implicitly know.

[0042] Knowledge Graph DB32 contains knowledge data that describes multiple known traffic scenarios in a graph structure. Known traffic scenarios refer to anticipated traffic scenarios or past traffic scenarios. A traffic scenario refers to a summary of traffic conditions that change moment by moment. In Knowledge Graph DB32, known traffic scenarios are described as conditional terms, and possible events (e.g., dangerous events) are described as resultal terms. In a traffic scenario, each traffic participant is assigned an identifier (ID), and the position and speed of each traffic participant are associated with each participant. Traffic participants may include, for example, the vehicle itself (SbjCar), as well as other vehicles (ObjCar), motorcycles (ObjMotorcycle), bicycles (ObjBicycle), and pedestrians (ObjPedestrian) that are present around the vehicle. In a traffic scenario, the lane each vehicle is traveling in and its type are also associated with each vehicle.

[0043] A known traffic scenario is composed of a plurality of known traffic scenes in an integrated time series. A known traffic scene refers to an assumed traffic scene or a past traffic scene. A traffic scene refers to an arrangement of traffic situations at a certain moment. In a traffic scenario, a time order is associated with a plurality of traffic scenes. A known traffic scenario is composed of a plurality of traffic elements. The traffic elements may include, for example, object types, road structures, object speeds, object positions, or object states, etc. The object types may include traffic participants (e.g., vehicles (passenger cars), motorcycles, bicycles, and pedestrians). The road structures include, for example, intersections, traffic lights, and driving lanes. The object speeds may include, for example, a constant speed, deceleration, and acceleration. The object positions may include, for example, identifiers indicating moving to a certain place, creating a blind spot at a certain place, approaching a certain place or a certain object, stopping at a certain place, indicating the positional relationship with a certain place or a certain object, being at a position on the verge of colliding with a certain object, and entering a certain place. The object states may include, for example, the lighting state of a traffic light. The traffic elements may further include, for example, the transition time between scenes. The transition time between scenes refers to the difference between the time of the first scene and the time of the second scene in two scenes (the first scene, the second scene) that are continuous over time.

[0044] The traffic elements may further include, for example, a risk level indicating the risk of a scene. The risk level indicating the risk of a scene refers to an index indicating the possibility that, in a certain scene, the main vehicle and a traffic participant (other vehicle, motorcycle, bicycle, or pedestrian) to be monitored as seen from the main vehicle will interfere (contact, collide) with each other in the future. The risk level is defined, for example, by the energy at the time of interference (contact, collision) or the statistical frequency.

[0045] In the knowledge graph DB32, a certain known traffic scenario (hereinafter referred to as "Known Traffic Scenario I") is composed of multiple known traffic scenes (scene a, scene b, scene c) in an integrated time series, as shown in Figure 9, for example. In Known Traffic Scenario I, the multiple known traffic scenes (scene a, scene b, scene c) are linked in a temporal order. In Known Traffic Scenario I, each known traffic scene (scene a, scene b, scene c) is composed of multiple traffic elements. Known traffic scene (scene a) is composed of multiple traffic elements (traffic elements a1, a2, a3, a4, etc.), as shown in Figure 9, for example. Known traffic scene (scene b) is composed of multiple traffic elements (traffic elements b1, b2, b3, b4, etc.), as shown in Figure 9, for example. Known traffic scene (scene c) is composed of multiple traffic elements (traffic elements c1, c2, c3, c4, etc.), as shown in Figure 9, for example.

[0046] In the knowledge graph DB32, each known traffic scenario is composed of a directed graph containing multiple nodes (entities) and multiple edges (relationships) that connect two nodes (entities). Each node (entity) corresponds to a known traffic scene or traffic element. In each known traffic scenario, each entity and each relation is associated with a label, and a sentence (subject, predicate, object) is formed by two entities and one relation that connects these two entities.

[0047] The knowledge space DB33 has knowledge data obtained by numerically representing (e.g., spatially representing) a plurality of known traffic scenarios included in the knowledge graph DB32. As shown in, for example, FIG. 10, the knowledge space DB33 has a knowledge graph 33A that represents a known traffic scenario I included in the knowledge graph DB32 as a spatial vector. As shown in, for example, FIG. 10, the knowledge graph 33A includes a plurality of identifiers (IDs) and is table data in which an attribute, a label, and a spatial vector are associated with each identifier (ID). The identifier (ID) is for identifying each entity and each relation in the known traffic scenario I, and one is assigned to each entity and each relation.

[0048] Examples of labels assigned to entities and relations include labels indicating object types, road structures, object speeds, object positions, and object states.

[0049] Examples of labels indicating object types include the following labels: - Label indicating the main vehicle (SbjCar) - Labels indicating subordinate vehicles (ObjCar1, ObjCar2,...) - Labels indicating motorcycles (ObjMotorcycle1, ObjMotorcycle2,...) - Labels indicating pedestrians (ObjPedestrian1, ObjPedestrian12,...) - Labels indicating bicycles (ObjBicycle1, ObjBicycle2,...)

[0050] Examples of labels indicating road structure include the following: • Intersection label • Signal Intersection label • No Signal Intersection label • Traffic Signal label • Subject Lane label • Opposite Direction Lane label • Crossroad label

[0051] Examples of labels that indicate object velocity include the following: • A label indicating constant velocity (Constant) • A label indicating deceleration (Deceleration) • A label indicating acceleration (Acceleration)

[0052] Examples of labels that indicate the position of an object include the following: • Labels indicating the location where the object is moving (IsRunningOn) • Labels indicating that the object has velocity (HasSpeed) • Labels indicating that the object is creating a blind spot (CreateBlindIn) • Labels indicating that the object is approaching (ApproachTo) • Labels indicating that the object is stationary (stopAt) • Labels indicating the relative position of the object or road structure (nextRoadSegment) • Labels indicating that the object is in a position to collide (NearCrashTo) • Labels indicating that the object is entering (EnterIn) • Labels indicating that the object is moving away (Leave)

[0053] Examples of labels that indicate the state of an object include the following: • Labels indicating the lighting status of a traffic light (Light_turns_green, Light_turns_yellow, Light_turns_red)

[0054] In the knowledge space DB33, sets of (subject, predicate, object) are converted into numerical representations using an "embedding algorithm" so that specific operations can be performed on the sets of (subject, predicate, object) defined in the knowledge graph DB32. Examples of "embedding algorithms" include TransE or RESCAL. Here, the numerical representation is a spatial vector, as shown in Figure 10. When the numerical representation is a spatial vector, the "specific operation" means that when the entity corresponding to the subject is the first vector value, the relation corresponding to the predicate is the second vector value, and the entity corresponding to the object is the third vector value, then "the first vector value + the second vector value = the third vector value".

[0055] For example, in the subject-predicate-object set (SbjCar IsRunningOn subjectLane), the vector value of SbjCar is [0,1,2], the vector value of IsRunningOn is [1,1,1], and the vector value of subjectLane is [1,2,3]. In this case, [0,1,2] + [1,1,1] = [1,2,3] holds true.

[0056] The "embedding algorithm" is stored, for example, in the memory unit 30. The calculation process for deriving the numerical representation using the "embedding algorithm" is performed, for example, in the risk prediction unit 41 described later. The numerical representation may be, for example, a two-dimensional vector or a vector of four or more dimensions. The numerical representation may also be a representation other than a vector, for example, a matrix.

[0057] The knowledge space DB33 may include sets of (subject, predicate, object) indicating the transition time of a known traffic scene, sets of (subject, predicate, object) indicating the risk level of a known traffic scene, and sets of (subject, predicate, object) indicating the risk level of a known traffic scenario. Figure 10 illustrates, for example, a set of (subject, predicate, object) indicating the transition time of a known traffic scene (scene a): (Scenea, HasTime, 0Sec). "Scenea, HasTime, 0Sec" means "the transition time of scene a is 0 seconds."

[0058] Furthermore, Figure 10 illustrates, for example, a set of (subject, predicate, object) indicating the risk level of a known traffic scene, namely (Scene a, HasRisk, RiskLevel_a). "Scene a, HasRisk, RiskLevel_a" means "The risk level of Scene a is a." Also, Figure 10 illustrates, for example, a set of (subject, predicate, object) indicating the risk level of a known traffic scenario, namely (Scenario I, HasRisk, RiskLevel_I). "Scenario I, HasRisk, RiskLevel_I" means "The risk level of Scenario I is RiskLevel_I." Note that RiskLevel_a and RiskLevel_I are actually expressed as specific numerical values.

[0059] Figure 11 shows an example of a known traffic situation (hereinafter referred to as "Known Traffic Situation A"). A known traffic scene (scene a) in the knowledge graph DB 32 is generated, for example, based on the traffic context (see Figure 12) obtained by interpreting the data acquired by the data acquisition unit 411 described later in Known Traffic Situation A shown in Figure 11.

[0060] In known traffic situation A, vehicle (own vehicle) 100a is traveling on a road La with one lane in each direction. Road La consists of a driving lane La1 on which vehicle 100a is traveling, and an opposing lane La2 provided along driving lane La2 via a center line. An intersection ISa is provided on road La in front of vehicle 100a. Road La intersects with road Lb at intersection ISa. Road Lb is, for example, a road with one lane in each direction. Pedestrian crossings CW1 are provided on road La, both before and after intersection ISa in relation to vehicle 100a. Traffic lights TL1 are also provided on road La, both before and after intersection ISa in relation to vehicle 100a. Pedestrian crossings CW2 are provided on road Lb, both to the left and to the right of intersection ISa. Road Lb is further provided with traffic lights TL2 both before and after intersection ISa in relation to vehicle 100a.

[0061] A motorcycle (100 km / h) is traveling in the oncoming lane La2. The motorcycle (100 km / h) is slowing down and flashing its right turn signal in order to turn right at intersection ISa. The driver of vehicle 100a can see the motorcycle (100 km / h) traveling towards intersection ISa in the oncoming lane La2.

[0062] Figure 12 shows an example of a traffic context obtained by interpreting known traffic situation A. The traffic context of known traffic situation A consists of multiple sentences (subject, predicate, object), as shown in Figure 12.

[0063] In Figure 12, "Scenea, HasTime, 0sec" means "The transition time for scene a is 0 seconds." "SbjCar, isRunningOn, subjectLane" means "Vehicle 100a is traveling in lane La1." "subjectLane, nextRoadSegment, SignalIntersection" means "Lane La1 has a signalized intersection IS as a road segment ahead of vehicle 100a."

[0064] "SbjCar, HasSpeed, Deceleration" means "Vehicle 100a is traveling at a reduced speed." "SbjCar, ApproachTo, SignalIntersection" means "Vehicle 100a is approaching intersection ISa." "SbjCar, Lighting, RightDirectionIndicator" means "Vehicle 100a has its right-turn indicator flashing."

[0065] "ObjMotorcycle, isRunningOn, OppositeDirectionLane" means "Motorcycle 100b is running in the opposite lane La2." "ObjMotorcycle, HasSpeed, Deceleration" means "Motorcycle 100b is running at a reduced speed." "ObjMotorcycle, ApproachTo, SignalIntersection" means "Motorcycle 100b is approaching intersection IS." "ObjMotorcycle, islighting, RightDirectionIndicator" means "Motorcycle 100b has its right-turn signal flashing." "SignalIntersection, Light_turns_green, Carmoving" means "Vehicle 100a and motorcycle 100b are in a state of movement when traffic light La1 is lit green."

[0066] Figure 13 shows an example of a known traffic situation (hereinafter referred to as "Known Traffic Situation B") following Known Traffic Situation A. Known Traffic Situation B is, for example, the traffic situation one second after Known Traffic Situation A. Known traffic scenes (scene b) in the knowledge graph DB 32 are generated, for example, based on the traffic context (see Figure 14) obtained by interpreting the data acquired by the data acquisition unit 411 described later in Known Traffic Situation B shown in Figure 13. In Known Traffic Situation B, vehicle 100a has traveled in the driving lane La1 and is slightly closer to intersection ISa compared to scene a. Motorcycle 100b has entered intersection ISa and is turning right at intersection ISa.

[0067] Figure 14 shows an example of a traffic context obtained by interpreting known traffic situation B. The traffic context of known traffic situation B consists of multiple sentences (subject, predicate, object), as shown in Figure 14.

[0068] In Figure 14, "Sceneb, HasTime, 1sec" means "The transition time to scene b is 1 second." "SbjCar, isRunningOn, subjectLane" means "Vehicle 100a is traveling in running lane La1." "subjectLane, nextRoadSegment, SignalIntersection" means "Running lane La1 has a signalized intersection ISa as a road segment ahead of vehicle 100a."

[0069] "SbjCar, HasSpeed, Deceleration" means "Vehicle 100a is traveling at a reduced speed." "SbjCar, ApproachTo, SignalIntersection" means "Vehicle 100a is approaching intersection ISa." "SbjCar, Lighting, RightDirectionIndicator" means "Vehicle 100a has its right-turn indicator flashing."

[0070] "ObjMotorcycle, isRunningOn, OppositeDirectionLane" means "Motorcycle 100b is running in the opposite lane La2." "ObjMotorcycle, HasSpeed, Deceleration" means "Motorcycle 100b is running at a reduced speed." "ObjMotorcycle, EnterIn, SignalIntersection" means "Motorcycle 100b is entering intersection IS." "ObjMotorcycle, islighting, RightDirectionIndicator" means "Motorcycle 100b has its right-turn signal flashing." "SignalIntersection, Light_turns_green, Carmoving" means "Vehicle 100a and motorcycle 100b are in a state of movement when traffic light La1 is lit green."

[0071] Figure 15 shows an example of a known traffic situation (hereinafter referred to as "Known Traffic Situation C") following Known Traffic Situation B. Known Traffic Situation C is, for example, the traffic situation one second after Known Traffic Situation B. The Known Traffic Scene (Scene C) in the Knowledge Graph DB 32 is generated, for example, in Known Traffic Situation C shown in Figure 15, based on the traffic context (see Figure 16) obtained by interpreting the data acquired by the data acquisition unit 411 (various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, and various control signals to various devices of the vehicle 100a).

[0072] In known traffic situation C, vehicle 100a is entering intersection IS and making a right turn. Motorcycle 100b is also making a right turn at intersection IS and is approaching pedestrian crossing CW2 on road Lb.

[0073] Figure 16 shows an example of a traffic context obtained by interpreting a known traffic situation C. The traffic context of a known traffic situation C consists of multiple sentences (subject, predicate, object), as shown in Figure 16.

[0074] In Figure 16, "Scenec, HasTime, 2sec" means "The transition time to scene c is 2 seconds." "SbjCar, isRunningOn, subjectLane" means "Vehicle 100a is traveling in running lane La1." "subjectLane, nextRoadSegment, SignalIntersection" means "Running lane La1 has a signalized intersection IS as a road segment ahead of vehicle 100a."

[0075] "SbjCar, HasSpeed, Deceleration" means "Vehicle 100a is traveling at a reduced speed." "SbjCar, EnterIn, SignalIntersection" means "Vehicle 100a is entering intersection ISa." "SbjCar, Ilighting, RightDirectionIndicator" means "Vehicle 100a has its right-turn indicator flashing."

[0076] "ObjMotorcycle, Approach To, SbjCar" means "Motorcycle 100b is approaching vehicle 100a." "ObjMotorcycle, Has Speed, Constant" means "Motorcycle 100b is traveling at a constant speed." "ObjMotorcycle, Enter In, Signal Intersection" means "Motorcycle 100b is entering intersection IS." "ObjMotorcycle, islighting, Right Direction Indicator" means "Motorcycle 100b is flashing its right turn signal." "SignalIntersection, Light_turns_green, Carmoving" means "Vehicle 100a and motorcycle 100b are in a state of movement when traffic light La1 is lit green."

[0077] The control unit 40 is capable of controlling the entire vehicle 100a. The control unit 40 is, for example, a so-called ECU (Electronic Control Unit) and is composed of, for example, one or more processors and one or more memories. The control unit 40 may also be composed of, for example, a CPU (Central Processing Unit). In this case, the control unit 40 is capable of controlling the entire vehicle 100a by, for example, executing a program stored in a memory unit. The control unit 40 corresponds to one specific example of a "vehicle control device" according to one embodiment of the present disclosure.

[0078] The control unit 40 includes, for example, a locator unit. The locator unit is capable of acquiring the position coordinates of the vehicle 100a based on the positioning signal received through the communication unit 20. The locator unit is capable of estimating the vehicle's position on the road map by map matching the acquired position coordinates onto route map data. Based on the acquired position coordinates of the vehicle 100a, the locator unit is capable of acquiring map data for a predetermined range including the vehicle 100a from the map data stored in the road map DB (database) 31, which will be described later.

[0079] The locator unit can switch to autonomous navigation, which estimates the vehicle's position on a road map based on vehicle speed, angular velocity, and longitudinal acceleration detected by the sensor unit 10, in environments where it is not possible to receive effective positioning signals from positioning satellites due to reduced sensitivity, such as when driving in a tunnel.

[0080] As described above, the locator unit estimates the position of the vehicle 100a on the road map (vehicle position) based on the positioning signal received through the communication unit 20 or the information detected by the sensor unit 10. Based on the estimated vehicle position on the road map, it is possible to determine the type of road the vehicle 100a is traveling on.

[0081] The locator unit can update the road map data stored in the road map DB 31 to the latest state using road map data acquired through external communication (vehicle-to-infrastructure communication and vehicle-to-vehicle communication) via the communication unit 20. This information update is performed not only on static information but also on quasi-static, quasi-dynamic, and dynamic information. As a result, the road map data is composed of road information and traffic information acquired through communication with the outside of the vehicle, and information on moving objects such as vehicles traveling on the road is updated in near real time.

[0082] The locator unit verifies the road map data based on the driving environment information recognized as described above, and can update the road map data stored in the road map DB31 to the latest state. This information update is performed not only on static information, but also on quasi-static information, quasi-dynamic information, and dynamic information. As a result, information on moving objects such as vehicles traveling on the road, as recognized as described above, is updated in real time.

[0083] The control unit 40 includes, for example, a hazard prediction unit 41, as shown in Figure 8. The hazard prediction unit 41 is capable of estimating whether or not the vehicle 100a is in a dangerous traffic situation. If, as a result of the estimation, the vehicle 100a is in a dangerous traffic situation, the hazard prediction unit 41 is capable of predicting hazards based on the most likely dangerous traffic scenarios that the vehicle 100a is facing. The hazard prediction unit 41 is capable of outputting the predicted hazard (hazardous event) to the driving control unit 42. The hazard prediction unit 41 corresponds to one specific example of the "information processing device" according to one embodiment of the present disclosure.

[0084] The risk prediction unit 41 includes, for example, a data acquisition unit 411, a scenario creation unit 412, a similarity determination unit 413, and a knowledge acquisition unit 414, as shown in Figure 8. The data acquisition unit 411 corresponds to one specific example of the "acquisition unit" according to one embodiment of the present disclosure. The scenario creation unit 412, the similarity determination unit 413, and the knowledge acquisition unit 414 correspond to one specific example of the "processing unit" according to one embodiment of the present disclosure.

[0085] The data acquisition unit 411 is capable of periodically acquiring data about the status or condition of the vehicle 100a. Specifically, the data acquisition unit 411 is capable of acquiring various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, and various control signals for various devices of the vehicle 100a. Furthermore, the data acquisition unit 411 is capable of acquiring map data of the area around the vehicle 100a from the road map DB 31 in the storage unit 30.

[0086] The road data surrounding the vehicle 100a, which is included in the various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, various control signals for various devices of the vehicle 100a, and the map data of the area around the vehicle 100a obtained from the road map DB 31, includes, for example, the names (labels), locations, number of lanes and presence or absence of a median strip of roads La and Lb, the names (labels), locations and types of intersections ISa, the names (labels), locations and lighting status of traffic lights TL1 and TL2, and the names (labels) and locations of pedestrian crossings CW1 and CW2.

[0087] Traffic data about vehicle 100a and traffic participants around vehicle 100a, which are included in various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, various control signals to various devices of vehicle 100a, and map data of the area around vehicle 100a obtained from the road map DB 31, include, for example, the name (label), location and speed of vehicle 100a, and the names (labels), locations and speeds of traffic participants around vehicle 100a. This data corresponds to one specific example of "traffic data about the vehicle and traffic participants around the vehicle." The data including the "road data around the vehicle" and the "traffic data about the vehicle and traffic participants around the vehicle" described above corresponds to one specific example of "surrounding conditions data for the vehicle" according to one embodiment of this disclosure.

[0088] The scenario creation unit 412, the similarity determination unit 413, and the knowledge acquisition unit 414 are capable of processing multiple known traffic scenarios read from the storage unit 30 and data acquired by the data acquisition unit 411. Here, the data acquired by the data acquisition unit 411 includes, for example, image data of the area in front of the vehicle 100a (image data Ia and distance image data Ib), map data and situational data of the area around the vehicle 100a.

[0089] The scenario creation unit 412 is capable of generating external environment recognition data as shown in Figure 3, based on the map data and situation data acquired by the data acquisition unit 411. Furthermore, the scenario creation unit 412 is capable of interpreting the traffic context as shown in Figure 4, based on the generated external environment recognition data.

[0090] The scenario creation unit 412 can, for example, generate external environment recognition data based on map data and situation data acquired by the data acquisition unit 411 in the surrounding traffic conditions Sa, and interpret a new traffic context based on the newly generated external environment recognition data. For example, suppose that vehicle 100a is in the surrounding traffic conditions Sa as shown in Figure 2. In this case, the scenario creation unit 412 can interpret the traffic context (see Figure 4) based on the data acquired by the data acquisition unit 411 in the surrounding traffic conditions Sa. For example, the scenario creation unit 412 can generate external environment recognition data based on map data and situation data acquired by the data acquisition unit 411 in the surrounding traffic conditions Sa, and interpret the traffic context based on the generated external environment recognition data.

[0091] The scenario creation unit 412 can, for example, generate new external environment recognition data based on the situation data acquired by the data acquisition unit 411 one second after the surrounding traffic situation Sa, and interpret a new traffic context based on the newly generated external environment recognition data. For example, suppose that vehicle 100a is in a certain traffic situation (hereinafter referred to as "surrounding traffic situation Sb") one second after the surrounding traffic situation Sa, as shown in Figure 5. At this time, the scenario creation unit 412 can interpret the traffic context based on the data acquired by the data acquisition unit 411 in the surrounding traffic situation Sb. The scenario creation unit 412 can, for example, generate new external environment recognition data based on the situation data acquired by the data acquisition unit 411 one second after the surrounding traffic situation Sb, and interpret a new traffic context based on the newly generated external environment recognition data.

[0092] The scenario creation unit 412 can create a numerical representation of the obtained traffic context using the knowledge space DB 33. The scenario creation unit 412 can determine whether or not there are unknown traffic elements (unknown elements UE) in the traffic context that are not included in the knowledge data stored in the memory unit 30. If the scenario creation unit 412 detects an unknown element UE in the traffic context, it can set the label of the unknown element UE to a character indicating that it is an unknown element UE (ObjUnknown).

[0093] The scenario creation unit 412 is capable of generating multiple surrounding traffic scenes with a graph structure from multiple traffic contexts with time series in which the labels of the unknown element UE have been completed in this manner. For example, the scenario creation unit 412 is capable of generating two surrounding traffic scenes (Scene 1, Scene 2) with a graph structure from two traffic contexts with time series in which the labels of the unknown element UE have been completed. Surrounding traffic scene (Scene 1) is a traffic scene corresponding to surrounding traffic condition Sa. Surrounding traffic scene (Scene 2) is a traffic scene corresponding to surrounding traffic condition Sb. The scenario creation unit 412 is further capable of creating a surrounding traffic scenario by integrating the multiple surrounding traffic scenes with time series that have been generated. For example, as shown in Figure 17, the scenario creation unit 412 is capable of creating surrounding traffic scenario II by integrating the two surrounding traffic scenes (Scene 1, Scene 2) with time series that have been generated.

[0094] Surrounding traffic scenario II is composed of multiple integrated time-series surrounding traffic scenes (Scene 1, Scene 2), as shown in Figure 17, for example. In surrounding traffic scenario II, the surrounding traffic scenes (Scene 1, Scene 2) are linked in a temporal order. In surrounding traffic scenario II, each surrounding traffic scene (Scene 1, Scene 2) is composed of multiple traffic elements. Surrounding traffic scene (Scene 1) is composed of multiple traffic elements (traffic elements α1, α2, α3, α4, etc.), as shown in Figure 17, for example. Surrounding traffic scene (Scene 2) is composed of multiple traffic elements (traffic elements β1, β2, β3, β4, etc.), as shown in Figure 17, for example.

[0095] The Peripheral Traffic Scenario II is composed of a directed graph containing multiple nodes (entities) and multiple relations (edges) that connect two nodes (entities). In Peripheral Traffic Scenario II, each entity and each relation is associated with a label, and a sentence (subject, predicate, object) is formed by two entities and one relation that connects these two entities.

[0096] The scenario creation unit 412 can, for example, read out a numerical representation (e.g., spatial vector V1) associated with the same label as each traffic element (e.g., α1, α2, α3, α4, etc.) included in the surrounding traffic scene (scene 1) from the knowledge graph 33A, etc., included in the knowledge space DB 33, for each traffic element. The scenario creation unit 412 can, for example, calculate a new numerical representation (e.g., spatial vector V2) based on the read-out numerical representations (e.g., spatial vector V1), and make the newly calculated numerical representation (e.g., spatial vector V2) the numerical representation (e.g., spatial vector V3) corresponding to the surrounding traffic scene (scene 1). The scenario creation unit 412 can, for example, calculate the centroid vector of the read-out vector values, and make the calculated centroid vector the spatial vector V3 corresponding to the surrounding traffic scene (scene 1).

[0097] The scenario creation unit 412 can, for example, read out numerical representations (e.g., spatial vector V4) associated with the same labels as each traffic element (e.g., β1, β2, β3, β4, etc.) included in the surrounding traffic scene (scene 2) from the knowledge graph 33A, etc., included in the knowledge space DB 33, for each traffic element. The scenario creation unit 412 can, for example, calculate a new numerical representation (e.g., spatial vector V5) based on the read-out numerical representations (e.g., spatial vector V4), and make the newly calculated numerical representation (e.g., spatial vector V5) the numerical representation (e.g., spatial vector V6) corresponding to the surrounding traffic scene (scene 2). The scenario creation unit 412 can, for example, calculate the centroid vector of the read-out vectors, and make the calculated centroid vector the spatial vector V6 corresponding to the surrounding traffic scene (scene 2). The surrounding traffic scenes (scene 1, scene 2) are represented by spatial vectors common to the spatial vectors in the knowledge space DB 33.

[0098] For example, if the scenario creation unit 412 detects an unknown element UE in a sentence included in the traffic context, it can read numerical representations of multiple known elements other than the unknown element UE from the knowledge space DB 33 among the multiple traffic elements that constitute the sentence containing the unknown element UE. The scenario creation unit 412 can derive a numerical representation of the unknown element UE by performing calculations using the numerical representations of the multiple known elements that have been read.

[0099] The similarity determination unit 413 is capable of calculating the similarity Sm between each known traffic scenario included in the knowledge graph DB 32 or knowledge space DB 33 and the surrounding traffic scenario created by the scenario creation unit 412.

[0100] The similarity determination unit 413 can, for example, read numerical representations (e.g., spatial vectors Va, Vb, Vc) of each known traffic scene (scene a, scene b, scene c) that constitutes the known traffic scenario I from the knowledge space DB 33. Spatial vector Va is the spatial vector of the known traffic scene (scene a), and is the spatial vector of the first scene in the known traffic scenario I. Spatial vector Vb is the spatial vector of the known traffic scene (scene b), and is the spatial vector of the second scene in the known traffic scenario I. Spatial vector Vc is the spatial vector of the known traffic scene (scene c), and is the spatial vector of the third scene in the known traffic scenario I.

[0101] The similarity determination unit 413 can calculate the similarity S1 between each known traffic scene and each surrounding traffic scene by comparing the numerical representation of each known traffic scene read from the knowledge space DB 33 with the numerical representation of each surrounding traffic scene created by the scenario creation unit 412.

[0102] The similarity determination unit 413 can calculate the similarity S1 between each known traffic scene (scene a, scene b, scene c) and each surrounding traffic scene (scene 1, scene 2) by, for example, comparing the numerical representation (e.g., spatial vectors Va, Vb, Vc) of each known traffic scene (scene a, scene b, scene c) read from the knowledge space DB 33 with the numerical representation (e.g., spatial vectors V3, V6) of each surrounding traffic scene (scene 1, scene 2) created by the scenario creation unit 412. The similarity determination unit 413 can calculate the similarity S1 between each spatial vector Va, Vb, Vc and spatial vector V3, and the similarity S1 between each spatial vector Va, Vb, Vc and spatial vector V6.

[0103] The similarity determination unit 413 can calculate the similarity S1 by, for example, applying a first-order norm, a second-order norm, or cosine similarity to the spatial vectors Va, Vb, Vc and the spatial vectors V2, V4. The similarity determination unit 413 can, for example, derive the difference between the first-order norm of each spatial vector Va, Vb, Vc and the first-order norm of each spatial vector V2, V4, and use the derived difference as the similarity S1. The similarity determination unit 413 may also, for example, derive the difference between the second-order norm of each spatial vector Va, Vb, Vc and the second-order norm of each spatial vector V2, V4, and use the derived difference as the similarity S1. The similarity determination unit 413 may, for example, derive the difference between the cosine similarity of each spatial vector Va, Vb, and Vc and the cosine similarity of each spatial vector V2 and V4, and use the derived difference as the similarity S1.

[0104] The similarity determination unit 413 calculates a similarity S2 based on the calculated similarity S1 for each known traffic scene, and the calculated similarity S2 can be used as the similarity Sm of the surrounding traffic scenario to the known traffic scene.

[0105] The similarity determination unit 413 can, for example, calculate the similarity S1 between each known traffic scene (scene a, scene b, scene c) in known traffic scenario I and the surrounding traffic scene (scene 1) in surrounding traffic scenario II, and set the largest similarity among the multiple calculated similarity S1s as similarity Sm1, and set the known traffic scene having similarity Sm1 as scene m1. Scene m1 is, for example, scene a. The similarity determination unit 413 can, for example, calculate the similarity S1 between each known traffic scene (scene a, scene b, scene c) in known traffic scenario I and the surrounding traffic scene (scene 2) in surrounding traffic scenario II, and set the largest similarity among the multiple calculated similarity S1s as similarity Sm2, and set the known traffic scene having similarity Sm2 as scene m2. Scene m2 is, for example, scene b. The similarity determination unit 413 can, for example, calculate similarity S2 using similarity Sm1 and similarity Sm2. The similarity determination unit 413 can, for example, use the average of similarity Sm1 and similarity Sm2 as similarity S2. The similarity determination unit 413 can, for example, use similarity S2 as the similarity Sm for known traffic scenes in surrounding traffic scenario II. The similarity determination unit 413 can, for example, calculate similarity S2 in the same manner as above for other known traffic scenarios, and use the calculated similarity S2 as the similarity Sm for known traffic scenes in surrounding traffic scenarios.

[0106] The similarity determination unit 413 is capable of calculating the similarity Sm between each known traffic scenario included in the knowledge graph DB 32 or knowledge space DB 33 and the surrounding traffic scenario. If the similarity Sm is less than the threshold Sth for any known traffic scenario, the similarity determination unit 413 can use the image data (image data Ia or distance image data Db) and the surrounding traffic scenario as transmission data Ds to be sent to the server device 200.

[0107] The knowledge acquisition unit 414 is capable of outputting transmission data Ds, which includes image data (image data Ia or distance image data Ib) and a surrounding traffic scenario, to the communication unit 20. When the communication unit 20 receives the transmission data Ds from the knowledge acquisition unit 414, it is capable of transmitting the acquired transmission data Ds to the server device 300. In response to this transmission, the communication unit 20 is capable of receiving a new surrounding traffic scenario from the server device 300 that has a predetermined relationship to the image data (image data Ia or distance image data Ib) and the surrounding traffic scenario. The communication unit 20 is capable of outputting the received new surrounding traffic scenario to the knowledge acquisition unit 414.

[0108] The knowledge acquisition unit 414 is capable of storing new surrounding traffic scenarios acquired from the server device 300 via the communication unit 20 in the storage unit 30. The knowledge acquisition unit 414 is capable of describing the new surrounding traffic scenarios in a graph structure and storing the resulting knowledge data in the storage unit 30 as a knowledge graph DB 32. The knowledge acquisition unit 414 is capable of representing the new surrounding traffic scenarios numerically (for example, spatially) and storing the resulting knowledge data in a knowledge space DB 33.

[0109] The knowledge acquisition unit 414 is capable of calculating a similarity Sx between each known traffic scenario included in the knowledge space DB 33 and a new surrounding traffic scenario acquired from the server device 300. The similarity Sx can be calculated in the same way as the similarity Sm calculation method described above. The knowledge acquisition unit 414 is capable of determining whether the similarity Sx value of each known traffic scenario included in the knowledge space DB 33 is smaller than a predetermined threshold Sth. If there is one or more similarity Sx values ​​among the calculated multiple similarity Sx values ​​that are greater than or equal to the threshold Sth, the knowledge acquisition unit 414 can select the known traffic scenario with the largest similarity Sx among the one or more similarity Sx values ​​that are greater than or equal to the threshold Sth as the driving support traffic scenario. If the similarity Sx is less than the threshold Sth for any of the known traffic scenarios included in the knowledge space DB 33, the knowledge acquisition unit 414 can store the new surrounding traffic scenario in the storage unit 30 and select the new surrounding traffic scenario as the driving support traffic scenario.

[0110] The knowledge acquisition unit 414 is capable of setting the intensity of at least one of the hazard warning control and hazard avoidance control based on the hazard level of the driving assistance traffic scenario. The knowledge acquisition unit 414 is capable of outputting the set intensity to the driving control unit 42, which will be described later.

[0111] The control unit 40 further includes a driving control unit 42, as shown in Figure 8, for example. The driving control unit 42 is capable of controlling the driving of the vehicle 100a (for example, the torque of the prime mover 50, the amount of brake pedal depression, and the steering angle of the steering wheel) and providing notifications to the driver of the vehicle 100a. The driving control unit 42 is capable of performing driving control and notification control using data acquired by the data acquisition unit 411 and data obtained by the knowledge acquisition unit 414.

[0112] The driving control unit 42 is capable of calculating a correction torque to correct the requested torque given to the accelerator control unit 421 (described later) based on the data acquired by the data acquisition unit 411 and the data obtained from the knowledge acquisition unit 414. The driving control unit 42 is capable of calculating a correction torque to correct the requested torque given to the brake control unit 422 (described later) based on the data acquired by the data acquisition unit 411 and the data obtained from the knowledge acquisition unit 414. The driving control unit 42 is capable of calculating a correction torque to correct the steering assist torque generated by the steering control unit 423 (described later) based on the data acquired by the data acquisition unit 411 and the data obtained from the knowledge acquisition unit 414. The driving control unit 42 is capable of generating notification data to notify the driver of the vehicle 100a based on the data acquired by the data acquisition unit 411 and the data obtained from the knowledge acquisition unit 414.

[0113] The driving control unit 42 includes, for example, an accelerator control unit 421, a brake control unit 422, a steering control unit 423, and a notification control unit 424, as shown in Figure 8.

[0114] The accelerator control unit 421 is capable of controlling the torque of the prime mover 50 based on the required torque corresponding to the amount the driver of the vehicle 100a depresses the accelerator pedal. The accelerator control unit 421 is also capable of controlling the torque of the prime mover 50 based on a target torque which is the required torque plus a correction torque. The prime mover 50 is configured to drive the steering wheels of the vehicle 100a and is capable of driving the steering wheels of the vehicle 1 according to the required torque or target torque input from the accelerator control unit 421.

[0115] The brake control unit 422 is capable of controlling the torque of the brake 60 based on the required torque corresponding to the amount the driver of the vehicle 100a presses the brake pedal. The brake control unit 422 is also capable of controlling the torque of the brake 60 based on a target torque which is the required torque plus a correction torque. The brake 60 is configured to brake the steering wheels of the vehicle 100a and is capable of braking the steering wheels of the vehicle 100a according to the required torque or target torque input from the brake control unit 422.

[0116] The steering control unit 423 can derive a steering assist torque to assist the steering torque generated by the driver's steering wheel operation, and set an EPS torque corresponding to the derived steering assist torque. The steering control unit 423 can output a control signal to the EPS motor 70 so that the output torque of the EPS motor 70 becomes the set EPS torque. The steering control unit 423 can output a control signal to the EPS motor 70 so that the output torque of the EPS motor 70 becomes the EPS torque considering the correction torque. The EPS motor 70 generates an output torque based on the input control signal and can control the steering angle of the steering wheel.

[0117] The accelerator control unit 421, brake control unit 422, and steering control unit 423 may include, for example, a CPU. In this case, the accelerator control unit 421, brake control unit 422, and steering control unit 423 can perform the various driving controls described above by, for example, executing control software stored in a memory unit.

[0118] The notification control unit 424 is capable of outputting notification data to the notification unit 80 for notification to the driver of the vehicle 100a. The notification control unit 424 is capable of generating a video signal including the notification data and outputting it to the notification unit 80. The notification control unit 424 is capable of generating an audio signal including the notification data and outputting it to the notification unit 80. The notification unit 80 is composed of, for example, a display panel and a speaker. When the notification unit 80 receives the video signal from the control unit 40 (notification control unit 424), it is capable of displaying an image on the display screen corresponding to the input video signal. When the notification unit 80 receives the audio signal from the control unit 40 (notification control unit 424), it is capable of outputting an audio from the speaker corresponding to the input audio signal.

[0119] Next, the server device 200 will be described. The server device 200 includes, for example, a communication unit 210, a storage unit 220, and a signal processing unit 230, as shown in Figure 18.

[0120] The communication unit 210 is a communication interface capable of receiving data from the driving control device 1000 via the network NW and transmitting data obtained by the signal processing unit 230 to the driving control device 1000. The communication unit 210 can receive transmission data Ds from the driving control device 1000 and output the received transmission data Ds to the signal processing unit 230. The communication unit 210 can acquire a new surrounding traffic scenario from the signal processing unit 230 and transmit the acquired new surrounding traffic scenario to the driving control device 1000 as a response to the reception of the transmission data Ds. Furthermore, the communication unit 210 is connected to the internet and is also a communication interface capable of receiving multiple network image data containing various moving objects from the internet. Metadata is associated with the network image data acquired by the communication unit 210.

[0121] The storage unit 220 is composed of, for example, non-volatile memory, such as EEPROM, flash memory, or resistive random-access memory. The storage unit 220 stores, for example, internet information 221, as shown in Figure 18. The internet information 221 includes, for example, multiple net image data of various moving objects collected on the internet. Metadata is associated with each net image data included in the internet information 221. The metadata includes, for example, features of the moving object captured in the corresponding net image data. Features of the moving object included in the metadata include, for example, the size of the moving object and the method of transportation of the moving object based on the Road Traffic Act. Among the multiple net image data included in the internet information 221, for example, image data of an electric kick scooter is included.

[0122] The signal processing unit 230 includes an internet search unit 231, a matching processing unit 232, and a scenario creation unit 233.

[0123] The Internet search unit 231 is capable of periodically searching for and collecting internet image data on the Internet via the communication unit 210. The Internet search unit 231 is also capable of storing the collected internet image data in the Internet information 221 of the storage unit 30.

[0124] The matching processing unit 232 is capable of performing image similarity matching between the image data (image data Ia or distance image data Ib) included in the transmitted data Ds and multiple net image data in the storage unit 220. The matching processing unit 232 can compare the feature quantities of moving objects captured in the image data included in the transmitted data Ds (first feature quantities) with the feature quantities of moving objects captured in each net image data (second feature quantities), and use the difference between the first feature quantities and the second feature quantities as the similarity of each net image data. The scenario creation unit 233 can acquire the metadata of the net image data with the highest similarity and generate a new surrounding traffic scenario based on the surrounding traffic scenario included in the transmitted data Ds and the acquired metadata. For example, the scenario creation unit 233 can replace the names of moving objects included in the metadata with the labels of unknown elements UE included in the surrounding traffic scenario, or correct the movement of unknown elements UE included in the surrounding traffic scenario based on the traffic methods of moving objects based on the Road Traffic Act included in the metadata. The scenario creation unit 233 is capable of outputting the generated new surrounding traffic scenario to the communication unit 210.

[0125] (Driving Assistance Procedure) Next, the driving assistance procedure in vehicle 100a will be described with reference to Figures 19, 20, and 21. Figures 19, 20, and 21 show an example of the driving assistance procedure in vehicle 100a.

[0126] Vehicle 100a (hazard prediction unit 31) first acquires various data obtained from the sensor unit 10, various data obtained from the outside via the communication unit 20, and various control signals for various devices of vehicle 100a. Based on the acquired data, vehicle 100a acquires location data of vehicle 100a and traffic participants around vehicle 100a. Vehicle 100a further acquires surrounding map data including the acquired location of vehicle 100a from the road map data DB 31 of the storage unit 30. Vehicle 100a further acquires image data Ia or distance image data Ib from the stereo camera. From the map data and image data Ia or distance image data Ib, etc., vehicle 100a acquires attribute data and location data of each structure constituting the road around vehicle 100a, and attribute data and location data of structures on the road around vehicle 100a. In this way, vehicle 100a acquires image data of the area in front of vehicle 100a (image data Ia or distance image data Ib), map data of the area around vehicle 100a, and situational data of the area around vehicle 100a (step S101).

[0127] Next, vehicle 100a generates external environment recognition data based on acquired image data, map data, and situation data, and interprets the traffic context based on the generated external environment recognition data (step S102). Vehicle 100a, for example, reinterprets the traffic context based on periodically obtained image data, map data, and situation data.

[0128] Vehicle 100a determines whether an unknown element UE exists among the multiple traffic elements that constitute the traffic context (step S103). If vehicle 100a detects an unknown element UE in one of the sentences included in the traffic context (step S103; Y), it sets the label of the unknown element UE to a character indicating that it is an unknown element UE (ObjUnknown). In this way, vehicle 100a completes the unknown element UE (step S104).

[0129] Vehicle 100a generates multiple surrounding traffic scenes in a graph structure from multiple time-series traffic contexts in which the unknown element UE has been complemented (step S105). Subsequently, vehicle 100a integrates the generated multiple surrounding traffic scenes in a time-series to generate a surrounding traffic scenario (step S106).

[0130] Next, vehicle 100a calculates the similarity Sm between each known traffic scenario included in the knowledge space DB33 and the generated surrounding traffic scenario (step S107). Vehicle 100a determines whether the similarity Sm value of each known traffic scenario included in the knowledge space DB33 is less than a predetermined threshold Sth (step S108). If there is one or more similarity Sms among the calculated multiple similarity Sms that are greater than or equal to the threshold Sth (step S108; N), vehicle 100a selects the known traffic scenario with the largest similarity Sm among the one or more similarity Sms that are greater than or equal to the threshold Sth as the driving assistance traffic scenario (step S120).

[0131] Vehicle 100a generates transmission data Ds, which includes image data (image data Ia or distance image data Ib) and surrounding traffic scenarios, if the similarity Sm is less than the threshold Sth in any known traffic scenario included in the knowledge space DB33 (step S108; Y). Vehicle 100a transmits the generated transmission data Ds to the server device 200 (step S109).

[0132] The server device 200 (signal processing unit 230) receives the transmission data Ds from the vehicle 100a (step S110). The server device 200 performs image similarity matching on the image data contained in the transmission data Ds and on a plurality of net image data read from the storage unit 220 (step S111). The server device 200 obtains metadata for the net image data with the highest similarity as a result of the image similarity matching (step S112). The server device 200 generates a new surrounding traffic scenario based on the surrounding traffic scenario contained in the transmission data Ds and the acquired metadata (step S113). The server device 200 transmits the generated new surrounding traffic scenario to the vehicle 100a (step S114).

[0133] Vehicle 100a receives a new surrounding traffic scenario from the server device 200 (step S115). For each known traffic scenario included in the knowledge space DB 33, vehicle 100a calculates the similarity Sx with the received new surrounding traffic scenario (step S116). The similarity Sx can be calculated in the same way as the similarity Sm calculation method described above. Vehicle 100a determines whether the similarity Sx value of each known traffic scenario included in the knowledge space DB 33 is less than a predetermined threshold Sth (step S117). If there is one or more similarity Sx values ​​greater than or equal to the threshold Sth among the calculated similarity Sx values ​​(step S117; N), vehicle 100a selects the known traffic scenario with the largest similarity Sx among the one or more similarity Sx values ​​greater than or equal to the threshold Sth as the driving assistance traffic scenario (step S120).

[0134] If the similarity Sx of any known traffic scenario included in the knowledge space DB33 is less than the threshold Sth (step S117; Y), vehicle 100a stores the new surrounding traffic scenario in the memory unit 30 (step S118). Subsequently, vehicle 100a uses the new surrounding traffic scenario as a driving assistance traffic scenario (step S119).

[0135] Vehicle 100a sets the intensity of at least one of the hazard warning control and hazard avoidance control based on the hazard level of the driver assistance traffic scenario defined in steps S120 and S119 (step S121). Vehicle 100a then implements the hazard warning control and hazard avoidance control according to the set intensity (step S122).

[0136] Vehicle 100a performs driving control and notification control using various data acquired by the data acquisition unit 411, a driving support traffic scenario, and the intensity set in the driving support traffic scenario. Based on the various data acquired by the data acquisition unit 411, the driving support traffic scenario, and the intensity set in the driving support traffic scenario, vehicle 100a calculates a correction torque to correct the requested torque given to the accelerator control unit 421. Based on the various data acquired by the data acquisition unit 411, the driving support traffic scenario, and the intensity set in the driving support traffic scenario, vehicle 100a calculates a correction torque to correct the requested torque given to the brake control unit 422. Based on the various data acquired by the data acquisition unit 411, the driving support traffic scenario, and the intensity set in the driving support traffic scenario, vehicle 100a corrects the steering assist torque generated by the steering control unit 423. Vehicle 100a generates notification data to be notified to the driver of vehicle 100a based on various data acquired by the data acquisition unit 411, a driving support traffic scenario, and the intensity set for the driving support traffic scenario.

[0137] The accelerator control unit 421 controls the torque of the prime mover 50 based on a target torque which is the required torque plus a correction torque. The prime mover 50 drives the steering wheels of the vehicle 100a according to the target torque input from the accelerator control unit 421. The brake control unit 422 controls the torque of the brake 60 based on a target torque which is the required torque plus a correction torque. The brake 60 brakes the steering wheels of the vehicle 100a according to the target torque input from the brake control unit 422. The steering control unit 423 outputs a control signal to the EPS motor 70 so that the output torque of the EPS motor 70 becomes the EPS torque which takes the correction torque into account. The EPS motor 70 generates an output torque based on the input control signal and controls the steering angle of the steering wheel.

[0138] The notification control unit 424 generates a video signal including the notification data and outputs it to the notification unit 80. The notification control unit 424 further generates an audio signal including the notification data and outputs it to the notification unit 80. When the notification unit 80 receives the video signal from the control unit 40 (notification control unit 424), for example, it displays an image on the display screen corresponding to the input video signal. When the notification unit 80 receives the audio signal from the control unit 40 (notification control unit 424), for example, it outputs an audio from the speaker corresponding to the input audio signal. In this way, driving assistance is performed in the vehicle 100a.

[0139] [Effects] Next, the effects of a vehicle 100a according to one embodiment of the present disclosure will be described.

[0140] In this embodiment, for each known traffic scenario stored in the memory unit 30, a similarity Sm is calculated between it and the created surrounding traffic scenario. If the similarity Sm is less than a threshold for any known traffic scenario, transmission data Ds, including image data and the surrounding traffic scenario, is sent to the server device 200. A new surrounding traffic scenario is received as a response to the transmission of the transmission data Ds. As a result, when an unknown element UE is detected in the vehicle 100a, danger warning control and danger avoidance control can be implemented based on the new surrounding traffic scenario in which the unknown element UE has been replaced with a known element. Thus, in this embodiment, logical inference can be operated even when a traffic context including an unknown element UE is obtained.

[0141] In this embodiment, the server device 200 performs image similarity matching on image data (image data Ia and distance image data Ib) and multiple net image data acquired from the storage unit 220. A new traffic scenario is then generated based on the metadata of the net image data with the highest similarity Sx and the surrounding traffic scenario obtained from the vehicle 100a. As a result, when an unknown element UE is detected in the vehicle 100a, the vehicle 100a can be provided with a new surrounding traffic scenario in which the unknown element UE is replaced with a known element. Consequently, the vehicle 100a can perform hazard warning control and hazard avoidance control based on the new surrounding traffic scenario. Thus, in this embodiment, logical inference can be operated even when a traffic context containing an unknown element UE is obtained.

[0142] <3. Modified Examples> Next, modified examples of the above embodiment will be described.

[0143] [Modification 3-1] In the above embodiment, the control unit 40 may have, for example, a new scenario creation unit 43 having an internet search unit 231, a matching processing unit 232, and a scenario creation unit 233, as shown in Figure 22. Furthermore, the storage unit 30 may store internet information 221, for example, as shown in Figure 22.

[0144] In this modified version, the functions of the server 200 are provided in the driving control device 1000. This allows logical inference to be performed without the cooperation of the server device 200, even when a traffic context including an unknown element UE is obtained.

[0145] [Modification 3-2] In the above embodiment, the server device 200 may, for example, be equipped with a storage unit 240 instead of a storage unit 220 and a signal processing unit 250 instead of a signal processing unit 230, as shown in Figure 23.

[0146] The storage unit 240 is composed of, for example, non-volatile memory, such as EEPROM, flash memory, or resistive random-access memory. The storage unit 240 stores, for example, a knowledge graph DB 241, a knowledge space DB 242, and a language processing AI 243, as shown in Figure 23.

[0147] Knowledge Graph DB241 and Knowledge Space DB242 are knowledge data that structure traffic rules, reasoning rules, and general knowledge, respectively, in a manner that can be used for analyzing traffic conditions. The knowledge data in Knowledge Graph DB241 and Knowledge Space DB242 includes data on the most recent moving objects.

[0148] Knowledge Graph DB241 contains knowledge data that describes multiple master traffic scenarios in a graph structure. A master traffic scenario refers to a hypothetical traffic scenario or a past traffic scenario. In Knowledge Graph DB241, the master traffic scenario is described as a condition term, and possible events (e.g., dangerous events) are described as result terms. A master traffic scenario is composed of multiple known traffic scenes in an integrated time series.

[0149] Knowledge space DB242 contains knowledge data that numerically represents (e.g., spatially represents) multiple master traffic scenarios included in knowledge graph DB241. Knowledge space DB242 also contains, for example, a knowledge graph that represents known traffic scenarios included in knowledge graph DB241 as spatial vectors.

[0150] The language processing AI 243 is a learning model that, upon receiving at least image data (image data Ia and distance image data Ib) and surrounding traffic data, can output a situational interpretation of the surrounding traffic conditions of vehicle 100a at the time the image data was acquired. The surrounding traffic data is data that describes the surrounding traffic conditions of vehicle 100a using at least one of numerical and character values. The language processing AI 243 is configured, for example, by a deep neural network (DNN).

[0151] In this modified example, the data acquisition unit 411 is capable of generating and acquiring surrounding traffic data based on the output of the sensor unit 10 (various data obtained from the sensor unit 10) and map data of the area around the vehicle 100a obtained from the road map DB 31. In this modified example, the similarity determination unit 413 is capable of transmitting at least the image data (image data Ia or distance image data Ib) as the transmission data Ds when the similarity Sm is less than the threshold Sth in any known traffic scenario. The similarity determination unit 413 is capable of outputting the transmission data Ds, which includes at least the image data (image data Ia or distance image data Ib) as the transmission data Ds, to the communication unit 20.

[0152] The signal processing unit 250 includes, for example, a situation interpretation unit 251 and a scenario creation unit 252, as shown in Figure 23.

[0153] The situation interpretation unit 251 is capable of inputting data obtained from the vehicle 100a to the language processing AI 243. The data input to the language processing AI 243 consists of image data (image data Ia and distance image data Ib) and surrounding traffic data, with at least image data being included. In response to the input to the language processing AI 243, the situation interpretation unit 251 is capable of obtaining a situation interpretation statement from the language processing AI 243 describing the surrounding traffic conditions of the vehicle 100a at the time the image data was acquired.

[0154] The scenario creation unit 252 is capable of generating a new surrounding traffic scenario based on the above situation interpretation statement. The scenario creation unit 252 is capable of generating a traffic context based on the above situation interpretation statement. The scenario creation unit 252 is capable of generating a surrounding traffic scene with a graph structure based on the generated time-series traffic contexts and the knowledge graph DB 241 (multiple master traffic scenarios). The scenario creation unit 252 is capable of generating a new surrounding traffic scenario by integrating the generated time-series surrounding traffic scenes based on the generated time-series surrounding traffic scenes and the knowledge space DB 242 (multiple master traffic scenarios). The scenario creation unit 252 is capable of outputting the generated new surrounding traffic scenario to the communication unit 210.

[0155] (Driving Assistance Procedure) Next, the driving assistance procedure in vehicle 100a will be described with reference to Figure 24. Figure 24 shows an example of the driving assistance procedure following Figure 19.

[0156] If, as a result of performing step S108, vehicle 100a finds that the similarity Sm is less than the threshold Sth in any known traffic scenario included in the knowledge space DB33 (step S108; Y), it generates transmission data Ds which includes at least image data from the image data (image data Ia or distance image data Ib) and surrounding traffic data. Vehicle 100a transmits the generated transmission data Ds to the server device 200 (step S109).

[0157] The server device 200 (signal processing unit 250) receives the transmission data Ds from the vehicle 100a (step S110). The server device 200 inputs the transmission data Ds received from the vehicle 100a to the language processing AI 243 and obtains a situation interpretation statement from the language processing AI 243 regarding the surrounding traffic conditions of the vehicle 100a when the image data was acquired (step S123). Based on the above situation interpretation statement, the server device 200 generates a new surrounding traffic scenario (step S124). The server device 200 transmits the generated new surrounding traffic scenario to the vehicle 100a (step S114).

[0158] In this modified example, for each known traffic scenario stored in the memory unit 30, a similarity Sm is calculated between the new scenario and the created surrounding traffic scenario. If the similarity Sm is less than a threshold for any known traffic scenario, transmission data Ds, which includes at least the image data, is sent to the server device 200. A new surrounding traffic scenario is received in response to the transmission of the transmission data Ds. As a result, when an unknown element UE is detected in the vehicle 100a, hazard warning control and hazard avoidance control can be implemented based on the new surrounding traffic scenario in which the unknown element UE has been replaced with a known element. Thus, in this embodiment, logical inference can be operated even when a traffic context containing an unknown element UE is obtained.

[0159] [Modification 3-3] In the above embodiment, the storage unit 30 may further store, for example, a rear-end collision priority list 34, a head-on collision priority list 35, a cutting-in accident priority list 36, a right-turn accident priority list 37, an accident statistics frequency table 38, and a write list 39, as shown in Figure 25. In the rear-end collision priority list 34, the head-on collision priority list 35, the cutting-in accident priority list 36, and the right-turn accident priority list 37, for example, as shown in Figures 26(A), 26(B), 26(C), and 26(D), the multiple traffic scenarios stored in the storage unit 30 are classified according to the type of accident.

[0160] The rear-end collision priority list 34 includes multiple traffic scenarios related to rear-end collisions. The intersection collision priority list 35 includes multiple traffic scenarios related to intersection collisions. The cut-in accident priority list 36 includes multiple traffic scenarios related to cut-in accidents. The right-turn accident priority list 37 includes multiple traffic scenarios related to right-turn accidents. In each of the rear-end collision priority list 34, intersection collision priority list 35, cut-in accident priority list 36, and right-turn accident priority list 37, multiple traffic scenarios are arranged in descending order of abstraction, for example, as shown in Figures 26(A), 26(B), 26(C), and 26(D).

[0161] Here, the level of abstraction refers to an indicator that shows how well a given accident type covers multiple traffic scenarios. For example, the element most commonly found among multiple hazardous situation patterns included in a particular accident type is designated as the first common element, the second most commonly found element among these hazardous situation patterns is designated as the second common element, and the nth most commonly found element among these hazardous situation patterns is designated as the nth common element. In this case, a traffic scenario that contains a high proportion of high-ranking common elements is assigned a high level of abstraction. On the other hand, a traffic scenario that contains only a low proportion of high-ranking common elements is assigned a low level of abstraction. Theoretically, a traffic scenario consisting only of the first common element is assigned the highest level of abstraction, and a traffic scenario that contains no common elements at all is assigned the lowest level of abstraction.

[0162] In the accident statistics frequency table 38, for example, as shown in Figure 27, the proportion of accident occurrences is assigned to each type of accident. In the write list 39, if there is a limit to the capacity that can be used to write traffic scenarios to the memory unit 30, the number of traffic scenarios that can be written within that limit and the number of traffic scenarios for each type of accident that can be written are specified. In the write list 39, for example, as shown in Figure 28, two traffic scenarios for rear-end collisions are assigned, two traffic scenarios for intersection collisions are assigned, and one traffic scenario for cutting in is assigned.

[0163] The knowledge acquisition unit 414 is capable of identifying accident patterns in new surrounding traffic scenarios. For example, the knowledge acquisition unit 414 can identify accident patterns in new surrounding traffic scenarios by analyzing them. Furthermore, the knowledge acquisition unit 414 is capable of deriving the level of abstraction of new surrounding traffic scenarios. For example, the knowledge acquisition unit 414 can determine the extent to which common elements are included in new surrounding traffic scenarios, and thereby derive the level of abstraction of new surrounding traffic scenarios.

[0164] The knowledge acquisition unit 414 is capable of setting a write capacity limit for each hazard type according to the frequency of occurrence of each hazard type. Specifically, the knowledge acquisition unit 414 is capable of setting a write capacity limit for each hazard type based on the priority list for each accident type (rear-end collision priority list 34 to accident statistics frequency table 38) and the write list 39 stored in the memory unit 30. The knowledge acquisition unit 414 is capable of storing new surrounding traffic scenarios in the memory unit 30 based on the level of abstraction of the hazard type, as long as the set write capacity limit for each hazard type is not exceeded. Specifically, the knowledge acquisition unit 414 is capable of comparing the level of abstraction of each scenario included in the priority list corresponding to the accident type of the new surrounding traffic scenario with the level of abstraction of the new surrounding traffic scenario, and determining the rank of the new surrounding traffic scenario in terms of abstraction. The knowledge acquisition unit 414 is capable of determining whether the rank obtained by the determination exceeds the maximum number of scenarios with the same accident type as the accident type of the new surrounding traffic scenario that can be included in the write list 39. The knowledge acquisition unit 414 can store a new surrounding traffic scenario in the memory unit 30 if the ranking obtained by the determination does not exceed the maximum number mentioned above. The knowledge acquisition unit 414 can not store a new surrounding traffic scenario in the memory unit 30 if the ranking obtained by the determination exceeds the maximum number mentioned above.

[0165] (Driving Assistance Procedure) Next, the driving assistance procedure in vehicle 100a will be explained with reference to Figure 29. Figure 29 shows an example of the driving assistance procedure following Figure 20.

[0166] Vehicle 100a determines whether the similarity Sx value of each known traffic scenario included in the knowledge space DB33 is less than a predetermined threshold Sth (step S117). If there is one or more similarity Sx values ​​greater than or equal to the threshold Sth among the calculated multiple similarity Sx values ​​(step S117; N), Vehicle 100a selects the known traffic scenario with the largest similarity Sx value among the one or more similarity Sx values ​​greater than or equal to the threshold Sth as the driving assistance traffic scenario (step S120).

[0167] If the similarity Sx of any known traffic scenario included in the knowledge space DB33 is less than the threshold Sth (step S117; Y), the vehicle 100a selects the known traffic scenario with the largest similarity Sm among one or more similarity Sm values ​​greater than or equal to the threshold Sth as the driving assistance traffic scenario (step S120). The vehicle 100a determines whether it is possible to write the new surrounding traffic scenario to the storage unit 30 (step S123). If the vehicle 100a is unable to write the new surrounding traffic scenario to the storage unit 30 (step S123; N), it selects the known traffic scenario with the largest similarity Sm among one or more similarity Sm values ​​greater than or equal to the threshold Sth as the driving assistance traffic scenario (step S120). If the vehicle 100a is able to write the new surrounding traffic scenario to the storage unit 30 (step S123; Y), it stores the new surrounding traffic scenario in the storage unit 30 (step S118). Vehicle 100a uses a new surrounding traffic scenario as the driver assistance traffic scenario (step S119).

[0168] Vehicle 100a sets the intensity of at least one of the hazard warning control and hazard avoidance control based on the hazard level of the driving assistance traffic scenario defined in steps S120 and S119 (step S121). Vehicle 100a performs hazard warning control and hazard avoidance control according to the set intensity (step S122). Vehicle 100a performs driving control and warning control in the same manner as in the above embodiment. In this way, driving assistance is performed in vehicle 100a.

[0169] In this modified version, a write capacity limit is set for each hazard type according to the frequency of occurrence of each hazard type, and new surrounding traffic scenarios are stored in the memory unit 30 based on the level of abstraction of the hazard type, within the limits of the set write capacity limit for each hazard type. This makes it possible to provide driving assistance that takes into account the characteristics of the area in which the vehicle 100a is traveling, even if there is a limit to the write capacity of the memory unit 30.

[0170] [Modification 3-4] In modification 3-1 described above, the memory unit 30 may further store, for example, a rear-end collision priority list 34, an intersection collision priority list 35, a cutting-in collision priority list 36, a right-turn collision priority list 37, an accident statistics frequency table 38, and a write list 39, as shown in Figure 30. In this case, similar to modification 3-3 described above, even if there is a limit to the write capacity of the memory unit 30, driving assistance that takes into account the characteristics of the area in which the vehicle 100a travels can be provided.

[0171] The present disclosure has been described above with reference to embodiments and their modifications, but the present disclosure is not limited to these embodiments, and various modifications are possible. The effects described herein are merely illustrative, and the effects of the present disclosure are not limited to those described herein. Therefore, other effects may be obtained with respect to the present disclosure.

[0172] The above embodiments and their modifications were based on the premise of a country or region with traffic regulations in which vehicle 100a etc. travels in the left lane. However, if the premise is a country or region with traffic regulations in which vehicle 100a etc. travels in the right lane, then in the first and second embodiments and their modifications, "right" shall be read as "left" and "left" as "right".

[0173] Furthermore, for example, this disclosure can take the following configuration: (1) A storage unit that stores a plurality of known traffic scenarios, each composed of a plurality of known traffic scenes in an integrated time series; a processing unit capable of processing the plurality of known traffic scenarios, image data in front of the vehicle, and map data and situational data around the vehicle; and a communication unit capable of transmitting data obtained by the processing unit to an external device and receiving data from the external device, wherein the processing unit is capable of interpreting a traffic context based on the map data and the situational data, thereby generating a plurality of surrounding traffic scenes in a time series, integrating the generated plurality of surrounding traffic scenes in a time series to create a surrounding traffic scenario, and for each of the known traffic scenarios contained in the storage unit, calculating the degree of similarity with the created surrounding traffic scenario, and if the degree of similarity is less than a threshold for any of the known traffic scenarios, using the image data and the surrounding traffic scenario as transmission data to be transmitted to an external device. Information processing device comprising: (2) Information processing device comprising: an acquisition unit capable of acquiring image data in front of a vehicle, an surrounding traffic scenario composed of a plurality of integrated time-series surrounding traffic scenes, and a plurality of net image data included in Internet information; an image similarity matching unit capable of performing image similarity matching on the image data acquired by the acquisition unit and the plurality of net image data acquired by the acquisition unit, acquiring metadata of the net image data with the highest similarity, and generating a new surrounding traffic scenario based on the surrounding traffic scenario acquired by the communication unit and the acquired metadata.(3) The information processing apparatus according to (2), further comprising: a communication unit capable of receiving the image data and the surrounding traffic scenario from the vehicle and transmitting the new surrounding traffic scenario generated by the processing unit to the vehicle as a response to the reception of the image data and the surrounding traffic scenario; and a storage unit for storing the plurality of net image data, wherein the acquisition unit is capable of acquiring the image data and the surrounding traffic scenario via the communication unit and acquiring the plurality of net image data from the storage unit. (4) The information processing apparatus according to (2), wherein the acquisition unit is capable of acquiring the image data from the output of a sensor mounted on the vehicle and acquiring the surrounding traffic scenario based on the output of the sensor and map data stored in the storage unit of the vehicle. (5) An information processing apparatus comprising: an acquisition unit capable of acquiring image data of the front of a vehicle, surrounding traffic data describing the surrounding traffic conditions of the vehicle in numerical and character form, and a plurality of master traffic scenarios, each composed of a plurality of master traffic scenes in an integrated time series; and a processing unit capable of inputting at least the image data from the image data and surrounding traffic data acquired by the acquisition unit to a language processing AI, obtaining a situation interpretation statement of the surrounding traffic conditions from the language processing AI, and generating a new traffic scenario based on the acquired situation interpretation statement and the plurality of master traffic scenarios. (6) The information processing apparatus according to (5), further comprising: a communication unit capable of receiving the image data and surrounding traffic data from the vehicle and transmitting the new surrounding traffic scenario obtained by the processing unit to the vehicle as a response to the reception of the image data and surrounding traffic data; and a storage unit for storing the plurality of master traffic scenarios, wherein the processing unit is capable of acquiring the image data and surrounding traffic data and acquiring the plurality of master traffic scenarios from the storage unit via the communication unit.(7) The processing unit sets a write capacity frame for each of the hazard types according to the frequency of occurrence of each hazard type, and stores the new surrounding traffic scenario in the storage unit based on the level of abstraction of the hazard type, within a range that does not exceed the set write capacity frame for each of the hazard types. The information processing device according to any one of (1), (2), and (4). (8) A vehicle control device comprising: a scenario acquisition unit capable of acquiring the new surrounding traffic scenario from the information processing device according to any one of (1) to (7); and a control unit capable of performing at least one of notification control and driving control based on the new surrounding traffic scenario acquired by the scenario acquisition unit. (9) A vehicle comprising: a notification device and a driving device; and a control unit capable of performing at least one of notification control for the notification device and driving control for the driving device based on the new surrounding traffic scenario acquired from the information processing device according to any one of (1) to (7).

[0174] The control unit 40 shown in Figures 8, 22, 25, and 30, the signal processing unit 230 shown in Figure 18, and the signal processing unit 250 shown in Figure 23 (hereinafter referred to as "control unit 40, etc.") can be implemented by a circuit including at least one semiconductor integrated circuit, such as at least one processor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC) and / or at least one field-programmable gate array (FPGA). At least one processor can be configured to execute all or part of the various functions of the control unit 40, etc. by reading instructions from at least one non-temporary and tangible computer-readable medium. Such a medium can take various forms, including, but is not limited to, various magnetic media such as hard disks, various optical media such as CDs or DVDs, and various semiconductor memories such as volatile memory or non-volatile memory (i.e., semiconductor circuits). Volatile memory may include DRAM and SRAM. Non-volatile memory may include ROM and NVRAM. An ASIC is an integrated circuit (IC) specialized to perform all or part of the various functions in the control unit 40, etc. An FPGA is an integrated circuit designed to be configurable after manufacturing to perform all or part of the various functions in the control unit 40, etc.

Claims

Each unit has a memory unit that stores multiple known traffic scenarios, which are composed of multiple known traffic scenes in an integrated time series. A processing unit capable of processing the aforementioned multiple known traffic scenarios, image data in front of the vehicle, and map data and situational data around the vehicle, A communication unit capable of transmitting data obtained by the processing unit to an external device and receiving data from the external device. Equipped with, The aforementioned processing unit, Based on the map data and the situational data, the traffic context is interpreted, thereby generating multiple time-series surrounding traffic scenes, and the generated time-series surrounding traffic scenes are integrated to create a surrounding traffic scenario. For each of the known traffic scenarios included in the memory unit, the degree of similarity with the created surrounding traffic scenario is calculated. If the similarity is below a threshold for any of the known traffic scenarios, the image data and the surrounding traffic scenario are used as transmission data to be sent to an external device. It is now possible to do so. The communication unit transmits the transmission data to the external device and, in response to this transmission, can receive from the external device a new traffic scenario having a predetermined relationship with the image data and the surrounding traffic scenario. Information processing device. An acquisition unit capable of acquiring image data from the front of the vehicle, a surrounding traffic scenario composed of multiple integrated time-series surrounding traffic scenes, and multiple network image data contained in internet information, A processing unit capable of performing image similarity matching on the image data acquired by the acquisition unit and the plurality of network image data acquired by the acquisition unit, acquiring metadata of the network image data with the highest similarity, and generating a new traffic scenario based on the surrounding traffic scenario acquired by the communication unit and the acquired metadata. Equipped with Information processing device. A communication unit capable of receiving the aforementioned image data and the surrounding traffic scenario from the vehicle, and transmitting the new traffic scenario generated by the processing unit to the vehicle as a response to the reception of the image data and the surrounding traffic scenario, A storage unit that stores the aforementioned plurality of network image data and Furthermore, The acquisition unit is capable of acquiring the image data and the surrounding traffic scenario via the communication unit, and acquiring the multiple network image data from the storage unit. The information processing apparatus according to claim 2. The acquisition unit is capable of acquiring the image data from the output of sensors mounted on the vehicle and acquiring the surrounding traffic scenario based on the sensor output and map data stored in the vehicle's memory unit. The information processing apparatus according to claim 2. An acquisition unit capable of acquiring image data of the area in front of a vehicle, surrounding traffic data describing the surrounding traffic conditions of the vehicle using at least one of numerical and textual information, and multiple master traffic scenarios, each composed of multiple master traffic scenes in an integrated time series. A processing unit capable of inputting at least the image data from the image data and surrounding traffic data acquired by the acquisition unit into a language processing AI, obtaining a situation interpretation statement of the surrounding traffic situation from the language processing AI, and generating a new traffic scenario based on the acquired situation interpretation statement and the plurality of master traffic scenarios. Equipped with Information processing device. A communication unit capable of receiving the aforementioned image data and surrounding traffic data from the vehicle, and transmitting the new traffic scenario obtained by the processing unit to the vehicle as a response to the reception of the image data and surrounding traffic data, A storage unit that stores the aforementioned multiple master traffic scenarios and Furthermore, The processing unit is capable of acquiring the image data and surrounding traffic data via the communication unit, and acquiring the multiple master traffic scenarios from the storage unit. The information processing apparatus according to claim 5. The processing unit sets a write capacity limit for each hazard type according to the frequency of occurrence of each hazard type, and stores the new surrounding traffic scenario in the storage unit based on the level of abstraction of the hazard type, within the limits not exceeding the set write capacity limit for each hazard type. An information processing apparatus according to any one of claims 1, 2, and 4. A scenario acquisition unit capable of acquiring the new traffic scenario from the information processing device described in any one of claims 1 to 6, A control unit capable of performing at least one of notification control and driving control based on the new traffic scenario acquired by the scenario acquisition unit. Equipped with Vehicle control system. Notification device and travel device, A control unit capable of performing at least one of the following: notification control for the notification device and driving control for the driving device, based on the new traffic scenario obtained from the information processing device according to any one of claims 1 to 6. Equipped with vehicle.

Images