Device for vehicle and program

Abstract

Provided is a vehicle control device including a processor and a communication circuit. The processor is a module that performs processing related to detection of another vehicle on the basis of data input from at least one sensor mounted in an own vehicle. The communication circuit is an interface for the processor to communicate with a camera that captures an image of an area forward of the vehicle. The processor acquires an image related to a merging lane from the camera using the communication circuit when the vehicle is traveling on the main lane of a motorway. Then the processor detects merging vehicles on the basis of the luminance feature of the road element related to the merging lane included in the acquired image. For example, the processor estimates the presence of a hidden merging vehicle on the basis of the road element of the merging lane having a feature (for example, high luminance) indicating illumination with a headlight.

Description

Vehicle apparatus and program Cross-reference to Related Applications

[0001] This application is based on Japanese Patent Application No. 2025-45858 filed on March 19, 2025, and the entire content of the base application is incorporated herein by reference.

[0002] The present disclosure relates to a technique for detecting a merging vehicle hidden by a shield such as a wall.

[0003] Patent Document 1 discloses a technique for changing the stopping position of the host vehicle in accordance with the road structure when merging from a merging lane to the main lane by changing lanes is impossible. When a sufficiently large off-road space (a so-called road shoulder) is provided alongside the main lane in a road section ahead of the disappearance point of the merging lane, the apparatus disclosed in Patent Document 1 adopts the off-road space as the stopping position. Further, the apparatus stops the vehicle within the merging lane when a sufficiently large off-road space is not provided.

[0004] Japanese Unexamined Patent Publication No. 2020-104634

[0005] The technique of the aforementioned Patent Document 1 is a technique used for a vehicle moving from a merging lane to the main lane, and is not directed to a vehicle traveling on the main lane. For vehicles traveling on the main lane (hereinafter also referred to as main lane vehicles), a technique for detecting a merging vehicle and performing control in accordance with the position of the detected merging vehicle or the like is required. In such a technique for main lane vehicles, it is desirable to be able to detect a merging vehicle as early as possible. For example, it is preferable that an apparatus used in a main lane vehicle is configured to be capable of detecting a merging vehicle traveling in a merging preparation section.

[0006] The merging preparation section herein is a road section located before the merging section. The merging section is a section where a merging vehicle can actually move into the main lane. The merging section may be understood as, for example, a section where the lane boundary is a white broken line. Generally, between the merging preparation section and the main lane, poles, three-dimensional medians, blocks, continuous yellow or white lines, or the like are provided.

[0007] However, merging preparation sections may have obstacles such as fences or hedges. When obstacles are present, the devices used by vehicles on the main line have difficulty detecting potential merging vehicles traveling through the merging preparation section. Obstacles here can be understood as three-dimensional objects that obstruct the driver's or sensor's field of view, for example, objects that are 1 meter or more above the road surface.

[0008] One of the purposes of this disclosure is to provide a technology for detecting merging vehicles at an earlier stage.

[0009] A vehicle device disclosed herein comprises a processing unit that performs processing related to the detection of other vehicles based on data input from one or more sensors mounted on the vehicle, and a communication circuit for the processing unit to communicate with one or more sensors, wherein one or more sensors include a camera that photographs the front of the vehicle, and the processing unit is configured to acquire images related to merging lanes from the camera using the communication circuit while the vehicle is traveling on the main lane of an expressway, and to detect merging vehicles based on the brightness characteristics of road elements related to merging lanes included in the acquired images.

[0010] Generally, merging vehicles hidden by obstacles cannot be detected by sensors. However, if a merging vehicle is present and its headlights are on, the road elements related to the merging lane will be illuminated by the light from those headlights and may have high brightness in the image. In other words, the image of the road elements related to the merging lane can indirectly function as a basis for determining whether or not a merging vehicle is present. The above configuration was created based on the above idea and detects merging vehicles based on the brightness characteristics of the image of road elements related to the merging lane included in the forward image captured by the camera. This makes it possible to detect the presence of a merging vehicle even if the actual vehicle is not visible. In other words, it becomes possible to detect merging vehicles at an earlier stage.

[0011] The program included in this disclosure is a program that causes a computer configured to receive video from a camera that takes pictures of the front of a vehicle to perform the following actions when the vehicle is traveling on the main lane of an expressway: receive images related to the merging lane from the camera via a communication circuit, and determine whether or not a merging vehicle is present based on the brightness characteristics of road elements related to the merging lane included in the received images.

[0012] The symbols in parentheses in the claims indicate a correspondence with the specific means described later in the embodiments, and do not limit the technical scope of this disclosure.

[0013] This is a block diagram illustrating an example of a vehicle system. This is a block diagram illustrating the function of a vehicle control device. This is a diagram illustrating terminology related to merging sections. This is a diagram illustrating the reduction of FOV on the merging lane due to obstructions. This is a diagram illustrating a method for detecting merging vehicles hidden by obstructions. This is a diagram illustrating an example of an image area used for image analysis to detect merging vehicles hidden by obstructions. This is a flowchart illustrating the operation of the processor related to passing through a merging section. This is a diagram illustrating a continuation of the flowchart shown in Figure 7. This is a diagram illustrating pre-emptive evasive control. This is a diagram illustrating the effect of lateral position adjustment. This is a diagram illustrating one form of light control when performing indirect search processing.

[0014] Embodiments of this disclosure will be described below with reference to the drawings. This disclosure is not limited to the embodiments described below. The configurations disclosed below may be implemented with various modifications without departing from the gist of the invention. Various modifications may be combined as appropriate, without causing any technical inconsistencies. This disclosure also includes configurations that are not explicitly stated, which are combinations of multiple modifications. In the following description, components having the same function may be denoted by the same reference numeral and their specific description may be omitted. Also, components having the same function may be denoted by the same or similar names and their specific description may be omitted. If only a part of a configuration is referred to, the description of the other parts may be applied elsewhere.

[0015] <Introduction> Figure 1 is a diagram showing an example of the schematic configuration of the vehicle system VS related to this disclosure. The vehicle system VS is a system installed in a vehicle Hv. Hereafter, the vehicle Hv on which the vehicle system VS is installed may also be referred to as the vehicle itself. In this disclosure, the term "vehicle lane" refers to the lane in which the vehicle is traveling, among the multiple lanes provided on the road. The vehicle lane can also be called the ego lane. An adjacent lane is a lane adjacent to the vehicle lane.

[0016] In this disclosure, "leading vehicle" basically means the vehicle that is in front of vehicle Hv, traveling in the same lane as vehicle Hv, and is the closest vehicle to vehicle Hv. "Following vehicle" means another vehicle traveling behind vehicle Hv in the same lane. The term "forward vehicle" broadly means another vehicle traveling in front of the vehicle. "Forward vehicle" includes not only the leading vehicle but also vehicles traveling diagonally in front. Similarly, "following vehicle" includes not only the following vehicle but also vehicles traveling diagonally behind vehicle Hv.

[0017] The term "merging vehicle" refers to a vehicle traveling on a merging lane. A merging lane is a lane that merges with the main lane of an expressway. An expressway, in this context, is a road where pedestrians and bicycles are prohibited from entering, such as toll roads like expressways. A merging lane can generally be understood as a lane that disappears after the merging section.

[0018] When vehicle Hv is traveling on the main lane of an expressway, merging vehicles are classified into competing vehicles and non-competing vehicles based on their position relative to vehicle Hv. A competing vehicle is any other vehicle traveling in the merging lane that could potentially influence the behavior of vehicle Hv. A vehicle that could potentially influence the behavior of vehicle Hv is any other vehicle that is positioned in front of or behind vehicle Hv in the merging section and traveling at a speed that allows it to do so. From another perspective, a vehicle that could potentially influence the behavior of vehicle Hv is any other vehicle that could potentially cause a collision or an excessively close approach (near-crash). A competing vehicle may be a vehicle positioned to the side of vehicle Hv on the merging lane. A non-competing vehicle is any other vehicle traveling in the merging lane that does not influence the behavior of vehicle Hv. In other words, a non-competing vehicle is a merging vehicle that is positioned and traveling at a speed that does not create a conflict of right-of-way with vehicle Hv. For example, another merging vehicle traveling more than a certain distance ahead of or behind a competing vehicle may be a non-competing vehicle.

[0019] In this disclosure, "driver" means a person seated in the driver's seat, i.e., a driver's seat occupant, regardless of whether they are actually performing driving operations. In one scenario, the driver may be understood as the person who should receive the authority and responsibility for driving operations from the vehicle system VS at the end of autonomous driving. The term "driver" may be replaced with "driver's seat occupant" or "vehicle user." The vehicle Hv may be a remotely operated vehicle that is remotely controlled by an operator located outside the vehicle. The person who takes over driving operations from the vehicle system VS may be an operator located outside the vehicle. The operator is a person who has the authority to control the vehicle remotely from outside the vehicle. The operator may also be included in the concept of a driver.

[0020] Vehicle Hv may be a vehicle equipped with so-called autonomous driving functions. Autonomous driving functions are functions that allow the vehicle to autonomously drive along a predetermined route. The degree of automation of driving operations (hereinafter referred to as the automation level) can be divided into multiple levels, as defined by the Society of Automotive Engineers (SAE International). The automation level can be divided into six stages, for example, levels 0 to 5. In the explanation of the automation level, the system mainly refers to the vehicle control device 30.

[0021] Level 0 is equivalent to manual driving, where the driver performs all driving tasks without system intervention. Driving tasks include steering, acceleration / deceleration, and surrounding area monitoring. Level 1 is a level where the system supports either steering or acceleration / deceleration. Level 2 is a level where the system performs subtasks of both longitudinal and lateral vehicle motion control in a limited area. Level 2 may also include system-performed speed control and steering control.

[0022] Level 3 means that the system performs all driving tasks within the Operational Design Domain (ODD), but in emergencies, control is transferred from the system to the driver. The ODD defines the conditions under which autonomous driving is possible. Level 4 is the level in which the system performs all driving tasks except in specific situations such as certain roads that are unsuitable or extreme environments. Level 5 is the level in which the system performs all driving tasks in all environments.

[0023] Automation levels 3 through 5 are categorized as autonomous driving. The longitudinal and lateral control of dynamic driving tasks by the system at automation level 2 and above will also be referred to as autonomous driving control below. Autonomous driving control includes autonomous driving.

[0024] The vehicle Hv (in other words, the vehicle system VS) described below is equipped with, as an example, Level 3 equivalent autonomous driving functionality. Of course, the vehicle Hv may also be equipped with vehicle control functions up to Level 2. The vehicle Hv may also be equipped with Level 4 or higher autonomous driving functionality. The configuration of the vehicle system VS disclosed below may be modified as appropriate to conform to the laws and customs of the region in which the vehicle system VS is used, the characteristics of the vehicle on which it is installed / the installed equipment, etc.

[0025] <Overall Configuration of Vehicle System VS> The vehicle system VS includes, as an example, multiple devices as shown in Figure 1. Specifically, the vehicle system VS includes an environmental sensor 11, a vehicle state sensor 12, a locator 13, a map storage unit 14, a wireless communication device 15, an occupant state sensor 16, a display 17, and a speaker 18. The vehicle system VS also includes an operating member 21, a motion actuator 22, an auxiliary actuator 23, and a vehicle control device 30. The term "device" may include sensors and circuits. Some devices may be configured as subsystems; that is, devices may be subsystems.

[0026] The vehicle control device 30 is interconnected with some of the above-mentioned devices via the in-vehicle network 10, enabling mutual communication. The in-vehicle network 10 is a communication network established within the vehicle. Some devices may be directly connected to the vehicle control device 30 by cables (e.g., wire harnesses). Figure 1 illustrates a case where the operating member 21, motion actuator 22, and auxiliary actuator 23 are connected to the vehicle control device 30 by cables, while other devices are connected to the vehicle control device 30 via the in-vehicle network. The connection configurations between devices may be modified as appropriate.

[0027] The environmental sensor 11 is a device that senses the surrounding environment of the vehicle Hv. The environmental sensor 11 detects targets present in the external environment of the vehicle Hv. The environmental sensor 11 includes multiple sensors. The environmental sensor 11 includes multiple cameras. Cameras may be referred to as image sensors. The multiple cameras include a forward camera 111 that photographs the area in front of the vehicle. There may be only one forward camera 111, but here we assume that there are multiple. At least one of the multiple forward cameras 111 is a visible light camera. One of the multiple forward cameras 111 may be a Time of Flight (ToF) camera or an infrared camera. A ToF camera is a camera that generates distance images.

[0028] In addition to the camera, the environmental sensor 11 may include at least one, preferably more than one, of LiDAR, radar, ultrasonic sonar, and acoustic sensors. LiDAR stands for Light Detection and Ranging or Laser Imaging Detection and Ranging. Multiple types of environmental sensors 24 may be combined and implemented on the vehicle Hv to monitor the front, right, left, and rear directions of the vehicle Hv.

[0029] The objects detected by the environmental sensor 11 may include both dynamic and static objects. Dynamic objects refer to so-called moving objects such as other vehicles, pedestrians, and cyclists. The terms dynamic objects, moving objects, road users, and traffic participants may be interchangeable. Static objects are features such as road edges, road markings, and structures installed along the road. Road markings may include at least one of lane marks indicating lane boundaries, pedestrian crossings, stop lines, traffic guides, and regulatory arrows. Structures installed along the road may include at least one of road signs, guardrails, traffic lights, utility poles, curbs, walls, fences, and commercial signs. Trees may also be detected as static objects.

[0030] The front camera 111 may also be configured to detect the illumination status of other vehicles' lighting devices, such as hazard lights, brake lights, and turn signals. The front camera 111 may also be configured to detect the illumination status of traffic lights. The environmental sensor 11, including the front camera 111, inputs a data signal indicating the detection result to the vehicle control device 30. The output signal of the environmental sensor 11 may be any signal that allows the vehicle control device 30 to identify the driving environment, and may be a data signal before object recognition processing, such as a video signal. The object recognition function may be provided in the vehicle control device 30 instead of the sensor.

[0031] The vehicle state sensor 12 is a sensor that detects the vehicle state. The vehicle state here may be the internal state of the vehicle Hv. The vehicle state sensor 12 includes multiple types of sensors in the vehicle Hv. The vehicle state sensor 12 may include at least one, more preferably more, of the following: an acceleration sensor, a jerk sensor, a vehicle speed sensor, and a yaw rate sensor. The vehicle state sensor 12 may also include at least one, more preferably more, sensors that detect the operation state of the control devices by the driver, such as a steering angle sensor, a brake sensor, and an accelerator sensor. The accelerator sensor is a sensor that detects the amount / force of depression of the accelerator pedal. The brake sensor is a sensor that detects the amount / force of depression of the brake pedal. The vehicle state sensor 12 inputs a data signal indicating the detection result to the vehicle control device 30 via the in-vehicle network 10.

[0032] The locator 13 is a device that generates and outputs position information of the vehicle Hv (more precisely, the locator 13) using navigation signals transmitted from positioning satellites that constitute the GNSS (Global Navigation Satellite System). The locator 13 may include an inertial sensor in addition to the GNSS receiver. The locator 13 may determine the position and direction of travel of the vehicle Hv by combining the navigation signals received by the GNSS receiver, the measurement results of the inertial sensor, and the vehicle speed information flowing through the in-vehicle network 10. In this disclosure, the data indicating the position of the vehicle Hv generated and output by the locator 13 is also referred to as the vehicle position data. The vehicle position data may also be understood as an example of sensor data. The locator 13 outputs the vehicle position data to the vehicle control device 30.

[0033] The map storage unit 14 is a storage device in which map data is stored. The map data held by the map storage unit 14 may be so-called HD (High Definition) map data. The map data stored in the map storage unit 14 includes data such as the three-dimensional shape of roads, the installation locations of road markings (e.g., lane marks), and the installation locations of traffic signs, with the accuracy necessary for autonomous driving. The map storage unit 14 may also store a navigation map that shows the road connections. The navigation map may be used to search for a route from the current location to the destination. The map data stored in the map storage unit 14 may be updated by data received by the wireless communication device 15 from a map server or the like. The map storage unit 14 may also be a storage device for temporarily holding map data received by the wireless communication device 15 from the map server until the data expires. The map storage unit 14 may be built into the vehicle control device 30.

[0034] The wireless communication device 15 is a device for vehicle Hv to communicate wirelessly with an external device. The external device may include at least one of the following: another vehicle, a server, a traffic information center, a roadside unit, and a portable device. The portable device may be, for example, a smartphone. The wireless communication device 15 is configured to perform cellular communication. Cellular communication refers to wireless communication compliant with LTE (Long Term Evolution), 4G, or 5G, etc.

[0035] The occupant status sensor 16 is a sensor that detects the driver's status. The occupant status sensor 16 may be, for example, a driver status monitor (hereinafter referred to as DSM). The DSM is a sensor that detects the driver's face orientation, eye position, gaze direction, and degree of eyelid opening, etc., based on the driver's face image. Eye position is the position of the eyes and may be expressed as height from the road surface, etc. The occupant status sensor 16 transmits driver status data indicating the driver's face orientation, eye position, gaze direction, degree of eyelid opening, etc., to the vehicle control device 30. Driver status data may also be understood as a type of sensor data.

[0036] The display 17 is a device that displays images. Images may be used as a substitute for images. The display 17 may be a liquid crystal display or an organic EL display. The display 17 is located on the instrument panel. The display 17 may also be a so-called head-up display that displays images by projecting image light onto a predetermined area of ​​the windshield. The display 17 displays an image corresponding to a signal input from the vehicle control device 30.

[0037] The speaker 18 is a device that outputs sound corresponding to a signal input from the vehicle control device 30. The term "sound" in this disclosure includes notification sounds, voices, music, etc. The display 17 and speaker 18 are devices for informing the driver of information (hereinafter also referred to as notification devices). The vehicle system VS may also include a vibrator, ambient light, etc., as notification devices.

[0038] The operating member 21 is a component for the driver to input instructions to the vehicle system VS. The operating member 21 includes components for acceleration, deceleration, and steering, namely the accelerator pedal, brake pedal, steering wheel, and shift lever. The operating member 21 may also include a turn signal switch, a light switch, and a mode switch.

[0039] The turn signal switch is a switch for illuminating the turn signal lamps as direction indicators. The turn signal switch also functions as an approval switch for the vehicle Hv user (e.g., driver) to approve a lane change. The light switch is a switch for switching the illumination state of the headlights, i.e., automatic, off, on, high beam, and low beam. The mode switch is a switch for switching the operation mode (in other words, the automation level), which will be described later. Some of the switches may be switches provided on the spokes of the steering wheel (so-called steering switches). In addition, the operating member 21 may include a touch panel stacked on the display 17.

[0040] The operating member 21 outputs an operating signal, which is an electrical signal corresponding to the driver's operation, to the vehicle control device 30. The operating signal includes information indicating the driver's operation. Some of the operating members 21 may be connected to the vehicle control device 30 via the in-vehicle network 10 or other devices.

[0041] The motion actuator 22 is an actuator that generates power corresponding to acceleration, deceleration, or steering of the vehicle Hv. The term actuator may be replaced with equipment, system, subsystem, or mechanism. The motion actuator 22 includes a drive mechanism related to the propulsion of the vehicle. The drive mechanism may be a powertrain and include at least one of an engine and a drive motor. In other words, the drive mechanism may be an engine, an EV system, or a hybrid system. The motion actuator 22 also includes a brake actuator as a braking mechanism and a steering actuator as a steering mechanism. The brake actuator may include at least one of a hydraulic brake and an electric regenerative brake. The steering actuator may be an EPS (Electric Power Steering) motor.

[0042] The motion actuator 22 operates based on control signals input from the vehicle control device 30 and controls the motion of the vehicle Hv. Other ECUs, such as a steering ECU for steering control, a power unit control ECU for controlling the drive source, or a brake ECU, may be interposed between the vehicle control device 30 and the motion actuator 22. Since the motion actuator 22 is an actuator that directly affects the movement of the vehicle Hv, it is also referred to as a primary actuator in this disclosure.

[0043] The auxiliary actuator 23 is an on-vehicle facility that does not directly affect the movement of the vehicle Hv, but enables safe and legal driving. The auxiliary actuator 23 includes a plurality of lighting devices, a horn, a defogging device for a windshield / camera, a cleaning device for the environmental sensor 11, a wiper motor for a windshield, a wiper motor for a rear glass, and the like. The lighting devices include a headlight, a fog lamp, a brake lamp, a turn signal lamp, a clearance lamp, a hazard lamp, a back-up lamp, and the like.

[0044] The vehicle control device 30 is a device that controls the motion actuator 22 and the auxiliary actuator 23 based on the detection result of the environmental sensor 11, thereby executing part or all of driving operations instead of a driver. The vehicle control device 30 may be automated driving systems (ADS). The vehicle control device 30 may be implemented in the form of a device (that is, an automatic driving device). The descriptions of "system" and "device" may be mutually replaced as appropriate.

[0045] The vehicle control device 30 may be implemented using one or more computers. The one or more computers that constitute the vehicle control device 30 correspond to the vehicle control device. The vehicle control device 30 includes a processor 31, a memory 32, a storage 33, a communication circuit 34, and a bus connecting these components, etc. The processor 31 may be a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The vehicle control device 30 may include a plurality of processors 31, and one of the plurality of processors 31 may be an arithmetic core optimized for processing using a neural network (a so-called NPU: Neural Processing Unit). The processor 31 corresponds to a processing unit.

[0046] The memory 32 is a rewritable volatile storage medium. The memory 32 is, for example, a RAM (Random Access Memory). The storage 33 is a rewritable non-volatile storage medium such as a flash memory. A vehicle control program, which is a program executed by the processor 31, is stored in the storage 33. Execution of the vehicle control program by the processor 31 corresponds to execution of a vehicle control method.

[0047] The communication circuit 34 is hardware for the processor 31 to communicate with other devices constituting the vehicle system VS, such as the environmental sensor 11 and the like. The communication circuit 34 may include a circuit adapted to a communication method with other devices. The communication circuit 34 may be an input / output circuit or an input / output port. The communication circuit 34 corresponds to a communication circuit. The communication circuit 34 may be compatible with any type of wired communication or wireless communication. Part or all of the wireless communication device 15 may be included in the communication circuit 34. Digital data corresponding to a signal received by the communication circuit 34 may be temporarily stored in the memory 32. The data stored in the memory 32 is appropriately referenced by the processor 31.

[0048] The communication circuit 34 receives information necessary for implementing vehicle control such as automatic driving or driving support. Receiving may also be rephrased as acquiring. The communication circuit 34 acquires sensor data (that is, detection results) from the environmental sensor 11. The sensor data includes data about objects existing around the host vehicle, such as moving objects, features, and obstacles. Data of a detected object may include the position, moving speed, type, and size of the detected object.

[0049] Sensor data related to features may include lane mark data and road edge data. The lane mark data may include not only position data but also line type data. The line type can be expressed as a continuous line (solid line) or a broken line. The communication circuit 34 may include camera images in the sensor data. The communication circuit 34 also receives image data of the forward camera 111.

[0050] Furthermore, the communication circuit 34 acquires sensor data related to the vehicle's state, such as the vehicle Hv's speed, acceleration, yaw rate, and external illumination, from the vehicle state sensor 12. In addition, the communication circuit 34 acquires its own vehicle position data from the locator 13. The communication circuit 34 acquires map data of the road section related to the vehicle Hv's travel by referring to the map storage unit 14. The communication circuit 34 may acquire data transmitted from external devices in cooperation with the wireless communication device 15. For example, the communication circuit 34 may acquire vehicle data transmitted from a vehicle ahead via vehicle-to-vehicle communication. The vehicle data may be a dataset including speed and the operating status of brakes and turn signals.

[0051] The communication circuit 34 also acquires information indicating the driver's operation of the vehicle system VS based on the signal from the operating member 21. The processor 31 accepts brake and accelerator operations and driving operations based on the signal from the operating member 21 received via the communication circuit 34. The processor 31 also accepts instructions regarding changes in the operating mode via the operating member 21. When the operating mode is the Level 2 mode described later, the vehicle control device 30 may acquire the driver's instructions regarding acceptance or rejection of automatic lane changes from the operating member 21 (e.g., a turn signal switch) via the operating member 21. The vehicle control device 30 may also be configured to acquire various driver instructions by voice recognition. A device related to voice input, such as a microphone, can also be included in the operating member 21.

[0052] The communication circuit 34 may acquire information indicating the operating status of devices connected to the vehicle control device 30, such as whether the environmental sensor 11 is functioning normally. The communication circuit 34 acquires driver status data indicating eye position and gaze direction from the occupant status sensor 16.

[0053] The various data acquired sequentially by the communication circuit 34 are stored in a temporary storage medium such as memory 32 and used by the environment recognition unit F1, mode management unit F2, etc. Data acquired after a certain period of time may be discarded. Various data (also called information) may be acquired by generation, conversion, determination, or calculation based on signals received from other devices.

[0054] The communication circuit 34 or processor 31 may have a function to generate other data (also referred to as secondary data) based on raw data (also referred to as secondary data) received from other devices. For example, the processor 31 obtains luminance features of various road elements related to the merging section by analyzing the video data captured by the front camera 111. The luminance feature of a single road element may be the average or maximum value of the luminance values ​​of multiple pixels corresponding to that road element. The luminance feature of a road element may be a feature quantity derived from the luminance information of the image region representing the road element. The luminance feature of a road element may be the average, median, or maximum value of the luminance value for each color element. The luminance feature of the road surface may be the amount of change in luminance value according to distance. The luminance feature may also be the difference in luminance relative to a reference image. The reference image may be an image of a road element that is not illuminated by headlights at night. The reference images may be prepared in advance for each road element or each merging section and stored in the storage 33. The reference images for each road element or each merging section may be used in the indirect search process described later.

[0055] The processor 31 of this embodiment detects potential merging vehicles based on the brightness characteristics of road elements related to the merging section captured by the forward camera 111. A potential merging vehicle is a merging vehicle whose actual form is hidden by a wall or the like. A potential merging vehicle may be, for example, another vehicle traveling in an acceleration section separated from the main road by an obstruction. The obstruction may be a wall, fence, or planted area. The obstruction may be a three-dimensional object with a height of 1 meter or more relative to the road surface. A specific method for detecting potential merging vehicles will be described separately.

[0056] <Operating Modes of the Vehicle Control Device> The vehicle control device 30 has multiple operating modes with different levels of automation. Each operating mode has a different range of driving tasks that the driver is responsible for, in other words, a different range of driving tasks in which the vehicle control device 30 intervenes. The term "operating mode" can be rephrased as "driving mode." Here, as an example, the vehicle control device 30 is configured to be switchable between multiple operating modes, including manual driving mode, level 2 mode, and level 3 mode.

[0057] The manual driving mode is an operating mode in which the driver performs all driving tasks. The Level 2 mode is an operating mode corresponding to automation level 2.0, in which the vehicle control device 30 substantially controls the behavior of the vehicle Hv. Here, the Level 2 mode may be an operating mode in which the driver's hands-off is permitted, or it may be an operating mode in which the driver's hands-on is required. In this disclosure, hands-on means holding the steering wheel. Hands-off means taking one's hands off the steering wheel. In Level 2 mode, the driver's eyes-on is required. Eyes-on means monitoring the area outside the vehicle (primarily forward) related to the direction of movement of the vehicle Hv.

[0058] Level 3 mode is an operating mode that performs vehicle control equivalent to automation level 3. Level 3 mode is an operating mode in which eyes-off is permitted to the extent that the driver can quickly return to driving operation in response to a request from the system. Eyes-off means the act of taking one's eyes off the area outside the vehicle that is related to the direction of movement of the vehicle Hv. In this disclosure, unless otherwise noted (basically), the term automated driving means automated driving control of level 3 or higher. Hereafter, the term automated / autonomous driving may be abbreviated as AD. In other embodiments, the vehicle control device 30 may have an operating mode equivalent to level 4 or 5. In this disclosure, operating modes of level 3 or higher may also be referred to as automated driving mode, i.e., AD mode.

[0059] While in AD mode, the processor 31 automatically performs steering, acceleration, deceleration (in other words, braking) of the vehicle so that the vehicle Hv travels along the planned route set by the driver toward the destination. Even if no destination is set, the vehicle control device 30 may select a route to continue driving / circling within the range that satisfies the ODD and continue autonomous driving. The conditions for determining whether autonomous driving is possible or not, in other words, the detailed conditions that define the ODD, can be changed as appropriate. AD mode is terminated due to steering / pedal operation by the driver (so-called override), system limits, exit from the ODD, etc. The vehicle control device 30 may have a function to determine whether or not the vehicle Hv is within the ODD.

[0060] The processor 31 performs control to enable the vehicle to operate autonomously, even while in Level 2 mode. That is, the vehicle control device 30 performs recognition of the driving environment, planning of the driving trajectory, and motion control, even while in Level 2 mode. Motion control includes speed adjustment through acceleration and deceleration, and steering control. In order to promptly initiate autonomous driving or driver assistance in response to a request from the driver, the vehicle control device 30 may continue to perform recognition processing of the driving environment in the background (in other words, potentially) even in manual driving mode.

[0061] <Functions of the Vehicle Control Device> The vehicle control device 30 includes the functional units shown in Figure 2, which are realized by executing an automatic driving program. Specifically, the vehicle control device 30 has an environment recognition unit F1, a mode management unit F2, a planning unit F3, a vehicle control unit F4, and a vehicle control unit F4.

[0062] The environmental recognition unit F1 recognizes the driving environment of the vehicle Hv based on sensor data and map data acquired by the communication circuit 34. The sensor data may include at least one of the detection results of the environmental sensor 11 and the data received by the wireless communication device 15. The environmental recognition unit F1 may also recognize the driving environment of the vehicle Hv by performing sensor fusion processing that integrates the detection results of multiple environmental sensors 11.

[0063] The driving environment may include at least one of weather and road surface conditions. The weather and road surface conditions may be determined by combining the recognition results of the front camera 111 with weather information acquired by the communication circuit 34. The driving environment includes the positions and types of objects present around the vehicle Hv. The environment recognition unit F1 may acquire the speed and direction of movement of detected moving objects. Based on the various data acquired by the communication circuit 34, the environment recognition unit F1 recognizes the position and behavior of other vehicles.

[0064] The environmental recognition unit F1 acquires information regarding the structure (in other words, configuration) of the road within a predetermined distance in front of the vehicle Hv, based on at least one of sensor data and map data. The road structure may include the location of three-dimensional objects, the number of lanes, the location of lane markers, the location of the road edge, the road width, and the curvature of the road. The location of three-dimensional objects refers to the locations of walls, poles, blocks, guardrails, traffic signs, billboards, etc. Road markings may also be included in the road structure. The location of merging sections is also included in the road structure.

[0065] The environmental recognition unit F1 identifies the attributes (also called classifications) of the road on which the vehicle Hv is traveling, based on sensor data or map data acquired by the communication circuit 34. For example, the environmental recognition unit F1 acquires whether the road is a general road, a main line of an expressway, or an auxiliary road. The road is the road on which the vehicle Hv is traveling. An auxiliary road refers to roads connecting general roads and main lines, roads connecting two expressways, roads connecting main lines and parking areas, etc. An auxiliary road also includes merging roads that lead to the main line. The driving environment may include the attributes of these roads.

[0066] Furthermore, the driving environment may include the vehicle's lane number. The vehicle's lane number is a number indicating the position of the vehicle's lane on the road, and is determined relative to the left edge of the road. The vehicle's lane number directly or indirectly represents the number of lanes to the left or right of the vehicle's lane. The environment recognition unit F1 may identify the vehicle's lane number using the distance from the road edge to the vehicle Hv, the number of lane marks detected on the left and right, and at least one of the map data. The vehicle's lane number may also be identified using map data and vehicle position data.

[0067] The environmental recognition unit F1 also acquires information about the merging section that vehicle Hv is scheduled to pass through next, if vehicle Hv is traveling on the main road. The information about the merging section may be acquired based on map data or identified based on sensor data from the environmental sensor 11. The information about the merging section may include at least one of the merging start point, the merging end point, and the start point of the warning section. The merging start point is the start point of the merging section. The merging end point is the end point of the merging section. The information about the merging section may include information about the lane being merged into, which is a lane that flows in the same direction as the merging lane. The information about the lane being merged indicates whether the lane being merged into is the rightmost lane or the leftmost lane. The software / hardware module responsible for processing the recognition of the merging section corresponds to the merging section recognition unit F11.

[0068] Figure 3 shows an overview of a merging section. The example shown in Figure 3 has a road structure in which a merging lane connects from the left to a main road with two lanes. VL1 refers to the first lane of the main road, and VL2 refers to the second lane. VL0 is the merging lane. In Figure 3, the left side corresponds to the merging direction, and the first lane VL1 corresponds to the lane being merged into. The merging direction is the direction in which the merging lane exists when viewed from the main road, in other words, the direction from which a merging vehicle may be approaching when viewed from vehicle Hv. Hereafter, the direction opposite to the merging direction will be referred to as the evacuating direction. The lane being merged into is the lane of the main road that is directly connected to the merging lane. Arrow A3 in Figure 3 indicates the merging direction. Arrow A4 indicates the evacuating direction.

[0069] Furthermore, in this disclosure, among the lanes provided on the main line, the lane adjacent to the merging lane in the direction of evacuating is also referred to as an evacuating lane. An evacuating lane may also be called a non-merging lane. An evacuating lane is not directly connected to a merging lane and is not affected by, or is less affected by, merging vehicles.

[0070] In the diagram, 301 represents a lane mark indicating the left boundary of the first lane VL1. 302 represents a lane mark indicating the right boundary of the first lane VL1. The lane mark indicated by 302 is also the left boundary of the second lane VL2. 303 represents a lane mark indicating the right boundary of the second lane VL2. 304 is the median strip, and there may be an oncoming lane to its right (not shown). 311 represents a lane mark indicating the right boundary of the merging lane. 312 represents a lane mark indicating the left boundary of the merging lane. The lane marks for merging lanes exemplified by 311 and 312 will also be referred to as merging lane marks below. 313 represents a structure located further in the merging direction of the merging lane (hereinafter referred to as the "rear structure"). The rear structure 313 may be a fence or wall provided along the merging lane.

[0071] 305 represents a transition restrictor. Transition restrictor 305 is a three-dimensional object less than 1 meter in height that does not obstruct the driver's view. Transition restrictor 305 may be, for example, a pole, a block, or a plant. Transition restrictor 305 may be configured to physically restrict merging vehicles from moving from the merging lane VL0 to the main lane, while still allowing drivers of vehicles traveling on the main lane to recognize the presence of merging vehicles.

[0072] 306 is a barrier. The barrier 306 is a three-dimensional structure with a height of 2m or more. The barrier 306 is, for example, a wall or a fence. The barrier 306 may be partially provided along the main line for purposes such as noise reduction in residential areas, preventing vehicles from veering off the road, prohibiting pedestrians or animals from entering, or other purposes. The barrier 306 is not provided in the merging section, but may be partially formed along the warning section. The barrier 306 provided along the warning section completely or largely obstructs the view from drivers of vehicles traveling on the main line to the merging lane VL0.

[0073] The merging section 101 is a section of road where merging vehicles can enter the main road from the merging lane VL0. The merging section 101 may be a section where, for example, the boundary line 310 between the merging lane VL0 and the first lane VL1 has lane markings in a pattern that allows for vehicle transitions (for example, a dashed line pattern). The merging start point 102 is a point where merging vehicles can move, both physically and from the standpoint of regulatory markings. The merging start point 102 may be a point where there are no three-dimensional objects separating the merging lane VL0 and the first lane VL1, and where the prohibition of transition / crossing by road markings is lifted. In one phase, the merging start point 102 may be understood as the point where the lane marking pattern on the left side of the first lane VL1 switches from a continuous line to a dashed line. The merging start point may also be referred to as the merging start line. The confluence starting point 102 corresponds to the end point of the caution section 104, which will be described next.

[0074] The merging end point 103 is the point where the merging lane VL0 itself disappears. The merging end point 103 may be the point in a single phase where the left boundary line of the first lane VL1 changes from a dashed line to a continuous line. The merging end point 103 may also be the point where the width of the merging lane defined by lane marks 310 and 311 becomes less than a predetermined value (for example, 1.5 m). The merging end point 103 may also be referred to as the merging end line. The road section from the merging start point 102 to the merging end point 103 corresponds to the merging section 101.

[0075] The warning section 104 is a section of road located before the merging section on the main line (especially the lane being merged). The warning section 104 may be a section of road on the main line where the remaining distance to the merging start point 102 is less than a predetermined value. The predetermined value may be, for example, 200m, 400m, or 600m. This predetermined value can define the warning start point 105, which is the starting point of the warning section 104. The warning start point 105 may be variable depending on the speed, and may be defined, for example, by the remaining time until vehicle Hv reaches the merging start point 102 (hereinafter referred to as the remaining time before merging). The warning start point 105 may be the point where the remaining time before merging is a predetermined value (for example, 8 seconds or 10 seconds). The remaining time before merging can be calculated based on the predicted time of arrival at the merging start point. The warning section is a section of road on the main line side that is before the merging section, and may be called a preceding section or a main line preparation section.

[0076] In this disclosure, the road section 201 located before the merging start point in the merging lane VL0 is also referred to as the preparation section. The caution section 104 described earlier refers to a predetermined section on the main road side, whereas the preparation section 201 differs in that it refers to a specific section on the merging lane side. In addition, in this disclosure, the section 202 on the direction of travel side of the merging start point in the merging lane VL0 is referred to as the transition section.

[0077] The preparation section 201 may be, for example, a road section where the remaining distance to the merging start point is less than a predetermined value. The preparation section may be a section adjacent to the caution section. The preparation section may also be an acceleration section in one phase. The preparation section is located in the reverse direction from the merging section. Here, the reverse direction means the direction opposite to the direction of travel set on the road. The reverse direction corresponds to the rear for vehicle Hv and merging vehicle Mv. Arrow A1 shown in the upper left of Figure 3 indicates the direction of travel on the road. Arrow A2 indicates the reverse direction.

[0078] Figure 3 illustrates a road structure in which a barrier 306, such as a wall, exists between the preparation section 201 and the caution section 104. The barrier 306 is not present until the merging start point 102, and is replaced by a transition control 305 midway through the caution section 104. From the end of the barrier 306 to the merging section 101, poles serving as the transition control 305 are placed at predetermined intervals.

[0079] Information regarding the merging section 101 may include the remaining distance to the merging section. The remaining distance to the merging section is the distance from the vehicle Hv's current position to the merging start point. The remaining distance to the merging section may be obtained based on map data, or it may be determined based on data of guide signs detected by the front camera 111. The environmental recognition unit F1 may determine the remaining distance to the merging section based on vehicle data or sensor data received from the vehicle ahead. The environmental recognition unit F1 may calculate the remaining time until the vehicle Hv reaches the merging start point, which is the remaining time, from the remaining distance to the merging section and the control plan for the vehicle Hv's driving speed.

[0080] When the environmental recognition unit F1 detects that vehicle Hv is traveling on the main lane of the highway and has entered a warning zone, it may determine, based on information about the merging lane, whether vehicle Hv is traveling in the merging lane of the main lane. While traveling in the warning zone and the merging zone, the environmental recognition unit F1 detects explicit merging vehicles and potential merging vehicles based on camera images in the merging direction. Explicit merging vehicles are those whose physical bodies are located within the FOV of the environmental sensor 11. Explicit merging vehicles can also be described as merging vehicles that are not hidden by obstacles or merging vehicles that are visible. Potential merging vehicles are those whose physical bodies are outside the FOV, as described below. Potential merging vehicles correspond to merging vehicles that are hidden by obstacles or other vehicles.

[0081] The environmental recognition unit F1 may acquire data indicating the external environment related to ODD, in addition to the information described above. The environmental recognition unit F1 may also acquire traffic rules around the vehicle Hv based on sensor data or map data. Traffic rules may include speed limits and restrictions on lane changes. The environmental recognition unit F1 may generate an environmental model, which is a three-dimensional model that reproduces (represents) the driving environment of the vehicle Hv, as data indicating the external environment. The environmental model may also be called a world model. The environmental model may be a model in which objects detected by the environmental sensor 11, such as moving objects such as other vehicles, lane markers, road edges, and traffic lights, are placed in a three-dimensional space based on the vehicle Hv. The environmental recognition unit F1 may be understood as having a configuration that manages data related to the driving environment. Data management here may include data acquisition (generation) and updating.

[0082] In addition, the environmental recognition unit F1 may acquire information indicating the in-vehicle environment, such as in-vehicle temperature and driver status data. Specifically, it may acquire the driver's eye position, etc. Furthermore, the environmental recognition unit F1 acquires information on the mounting position of the front camera 111 in the vehicle Hv. The mounting position information of the front camera 111 may be registered in the storage 33.

[0083] The mode management unit F2 manages the operating mode of the vehicle control device 30 based on the information acquired by the communication circuit 34. The management of the operating mode may include the switching between manual driving and automatic driving, that is, the transfer of authority between the user and the vehicle control device 30, in other words, the management of the handover of driving (takeover). Furthermore, the management of the operating mode may include the transition from manual driving mode to Level 2 mode and vice versa, and the transition from Level 2 mode to AD mode and vice versa.

[0084] Mode management corresponds to automation level management. The mode management unit F2 estimates the operating mode / automation level intended by the driver based on the operation signal input from the operating member 21. Based on the estimation result, the mode management unit F2 switches the operating mode. For convenience in this disclosure, the currently applied operating mode is also referred to as the current mode.

[0085] The planning unit F3 is configured to create a driving plan based on data of the driving environment (e.g., an environmental model) managed by the environment recognition unit F1. While in AD mode or Level 2 mode, the planning unit F3 generates driving plan data for autonomous driving based on the results of the environment recognition unit F1's recognition of the driving environment. The driving plan may be called a control plan or a driving plan.

[0086] The driving plan data may include route data, track data, and motion plan data. Route data is data that shows the overall (long-term) driving plan, such as the route to the destination. The planning unit F3 may generate route data based on map data that shows the road connections, such as navigation map data.

[0087] Track data is data that shows the relatively local (short-term) driving trajectory. The track plan may include data such as the lane in which the vehicle Hv is traveling, the position of the vehicle Hv within the lane (so-called lateral position), and lane change points. The processor 31 is configured to perform lateral position adjustments according to the situation. Generating track data may also be called track planning or path planning.

[0088] The planning unit F3 may generate track data based on route data and driving environment data. Motion plan data is data that shows the target speed, steering angle, acceleration, etc. for each time point. Motion plan data may be generated based on track data. Thus, the driving plan data may include schedule information for acceleration and deceleration for speed adjustment along the set route / track, and schedule information for steering amount. The driving plan created by the planning unit F3 is input to the vehicle control unit F4.

[0089] The above driving plan may be created based on map data. Automatic driving control technology based on map data may also be called NOA (Navigate on Autopilot). The planning unit F3 may be an implementation of NOA. The planning unit F3 may create or modify the control plan using the detection results of the environmental sensor 11 instead of or in addition to the map data.

[0090] The planning unit F3 generates not only control plans directly related to the vehicle's operation, but also plans related to notifications to the driver using notification devices such as the display 17. For example, the planning unit F3 plans the timing for issuing warnings / requests to the driver, such as behavior warnings, mode change notifications, and TOR (takeover request). A behavior warning is a process that warns of planned vehicle behaviors such as lane changes, overtaking, and deceleration. A mode change notification is a process that notifies the driver that the operating mode will be changed, or that the operating mode will be changed. A TOR is a request made by the vehicle control device 30 to the driver to take over driving operations. A TOR can be rephrased as an intervention request or a takeover request.

[0091] Various notifications, including announcements, proposals, reports, and requests, may include displaying an icon image or text on the display 17, depending on their content. Depending on their importance and urgency, various notifications may also include some or all of the following: outputting a notification sound, outputting a voice message, flashing ambient lights, and vibrating a vibrator. The process for informing the driver of the information may involve generating a signal to drive the notification device and outputting it to the notification device. The planning unit F3 creates notification plan data indicating the content of the notification and the timing of the notification, and provides it to the vehicle control unit F4.

[0092] The vehicle control unit F4 generates control commands for the motion actuator 22 based on the control plan formulated by the planning unit F3. The vehicle control unit F4 then outputs the generated control commands to the motion actuator 22. In addition, the vehicle control unit F4 controls the illumination status of turn signals, headlights, hazard lights, etc., according to the driving plan and driving environment, based on the plan from the planning unit F3 and the external environment. In other words, the vehicle control unit F4 also controls the drive state of the auxiliary actuator 23.

[0093] The vehicle control unit F4 also provides notifications to the driver using notification devices such as the display 17 and the speaker 18. Various notifications are provided by displaying an image on the display 17 and at least one of the following: an audio message or notification sound output from the speaker 18. The notification sound may be a warning sound. Notifications to the driver may also be accompanied by the illumination of ambient lights or the activation of a vibrator. The vehicle control unit F4 executes various notifications based on the plan of the planning unit F3. In other words, the vehicle control unit F4 executes notifications to the driver based on requests from the planning unit F3.

[0094] <Detection of Potential Merging Vehicles> If an obstruction 306 is erected in the merging direction of the warning section, the sensor field of view of vehicle Hv is narrowed by the obstruction 306, as shown in Figure 4. The sensor field of view is the range in which objects can be detected by the environmental sensor 11, and is sometimes called FOV (Field of View). The FOV of vehicle Hv may be the range in which the front camera 111 can capture images. If there are multiple front cameras 111, the FOV of vehicle Hv may be a combination of the FOVs of the multiple front cameras. For the sake of simplicity, FOV is described here as the range in which the front camera 111 can capture images, but the FOV of vehicle Hv may be a combination of the FOVs of multiple types of environmental sensors 111. The FOV of vehicle Hv may also be a combination of the detection ranges of rear cameras, side cameras, radar, and LiDAR. In Figure 4, the dashed line indicates the original FOV boundary in the forward direction, with "FOV_R" representing the right edge of the FOV and "FOV_L" representing the left edge of the FOV. In contrast, the double-dotted line "FOV_L'" indicates the left edge boundary of the FOV restricted by the shielding object 306.

[0095] As shown in Figure 4, if there is an obstruction 306 between the warning zone and the merging lane, the timing at which a vehicle Hv in the warning zone can detect a merging vehicle (i.e., a potential merging vehicle) Mv in the preparation zone will be delayed. This is because the obstruction 306 delays the timing at which the actual merging vehicle Mv enters the FOV of vehicle Hv.

[0096] To address these challenges, the vehicle control device 30 of this embodiment estimates (in other words, predicts or detects) the presence of a hidden merging vehicle Mv based on the brightness characteristics of road elements related to the merging lane included in the image of the front camera 111. Road elements related to the merging lane include road elements that constitute the merging lane. For example, the road surface 321 of the merging lane, the merging lane marks 311, 312, and the far side structure 313 are road elements related to the merging lane. Road elements related to the merging lane may also include features associated with the merging lane. Road elements related to the merging lane dealt with in this disclosure may be features located within a height range that can be illuminated by the headlights of the merging vehicle.

[0097] At night, the merging vehicle Mv turns on its headlights. The light from the merging vehicle Mv's headlights illuminates road elements related to the merging lane in front of the merging vehicle Mv. Therefore, even if the merging vehicle Mv itself is not visible in the image of the front camera 111, the road elements illuminated by the merging vehicle Mv's headlights may be captured. For example, the road surface 321 of the merging lane, the merging lane markings 311, 312, and the structure 313 further back, which are illuminated by the merging vehicle Mv's headlights, can be captured before the actual image of the merging vehicle Mv is captured. Furthermore, the road surface 321 of the merging lane and the lane markings 311, etc., illuminated by the merging vehicle Mv's headlights may have different brightness values ​​than the surrounding road elements.

[0098] Here, since the area illuminated by the headlights is finite, the areas with high brightness on the road surface 321 and the merging lane markings 312 are limited, and the brightness of the areas not reached by the headlights is relatively low. In other words, even with continuously formed road elements, a difference in brightness can occur between the areas within and outside the range of illumination. The processor 31 may estimate the presence of a hidden merging vehicle Mv based on the fact that the road surface and merging lane markings are partially illuminated, specifically that the brightness of the relatively closer areas is higher than or equal to a predetermined value compared to the brightness of the relatively distant areas.

[0099] As shown in Figure 5, when the merging vehicle Mv has its headlights on, the vehicle control device 30 can detect the light emitted from the merging vehicle Mv's headlights, even if the merging vehicle Mv itself is outside the FOV. The presence of light emitted from the merging vehicle Mv's headlights indicates the presence of the merging vehicle Mv.

[0100] Based on the above concept, the processor 31 of this embodiment analyzes the image from the front camera 111 and detects the merging vehicle Mv based on the presence of road elements illuminated by the headlights of the merging vehicle Mv. Specifically, the processor 31 analyzes the image from the front camera 111 taken at a specific timing while driving through the warning section and obtains the brightness distribution as brightness features of road elements related to the merging lane. Then, the processor 31 detects a potential merging vehicle from the brightness features of road elements related to the merging lane.

[0101] For example, the processor 31 may detect a merging vehicle Mv based on the fact that pixels corresponding to the far-side structure 313, the merging lane mark 312, and the road surface 321 have a brightness value above a certain level. The processor 31 can detect this because lane markings and poles often have retroreflective properties and can have relatively high brightness values ​​when illuminated by headlights.

[0102] The processor 31 may also determine whether the road elements related to the merging lane are illuminated by headlights by comparing a pre-prepared reference image of the road elements with the actual reference image. Determining that the road elements related to the merging lane are illuminated by headlights corresponds to determining that there is a merging vehicle. As mentioned above, the reference image may be an image of a road element that is not illuminated by headlights at night. The processor 31 may detect a merging vehicle by comparing the brightness characteristics of the reference image with the brightness characteristics of the road elements that have actually been photographed, as described above.

[0103] Incidentally, the area in which the transition section 202 can be captured by the forward camera 111 is limited. The processor 31 may set a focus area according to the merging direction and determine the presence or absence of a merging vehicle Mv based on the image features (e.g., brightness distribution) of the focus area. In Figure 6, area Z1 represents the focus area when the merging direction is to the left, and area Z2 represents the focus area when the merging direction is to the right. Z3 corresponds to the area in which the vehicle's lane is likely to be captured. The area of ​​the camera image may be set considering the characteristics of the camera (vanishing point, etc.). By narrowing the image area to be processed, the processing load can be reduced.

[0104] Furthermore, the processor 31 may detect hidden merging vehicles by inputting the video from the front camera 111 to a predetermined trained detection model. The detection model here may be a machine learning model trained using video of the merging lane when no hidden merging vehicles exist and video of the merging lane when hidden merging vehicles exist. The detection model may be configured to output data indicating the probability of the presence of hidden merging vehicles from the input merging lane video. The probability may be expressed as a binary representation of 0 or 1, or as a probability value. The merging lane video may be video of the transition section 202, or video in the direction in which the transition section 202 exists. The merging lane video may be video showing a portion of the transition section, or video from the front camera 111 while driving in the caution section. The merging lane video may also be video of the aforementioned area of ​​interest taken while driving in the caution section. Based on the fact that the vehicle has entered the warning zone 104, the processor 31 may input the video from the forward camera 111 into the detection model and determine the presence or absence of a potential merging vehicle from the output of the detection model. The detection model may be configured to analyze the area of ​​interest. By narrowing the video area to be processed, as mentioned above, the processing load can be reduced, and the detection model can also be made smaller or lighter in terms of software.

[0105] Furthermore, as illustrated in Figure 5, if a preceding merging vehicle Mv2 is present on the merging lane, the rear of the preceding merging vehicle Mv2 will be illuminated by the lights of the merging vehicle Mv. Therefore, the processor 31 may determine that a merging vehicle Mv is present based on the fact that the rear of the preceding merging vehicle Mv2 is bright. The state in which the rear of the preceding merging vehicle Mv2 is bright means that the rear of the preceding merging vehicle Mv2 is illuminated by lights, and specifically, the brightness of the rear of the preceding merging vehicle Mv2, especially the brightness of the parts other than the taillights, may be above a predetermined value. Whether or not the rear of the preceding merging vehicle Mv2 is illuminated by headlights may also be determined using a trained model.

[0106] In this context, the preceding merging vehicle Mv2 is a merging vehicle traveling a predetermined distance or more ahead of vehicle Hv, and is not a competing vehicle. In the explanations of Figures 5 and 6, etc., merging vehicle Mv can be understood as a merging vehicle that could potentially be a competing vehicle.

[0107] The processor 31, acting as the environmental recognition unit F1, may detect potential (in other words, hidden) merging vehicles that could be competing vehicles using one or more of the methods described above. The processor 31 may also be configured to detect merging vehicles Mv whose physical form is partially or completely hidden by a method that partially modifies or improves upon the above methods. The process of indirectly searching for merging vehicles that could be competing vehicles from the brightness features of images of features constituting the merging lane and / or images of preceding merging vehicles is also called the indirect search process. The indirect search process can be understood as a process that estimates the existence of a hidden merging vehicle based on the fact that road structures and other vehicles are illuminated by a potential merging vehicle.

[0108] In addition, if the merging lane is partially visible through a gap in an obstruction, the processor 31 may determine whether a merging vehicle Mv exists from the image of that gap. For example, based on the detection results from the LiDAR / radar, the processor 31 identifies a gap region, which is an image area corresponding to a gap in an obstruction in the image captured by the camera. The gap region may be identified by comparing it with the color information of the obstruction. If the color of the gap region included in the captured image is not the color corresponding to the road surface, for example, if it is white, red, yellow, or blue, the processor 31 may determine that a merging vehicle Mv exists. Furthermore, if an object that can be recognized as part of a vehicle is captured in the gap region, the processor 31 may determine that a merging vehicle Mv exists. Whether or not an object that can be recognized as part of a vehicle is captured in the gap region may be determined using predetermined features or a trained model. The processor 31 may also determine that a merging vehicle Mv exists if a light source corresponding to a headlight, or a road element illuminated by it, is captured in the gap region.

[0109] <Example of Vehicle Control Device Operation> Here, the operation of the vehicle control device 30 when vehicle Hv is traveling on the main lane of a highway in AD mode will be explained using the flowcharts shown in Figures 7 and 8. The flowcharts shown from Figure 7 to 8 are examples and are executed from step S101. The description of the processor 31 as the executing entity in the following steps may be replaced with the vehicle control device 30. Also, the description of the processor 31 may be replaced with the environment recognition unit F1, mode management unit F2, planning unit F3, or vehicle control unit F4 depending on the context.

[0110] The following explanation will use the scenario shown in Figure 4, where the merging lane is located to the left of the main lane. In other words, "left" in the flowchart refers to the merging direction. In the following flowchart, "right" refers to the opposite direction of the merging direction (i.e., the retreat direction). The first lane in the flowchart may be replaced with the lane being merged into. The terms "second lane" or "passing lane" may be replaced with the lane adjacent to the merging lane in the retreat direction, i.e., the retreat lane, non-merging lane, or separating lane.

[0111] Step S101 is a step to determine whether vehicle Hv has entered a warning zone. Whether vehicle Hv has entered a warning zone may be determined based on vehicle position data and map data. For example, it may be determined that vehicle Hv has entered a warning zone when the remaining time before merging is less than a predetermined first threshold. The first threshold may be 10 seconds, for example. The processor 31 may also determine whether vehicle Hv has entered a warning zone based on information about the merging zone included in the sensor data. The processor 31 may also determine that vehicle Hv has entered a warning zone after detecting a merging caution sign with the front camera 111.

[0112] S101 may be executed periodically while driving on the main road. If the processor 31 determines that it has entered a warning zone (S101 YES), it determines in S102 whether vehicle Hv is driving in the first lane (i.e., the lane being merged into). S102 may include identifying the direction of merging based on map data or sign recognition results. S102 may also include identifying the vehicle's own lane. If the processor 31 determines that it is driving in the first lane (S102 YES), it executes the processes from S103 onwards.

[0113] Step S103 is a step in which it is determined whether there is an obstruction such as a wall in the merging direction (i.e., to the left). In S103, the processor 31 verifies whether there is an obstruction in the merging direction based on sensor data or map data for the merging direction. In this example, the processor 31 determines whether there is an obstruction on the left side of the first lane.

[0114] If no obstruction is detected in the merging direction (S103 NO), the processor 31 performs normal behavior in S110. Normal behavior here may basically be continuing to drive in the first lane, as before entering the warning zone. If the merging vehicle Mv is detected while continuing to drive in the first lane as normal behavior, the processor 31 may respond according to the position and speed of the merging vehicle Mv. Based on the position and speed of the merging vehicle Mv, the processor 31 predicts whether it will enter in front of or behind vehicle Hv. The prediction of the merging vehicle Mv's entry position may be based on the assumption that the merging vehicle Mv will accelerate at a reasonable rate.

[0115] If the processor 31 predicts that a merging vehicle Mv will cut in front of vehicle Hv, it may verify whether it is possible to change lanes in the direction of evacuation (for example, to the right). If it is possible to change lanes, it may be when there is sufficient empty space (hereinafter also referred to as necessary space) in the adjacent lane which is the target lane. If it is foreseen that a merging vehicle Mv will cut in front of vehicle Hv and it is possible to change lanes in the direction of evacuation, the processor 31 will implement evacuation control to the adjacent lane. Evacuation control to the adjacent lane refers to changing lanes from the merging lane to the evacuation lane (in this case, the second lane).

[0116] Furthermore, if the processor 31 foresees that merging vehicle Mv will cut in front of vehicle Hv and it is not possible to change lanes to the second lane, it will not change lanes but will decelerate on the first lane and yield the right of way to merging vehicle Mv. If the processor 31 foresees that merging vehicle Mv will enter behind vehicle Hv, it may continue driving at its current speed. Alternatively, if the processor 31 foresees that merging vehicle Mv will enter behind vehicle Hv, it may accelerate at a predetermined acceleration. The processor 31 may control vehicle Hv to avoid deceleration as much as possible.

[0117] On the other hand, if an obstruction is detected in the merging direction (to the left in this case) (S103 YES), the processor 31 determines in S104 whether the visibility is poor due to the preceding vehicle. Poor visibility is a state in which the field of view (FOV) in front has narrowed to a predetermined level, and the specific requirements for this can be defined as appropriate. The processor 31 determines whether the visibility is good or bad from the perspective of whether the merging section is sufficiently included in the FOV.

[0118] In a specific example, the processor 31 may determine that visibility is poor if the distance to the preceding vehicle is below a predetermined threshold (hereinafter also referred to as the reduced visibility determination value) (S104 YES). The reduced visibility determination value may be a fixed value, but it may also be adjusted according to the size or type of the preceding vehicle. If the preceding vehicle is a large vehicle such as a truck, a larger reduced visibility determination value may be applied than if the preceding vehicle is a light vehicle. The distance to the preceding vehicle may be determined based on sensor data. The type or size information of the preceding vehicle may also be determined based on sensor data. If there is no preceding vehicle, or if the distance to the preceding vehicle is greater than the reduced visibility determination value, the processor 31 may determine that forward visibility is good (S104 NO).

[0119] Whether or not forward visibility is good may be determined based on the image from the front camera 111. The processor 31 may also determine whether or not forward visibility is good based on the size of the road surface area captured in the image from the front camera 111. The processor 31 may be configured to make a negative determination in S104 even if forward visibility is poor due to weather conditions such as dense fog or heavy rain.

[0120] The decision in S104 may be made when the remaining time before merging reaches the second threshold. The second threshold is a smaller value than the first threshold. The second threshold may be 3 seconds or 5 seconds, for example. The second threshold may be set to be longer than the time required to make or execute the decision to change lanes, as described below.

[0121] If the processor 31 determines that forward visibility is poor (S104 NO), in S105 it determines whether the normal LC conditions, which are the conditions for performing a normal lane change, are met. LC stands for Lane Change. A normal lane change (hereinafter also referred to as normal LC) may be performed by steering at a predetermined steering speed when there is sufficient space on the target lane, causing the vehicle Hv to move to the target lane. The target lane is the escape lane. In the situation shown in Figure 4, the second lane VL2 may be the target lane. The lane in which the vehicle Hv is currently traveling is also called the current lane. In the situation shown in Figure 4, the first lane VL1 is the current lane.

[0122] The required space for normal LC may be designed as appropriate. Whether the required space exists in the target lane may be determined based on sensor data from the environmental sensor 11. Normal LC conditions include the presence of the required space on the target lane. S105 corresponds to the step of determining whether such normal LC is feasible.

[0123] The normal LC conditions may include not only the existence of the necessary space on the target lane, but also compliance with traffic rules, such as the lane markings between the target lane and the current lane being of a pattern that permits lane changes (e.g., dashed lines). The normal LC may include activating the turn signal lamps according to the direction of movement at a predetermined timing. The normal LC may include confirming the existence of the necessary space on the target lane based on sensor data from the environmental sensor 11 related to the direction of movement. The normal LC may be performed while maintaining a constant speed, or while accelerating at a predetermined acceleration. The storage 33 may include steering speed, maximum steering angle, or acceleration as control parameters that define the behavior of the vehicle Hv during a lane change.

[0124] In S105, if the processor 31 determines that a normal LC (Lane Change) can be performed, it plans to perform a normal LC, activates the turn signal lamps as appropriate, and starts the lateral movement of the vehicle Hv (S106). According to the processing flow in S104, S105, and S106, if the merging section cannot be seen due to obstruction by a preceding vehicle, the processor 31 moves the vehicle Hv from the first lane to the second lane in advance. In this manner, moving the vehicle Hv to an adjacent lane in advance in preparation for the appearance of a merging vehicle in a warning section will be referred to as pre-emptive evacuation control below.

[0125] As shown in Figure 9, pre-emptive evasive control prevents excessive proximity between vehicle Hv and merging vehicle Mv, even if Mv appears near the merging start point. Pre-emptive evasive control reduces the frequency of situations where vehicle Hv has to slow down and yield its right of way in response to a reckless cut-in. Specifically, even if merging vehicle Mv suddenly appears at the merging start point, vehicle Hv can continue driving without slowing down. As a result, the risk of compromising the driver comfort of vehicle Hv is reduced. Furthermore, by leaving space behind the preceding vehicle Pv in anticipation of the appearance of merging vehicle Mv, merging vehicle Mv can smoothly enter the main lane. Pre-emptive evasive control can improve the flow of traffic.

[0126] After the lane change is complete, the processor 31 may maintain driving in the second lane until the vehicle has exited the merging section. Upon exiting the merging section, the processor 31 may execute control to return vehicle Hv from the second lane to the first lane.

[0127] On the other hand, if the processor 31 determines in S105 that it is not possible to perform a normal LC (S105 NO), it continues driving in the first lane while monitoring the behavior of the preceding vehicle Pv. Subsequently, if it detects an action by the preceding vehicle Pv to change lanes to the right, it starts a cooperative LC. The action of the preceding vehicle Pv to change lanes to the right may be the illumination of the right-side turn signal lamp of the preceding vehicle Pv. The action of the preceding vehicle Pv to change lanes to the right may be the preceding vehicle Pv moving its driving position to the right of the lane center, or maintaining that state. The action of the preceding vehicle Pv to change lanes to the right corresponds to the preceding vehicle Pv taking a retreat action.

[0128] Cooperative Lane Change (LC) conceptually involves changing lanes while communicating with other vehicles traveling in the target lane. Cooperative Lane Change may include having other vehicles traveling in the target lane cooperate (e.g., by slowing down) to create space for vehicle Hv in the target lane. Cooperative Lane Change may include first activating the turn signal lamps and then monitoring the movement of other vehicles in the target lane. The other vehicles to be monitored are those that contribute to creating space for vehicle Hv to enter the target lane. Specifically, it is the vehicle closest to vehicle Hv to the right rear. If another vehicle happens to be directly beside vehicle Hv, cooperative Lane Change may include slowing down to move behind that other vehicle. The settings for control parameters such as steering speed and acceleration in cooperative Lane Change may differ from the settings for normal Lane Change. The steering speed and acceleration in cooperative Lane Change may be set to smaller values ​​than those in normal Lane Change. This allows for a smoother behavior of vehicle Hv.

[0129] Upon receiving confirmation that a space has been created in the target lane for vehicle Hv to enter, due to the cooperation of other vehicles in the target lane, or in other words, the acquisition of the right of way, the processor 31 executes a lane change. The control after the completion of the cooperative LC may be the same as the control after the completion of a normal LC. Upon receiving confirmation that vehicle Hv has left the merging section, the processor 31 may execute control to return vehicle Hv to the first lane.

[0130] According to S104-S107, if the forward view is obstructed by a preceding vehicle and there is no space in the second lane normally required for a merging lane, and the processor 31 detects that the preceding vehicle is moving to the second lane, the processor 31 moves vehicle Hv to the second lane to follow the preceding vehicle. The fact that the preceding vehicle is moving to the second lane suggests that there is a vehicle merging in the merging lane. With the above control, even if the environmental sensor 11 of vehicle Hv does not detect the actual merging vehicle, it is possible to avoid the merging vehicle cutting in. In other words, the behavior of vehicle Hv can be made more rational and safer.

[0131] Furthermore, evasive control in the form of following the preceding vehicle may be executed only when the processor 31 cannot observe the condition of the merging lane. In other words, evasive control in the form of following the preceding vehicle may be executed only when there is an obstruction on the left side of vehicle Hv and the forward view is obstructed by the preceding vehicle. In sections where there is no obstruction on the left side of vehicle Hv, such as merging sections, the merging lane may be included in the vehicle Hv's own FOV. In sections where there is no obstruction on the left side, the processor 31 may determine whether or not evasive control is necessary based on the condition of the merging lane detected by the vehicle's environmental sensor 11.

[0132] If the processor 31 determines in S104 that forward visibility is good (S104 YES), the processor 31 determines in S120 whether it is nighttime. Whether it is nighttime may be determined based on the detection result of the illuminance sensor. Alternatively, whether it is nighttime may be determined based on time information. The determination of whether it is nighttime may correspond to whether the vehicle Hv's headlights are on or not. If the processor 31 determines that it is not nighttime (S120 NO), it performs lateral position adjustment in S121 and continues driving in the first lane. Lateral position adjustment is a control that shifts the vehicle Hv's lateral position away from the center of the lane. Lateral position here refers to the driving position of the vehicle Hv in the road width direction or vehicle width direction. Lateral position adjustment is also called VLO (Vehicle Lateral Offset).

[0133] Lateral position adjustment in S121 is a control that shifts the vehicle Hv's driving position in the direction of retreat from the center of the merging lane. The amount by which the driving position is shifted from the lane center in lateral position adjustment is also called the offset amount. The offset amount may be a fixed value such as 0.5 m. Lateral position adjustment may be performed within the range in which the vehicle Hv does not encroach on the adjacent lane. The offset amount may be dynamically determined according to the lane width, for example, it may be a value such that the distance between the lane mark and the vehicle Hv is a predetermined value. The offset amount may be adjusted so that the distance between the lane mark in the direction of retreat and the wheels of the vehicle Hv in the direction of retreat is greater than or equal to a predetermined value.

[0134] Lateral position adjustment can increase the visible area for merging lanes (especially transition sections). Figure 10 illustrates the effect of lateral position adjustment. In Figure 10, 401 represents the left edge of the FOV when vehicle Hv is located in the center of the lane, and 402 represents the right edge of the FOV when vehicle Hv is located in the center of the lane. The dashed lines in the figure represent the laterally adjusted vehicle Hv and its FOV. 401a represents the left edge of the FOV of the laterally adjusted vehicle Hv, and 402a represents the right edge of the FOV. By adjusting the lateral position, the range of the merging lane in which vehicle Hv is visible increases, corresponding to the hatched area with diagonal lines. As a result, the processor 31 can more easily detect the merging vehicle Mv.

[0135] Furthermore, if the processor 31 determines that it is nighttime (S120 YES), in S122 it performs lateral position adjustment and continues driving in the first lane while performing indirect search processing. As described above, the indirect search processing is a process that searches for merging vehicles that may be competing vehicles based on the brightness characteristics of the images of the road elements that constitute the merging lane. The indirect search processing may also include detecting hidden merging vehicles based on the brightness characteristics of the images of the rear of the preceding merging vehicle.

[0136] After S121 or S122, if the processor 31 detects the presence of a merging vehicle Mv from the sensor data, or if it determines through indirect search processing that a potential merging vehicle exists (S123 YES), it determines in S124 whether a normal LC is possible. The determination in S124 may be the same as the determination in S105. If the processor 31 determines in S124 that a normal LC is possible (S124 YES), it executes a normal LC in S126. After the completion of the LC, the processor 31 may execute control to return vehicle Hv to the first lane after it has left the merging section.

[0137] On the other hand, if the processor 31 determines in S124 that normal LC is not possible (S124 NO), it starts cooperative LC in S125. S125 may be the same as S108.

[0138] S123 may be performed periodically while driving in the merging lane until the vehicle has left the merging section. If no merging vehicle Mv is detected (S123 NO), S127 determines whether the vehicle has left the merging section. This determination may be performed by comparing map data with the vehicle's own position data. The processor 31 may also determine that the vehicle has left the merging section if it detects that the merging section has disappeared based on the sensor data. If vehicle Hv is still located within the merging section or within the warning section (S127 NO), the processor 31 repeats S123.

[0139] Note that S105 may be omitted, and if forward visibility is poor due to the preceding vehicle, the vehicle may temporarily continue driving in the merging lane. Subsequently, in response to the preceding vehicle beginning to change lanes in the direction of evacuation, the processor 31 may perform a normal LC or a cooperative LC. When the preceding vehicle begins to change lanes in the direction of evacuation, if there is sufficient space in the target lane, the processor 31 creates and executes a normal LC plan. When the preceding vehicle begins to change lanes in the direction of evacuation, if there is not sufficient space in the target lane, the processor 31 creates and executes a cooperative LC plan.

[0140] Furthermore, the processor 31 may be configured to change lanes to the second lane when vehicle Hv is traveling in a warning zone and the preceding vehicle slows down as it approaches the merging point. This is because the preceding vehicle slowing down indicates the presence of a merging vehicle. The preceding vehicle's deceleration may be detected by a reduction in the distance between vehicles detected by radar or the like, or by the illumination of the brake lights.

[0141] Furthermore, the processor 31 may be configured to change lanes to the second lane when vehicle Hv is traveling in a warning zone and the preceding vehicle accelerates as it approaches the merging point. This is because the sudden acceleration of the preceding vehicle also suggests the presence of a merging vehicle. The acceleration of the preceding vehicle may be detected by an increase in the distance between vehicles detected by radar or the like. In this way, the processor 31 may use the movement of the preceding vehicle near the merging point as a clue to determine whether or not there is a merging vehicle.

[0142] <Summary> With the above configuration, if there is an obstruction between the merging lane and the main road, the processor 31 searches for potential merging vehicles based on the brightness features of the images of road elements related to the merging lane. This allows the processor 31 to detect merging vehicles whose bodies are outside the FOV of vehicle Hv. In other words, the processor 31 can detect merging vehicles earlier. In addition, the processor 31 estimates the presence of potential merging vehicles by using the brightness features of the rear of other vehicles traveling at a predetermined distance or more ahead of vehicle Hv in the merging lane as a clue. This also allows the processor 31 to detect merging vehicles earlier.

[0143] Furthermore, when the vehicle is traveling in a warning zone and there is no vehicle ahead, the processor 31 increases the visibility of the merging lane by adjusting its lateral position. This allows the processor 31 to detect merging vehicles even earlier.

[0144] In one embodiment included in this disclosure, if the processor 31 is traveling in a warning zone and there is an obstruction in the merging direction and forward visibility is poor due to a preceding vehicle, it will move to the lane in the escape direction in advance. This prior escape control reduces the impact (e.g., deceleration) that a merging vehicle has on the vehicle Hv, even if the merging vehicle appears immediately after entering the merging zone.

[0145] Furthermore, in one embodiment included in this disclosure, when vehicle Hv is traveling in a warning zone, the processor 31 detects the behavior of a preceding vehicle regarding a lane change in the direction of evacuation, and moves vehicle Hv to the evacuation lane. Based on the behavior of the preceding vehicle, the processor 31 may estimate the presence of a merging vehicle. This reduces the opportunity to decelerate vehicle Hv by changing lanes in advance, even when the traffic conditions ahead are unknown, if there is a high probability that a merging vehicle is present.

[0146] <Modification> The processor 31 may be configured to attempt to move to the second lane if, while the vehicle Hv is traveling in the merging lane and the remaining time until reaching the merging start point is less than a predetermined value, an image of the merging lane cannot be acquired. The case where an image of the merging lane cannot be acquired is when, even after analyzing the forward camera 111, road elements corresponding to the merging lane shown on the map (e.g., lane mark 313) cannot be detected. Thus, the condition for executing pre-emptive control related to the merging section may be the inability to acquire an image of the merging lane.

[0147] In this disclosure, a situation in which vehicle Hv is traveling in the merging lane and there is no space for vehicle Hv to change lanes in the direction of escape is also referred to as an escape-difficult situation. When the processor 31 detects a merging vehicle that corresponds to a competing vehicle in an escape-difficult situation, it may be configured to perform a process to adjust the distance to the preceding vehicle, either in place of or in conjunction with a coordinated LC trial. The adjustment of the distance to the preceding vehicle may be done by reducing the distance by accelerating, or by increasing the distance by decelerating. Whether to shorten or lengthen the distance to the preceding vehicle may be determined based on the position, speed, and acceleration of the merging vehicle.

[0148] The processor 31 may be configured to perform indirect search processing only when traveling in a caution zone or a merging zone. The processor 31 may be configured not to perform indirect search processing when traveling in a normal zone. A normal zone is a road section that is neither a merging zone nor a caution zone. Furthermore, the processor 31 may be configured to perform indirect search processing only when traveling in a caution zone or a merging zone AND traveling in a lane being merged into. By limiting the conditions under which indirect search processing is performed, the processing load on the processor 31 can be reduced.

[0149] Vehicle Hv has a left headlight and a right headlight. When performing indirect search processing, if the vehicle Hv's headlight on the merging direction is in high beam mode, the accuracy of detecting hidden merging vehicles may deteriorate. This is because it is impossible to distinguish whether the light received by the road elements constituting the merging lane is from the merging vehicle or from vehicle Hv. For this reason, when the processor 31 performs indirect search processing, it may temporarily switch the illumination state of the merging direction headlight from high beam to low beam, as shown in Figure 11. Alternatively, the processor 31 may temporarily turn off the merging direction headlight. In other words, the processor 31 reduces the amount of light emitted by the merging direction headlight for a predetermined period of time required to acquire the video used for indirect search processing. This may improve the accuracy of the indirect search processing. The length of the period for which the output of the merging direction headlight is reduced may correspond to the time required to acquire the image frames necessary for indirect search processing. For example, it may be 400 milliseconds or 600 milliseconds.

[0150] Figure 11 shows that 51H indicates the illumination range when the right headlight is illuminated in high beam mode. 52H indicates the illumination range when the left headlight is illuminated in high beam mode. 52L indicates the illumination range when the left headlight is illuminated in high beam mode. Dynamic (adaptive) control of the illumination range of the headlights can be achieved by switching the light source module being driven, changing the optical axis direction, or by other methods.

[0151] <Supplement> The above description mainly concerns the case where the vehicle control device 30 is operating in AD mode. Therefore, it describes the case where the processor 31 attempts to change lanes without obtaining driver approval for initiating a lane change. However, the above explanations may also apply to Level 2 mode.

[0152] When the vehicle control device 30 is operating in Level 2 mode, the processor 31 may perform an intention confirmation process to request the driver's approval for a lane change if it determines that a normal LC or cooperative LC is necessary. Determining the necessity of a normal LC or cooperative LC may occur, for example, when an affirmative determination is made in S105, S107, or S124, or when a negative determination is made in S125.

[0153] The intention confirmation process may include displaying a message on the display 17 asking whether it is OK to change lanes. The intention confirmation process may also include outputting a predetermined notification sound or an audio message from the speaker 18 asking whether it is OK to change lanes. These actions correspond to a proposal to change lanes. The processor 31 may obtain the driver's response through the driver's operation of the turn signal lever or voice input. If the processor 31 has made a proposal for a lane change and no positive response has been received after a predetermined response waiting time has elapsed, it may cancel the lane change. Conversely, if the processor 31 has made a proposal for a lane change and no negative response has been received after a predetermined response waiting time has elapsed, it may determine that the lane change has been accepted and start the lane change.

[0154] The term "remaining time to start merging" as described above may be replaced with "remaining distance to start merging," which is the remaining distance until vehicle Hv reaches the merging point. The processor 31 may execute the processing from S102 onward if the remaining time or distance until vehicle Hv reaches the starting point of the merging section is less than a predetermined value. The term "expressway" as described above may be replaced with "motor vehicle-only road." "Main line" may be understood as the main line of a motor vehicle-only road.

[0155] The vehicle control device 30, which performs everything from environmental recognition to control plan creation as described above, corresponds to a vehicle device in one phase. The vehicle control device 30 may be physically or software-separated into a functional unit that recognizes the driving environment, including merging vehicles (equivalent to an environmental recognition unit), and a functional unit that determines / plans vehicle behavior based on the detection results of merging vehicles. In other phases, the vehicle device may be an environmental recognition device that performs only environmental recognition.

[0156] <Addendum (1)> This specification discloses several technical ideas and several combinations thereof, as listed below.

[0157] [Technical Concept 1] A vehicle device comprising: a processing unit (31) that performs processing related to the detection of other vehicles based on data input from one or more sensors mounted on the vehicle; and a communication circuit (34) for the processing unit to communicate with the one or more sensors, wherein the one or more sensors include a camera that photographs the front of the vehicle; and the processing unit is configured to acquire an image related to a merging lane from the camera using the communication circuit when the vehicle is traveling on the main lane of an expressway, and to detect a merging vehicle based on the brightness characteristics of road elements related to the merging lane included in the acquired image.

[0158] [Technical Concept 2] The vehicle device according to Technical Concept 1, wherein the processing unit is configured to detect a merging vehicle based on the brightness of the road surface of the merging lane.

[0159] [Technical Concept 3] The vehicle device according to technical concept 1 or 2, wherein the processing unit is configured to detect the presence of a merging vehicle based on the brightness information of an image of a lane mark defining the merging lane.

[0160] [Technical Concept 4] The vehicle device according to any one of Technical Concepts 1 to 3, wherein the processing unit is configured to detect a merging vehicle based on brightness information of an image of a three-dimensional structure provided along the merging lane.

[0161] [Technical Concept 5] The vehicle device according to any one of Technical Concepts 1 to 4, wherein the processing unit is configured to detect a merging vehicle located to the side of the vehicle based on brightness information of an image of the rear of another vehicle traveling in the merging lane.

[0162] [Technical Concept 6] The processing unit is configured to determine whether the vehicle is traveling in the merging lane, which is the lane into which a merging vehicle enters from the merging lane on the main line, based on data received from one or more sensors using the communication circuit, and if the vehicle is traveling in the merging lane and the remaining time or distance to reach the starting point of the merging section falls below a predetermined value, an image of the merging lane cannot be obtained, the processing unit is configured to attempt or propose moving to a lane adjacent to the merging lane, as described in any one of Technical Concepts 1 to 5.

[0163] [Technical Concept 7] The processing unit is configured to detect a preceding vehicle based on data received from one or more sensors using the communication circuit, to determine whether the vehicle is traveling in the merging lane, which is the lane into which a merging vehicle enters from the merging lane on the main line, based on data received from one or more sensors using the communication circuit, and if the vehicle is traveling in the merging lane and the remaining time or distance to reach the starting point of the merging section falls below a predetermined value, and the distance to the preceding vehicle is below a predetermined value, the processing unit is configured to attempt or propose moving to a lane adjacent to the merging lane, as described in any one of Technical Concepts 1 to 6.

[0164] [Technical Concept 8] The vehicle device according to technical concept 7, wherein the processing unit is configured to acquire size information of the preceding vehicle based on data received from one or more sensors, and to change the predetermined value according to the size information.

[0165] [Technical Concept 9] The processing unit is configured to detect a preceding vehicle based on data received from one or more sensors using the communication circuit, to detect the operating state of the preceding vehicle's turn signal based on the data when a preceding vehicle is detected, to determine whether the vehicle is traveling in the merging lane, which is the lane into which a vehicle merges from a merging lane on the main line, based on data received from one or more sensors using the communication circuit, and to attempt or propose moving to a lane adjacent to the merging lane when the operation of the preceding vehicle's turn signal is detected while the vehicle is traveling in the merging lane and the remaining time or distance to reach the start point of the merging section is less than a predetermined value. This is a vehicle device according to any one of technical concepts 1 to 8.

[0166] [Technical Concept 10] The vehicle device according to any one of Technical Concepts 1 to 9, wherein the processing unit is configured to detect other vehicles and available space around the vehicle based on data received from one or more sensors using the communication circuit, to determine whether the vehicle is traveling in a merging lane, which is a lane into which a merging vehicle enters from a merging lane on the main road, based on data received from one or more sensors using the communication circuit, and when the vehicle is traveling in the merging lane, if the presence of a merging vehicle hidden by an obstruction is detected based on the brightness characteristics of road elements related to the merging lane, the device is configured to attempt or propose moving to a lane adjacent to the merging lane.

[0167] [Technical Concept 11] A vehicle device according to any one of Technical Concepts 1 to 10, configured to recognize that the vehicle is traveling in a warning section, which is the section before a merging section on the main line of an expressway, based on data input from one or more sensors, and to perform processing to detect a merging vehicle based on the brightness characteristics of road elements related to the merging lane included in the image acquired from the camera, provided that the vehicle is traveling in the warning section or the merging section.

[0168] [Technical Concept 12] A vehicle device according to any one of technical concepts 1 to 11, used in connection with an actuator for controlling the behavior of the vehicle, wherein the vehicle device is configured to determine the vehicle behavior in accordance with the detection result of the merging vehicle, and to generate a control signal corresponding to the determined vehicle behavior and output it to the actuator.

[0169] [Other Technical Ideas] This disclosure may apply not only when vehicle Hv, as the vehicle itself, is traveling in the merging lane of the main line of an expressway, but also when it is traveling in the merging lane. When the vehicle is traveling in the preparation section of the merging lane, the processor 31 may attempt to detect competing vehicles on the main line based on the brightness characteristics of the road elements of the merging lane. In other words, the indirect search process may also be performed in scenarios where the vehicle is transitioning from the merging lane to the main line. This disclosure further includes the following technical ideas and corresponding methods and programs.

[0170] A vehicle device comprising: a processing unit (31) that performs processing related to the detection of other vehicles based on data input from one or more sensors mounted on the vehicle; and a communication circuit (34) for the processing unit to communicate with at least one of the sensors, wherein the one or more sensors include a camera that photographs the front of the vehicle, and the processing unit is configured to, when the vehicle is traveling on a merging lane that connects to the main line of an expressway, acquire an image related to the merging lane that connects to the merging lane on the main line of the expressway from the camera using the communication circuit, and detect a merging vehicle based on the brightness characteristics of road elements related to the merging lane included in the acquired image.

[0171] <Addendum (2)> The various flowcharts shown in this disclosure are all examples, and the number of steps constituting the flowchart and the order of execution of processes can be changed as appropriate. The controls shown in each flowchart may be combined and executed in parallel to the extent that they do not contradict each other. Expressions such as acquisition, determination, detection, generation, and calculation may be used interchangeably. When a device acquires certain data, it also includes when the device generates such data based on signals input from other devices / sensors. Connections in this disclosure are not limited to direct connections by cables, but also include indirect connections. Indirect connections are connections via networks or other devices. For example, when a first device and a second device are connected, it includes when the first device is connected to the second device via a third device or a network, etc.

[0172] The devices, systems, and methods described in this disclosure may be implemented by a dedicated computer comprising a processor programmed to perform one or more functions embodied by a computer program. The devices and methods described in this disclosure may be implemented using dedicated hardware logic circuits. The devices and methods described in this disclosure may be implemented by one or more dedicated computers comprising a combination of a processor that executes a computer program and one or more hardware logic circuits. The processor may be any arithmetic core, such as a CPU, MPU, GPU, or DFP (Data Flow Processor). Some or all of the functions of the processing unit may be implemented as hardware. Some or all of the functions of the processing unit may be implemented using a system-on-a-chip (SoC), integrated circuit (IC), or field-programmable gate array (FPGA).

[0173] A computer program includes instructions that are executed by a computer. A computer program may be stored on a computer-readable, non-transitory tangible storage medium. The storage medium for a computer program may be a variety of media, such as an HDD (Hard-disk drive), an SSD (Solid State Drive), or flash memory.

Claims

1. A vehicle device comprising: a processing unit (31) that performs processing related to the detection of other vehicles based on data input from one or more sensors mounted on the vehicle; and a communication circuit (34) for the processing unit to communicate with the one or more sensors, wherein the one or more sensors include a camera that photographs the front of the vehicle, and the processing unit is configured to acquire an image related to a merging lane from the camera using the communication circuit when the vehicle is traveling on the main lane of an expressway, and to detect a merging vehicle based on the brightness characteristics of road elements related to the merging lane included in the acquired image.

2. The vehicle device according to claim 1, wherein the processing unit is configured to detect a merging vehicle based on the brightness of the road surface of the merging lane.

3. The vehicle device according to claim 1, wherein the processing unit is configured to detect the presence of a merging vehicle based on brightness information of an image of a lane mark defining the merging lane.

4. The vehicle device according to claim 1, wherein the processing unit is configured to detect a merging vehicle based on brightness information of an image of a three-dimensional structure provided along the merging lane.

5. The vehicle device according to claim 1, wherein the processing unit is configured to detect a merging vehicle located to the side of the vehicle based on brightness information of an image of the rear of another vehicle traveling in the merging lane.

6. The vehicle device according to any one of claims 1 to 5, wherein the processing unit is configured to determine whether the vehicle is traveling in a merging lane, which is a lane into which a merging vehicle enters from a merging lane on the main line, based on data received from one or more sensors using the communication circuit, and if the vehicle is traveling in the merging lane and the remaining time or distance to reach the starting point of the merging section falls below a predetermined value, an image of the merging lane cannot be obtained, the processing unit is configured to attempt or suggest moving to a lane adjacent to the merging lane.

7. The vehicle device according to any one of claims 1 to 5, wherein the processing unit is configured to detect a preceding vehicle based on data received from one or more sensors using the communication circuit, to determine whether the vehicle is traveling in a merging lane, which is a lane into which a merging vehicle enters from a merging lane on the main line, based on data received from one or more sensors using the communication circuit, and if the vehicle is traveling in the merging lane and the remaining time or distance to reach the start point of the merging section falls below a predetermined value, and the distance to the preceding vehicle is also below a predetermined value, the device is configured to attempt or propose moving to a lane adjacent to the merging lane.

8. The vehicle device according to claim 7, wherein the processing unit is configured to acquire size information of the preceding vehicle based on data received from one or more sensors, and to change the predetermined value according to the size information.

9. The vehicle device according to any one of claims 1 to 5, wherein the processing unit is configured to detect a preceding vehicle based on data received from one or more sensors using the communication circuit, to detect the operating state of the preceding vehicle's turn signal based on the data if a preceding vehicle is detected, to determine whether the vehicle is traveling in a merging lane, which is a lane into which a merging vehicle enters from a merging lane on the main line, based on data received from one or more sensors using the communication circuit, and if the processing unit is traveling in the merging lane and the remaining time or distance to reach the start point of the merging section is less than a predetermined value, and the operation of the preceding vehicle's turn signal is detected, the processing unit is configured to attempt or propose moving to a lane adjacent to the merging lane.

10. The vehicle device according to claim 1, wherein the processing unit is configured to detect other vehicles and available space around the vehicle based on data received from one or more sensors using the communication circuit, to determine whether the vehicle is traveling in a merging lane, which is a lane into which a merging vehicle enters from a merging lane on the main road, based on data received from one or more sensors using the communication circuit, and if the processing unit is traveling in the merging lane, and the presence of a merging vehicle hidden by an obstruction is detected based on the brightness characteristics of road elements related to the merging lane, the processing unit is configured to attempt or suggest moving to a lane adjacent to the merging lane.

11. The vehicle device according to claim 1, configured to recognize that the vehicle is traveling in a warning section, which is the section before a merging section on the main line of an expressway, based on data input from one or more sensors, and to perform processing to detect a merging vehicle based on the brightness characteristics of road elements related to the merging lane included in the image acquired from the camera, provided that the vehicle is traveling in the warning section or the merging section.

12. A vehicle device according to claim 1, used in connection with an actuator for controlling the behavior of the vehicle, wherein the vehicle device is configured to determine the behavior of the vehicle in accordance with the detection result of the merging vehicle, and to generate a control signal corresponding to the determined vehicle behavior and output it to the actuator.

13. A program that causes a computer configured to receive video footage from a camera that photographs the front of a vehicle to, when the vehicle is traveling on the main lane of an expressway, to receive images related to a merging lane from the camera via a communication circuit, and to determine whether or not a merging vehicle is present based on the brightness characteristics of road elements related to the merging lane included in the received images.

Images