Display image generation device

Abstract

This display image generation device for generating a display image including at least a part of the periphery of a host vehicle in a display range comprises: a captured image acquisition unit for acquiring a plurality of captured images of the periphery of the host vehicle captured by a plurality of imaging devices provided in the host vehicle; a multi-viewpoint image generation unit for generating a multi-viewpoint image from the plurality of captured images by viewpoint conversion based on installation information on the plurality of imaging devices in the host vehicle and the calculation results of the parallaxes of the plurality of captured images; a target object acquisition unit for selecting a target object from a plurality of objects included in the plurality of captured images on the basis of the positional relationships between the own vehicle and the objects and selecting the viewpoint position of a display image including the target object on the basis of the positional relationship between the host vehicle and the target object; and a display image generation unit for generating and outputting a display image to be displayed on an image display unit provided in the host vehicle, on the basis of the multi-viewpoint image and the viewpoint position. Thus, the peripheral image of a vehicle can be presented with a more accurately expressed sense of distance between the vehicle and the object.

Description

Indicates an image generation device

[0001] The present invention relates to a display image generation device.

[0002] As a technology used in parking support and the like, there is a technology that generates an overhead image of the vehicle periphery by performing a perspective transformation on a two-dimensional image of the vehicle periphery obtained by an imaging means and presents it to the driver. For example, the one described in Patent Document 1 is known.

[0003] Patent Document 1 includes a plurality of cameras that acquire image data of the periphery in a self-propelled movable body, a 3D calculation unit that calculates first 3D image data based on the image data, and a coordinate system of the first 3D image data. A coordinate conversion unit that converts the coordinate system to a vehicle coordinate system that looks down from a virtual camera virtually arranged outside the vehicle to obtain second 3D image data, and an integration unit that generates integrated data by integrating the second 3D image data and the data of the 3D model of the vehicle. A viewpoint position changing unit that changes the viewpoint position of the virtual camera, and a virtual camera attitude determination unit that determines the attitude of the virtual camera whose viewpoint position has been changed. The virtual camera attitude determination unit uses a virtual camera line-of-sight direction vector set as a vector connecting the changed viewpoint position and a fixation point that the virtual camera should fixate on when changing the viewpoint position of the virtual camera, and a virtual camera upward direction vector that is a reference for the direction of the virtual camera to determine the attitude of the virtual camera. The integration unit generates integrated data based on the attitude of the virtual camera and the changed viewpoint position. A peripheral display device is disclosed.

[0004] Japanese Patent Application Laid-Open No. 2011-004201

[0005] However, in the above prior art, for example, since an overhead image generated by performing an overhead conversion on an image of a fisheye camera (fisheye image) is presented as an image of the vehicle periphery, the relative positional relationship of tall objects is not accurately reflected, and the overhead image is different from the actual sense of distance between the vehicle and the object.

[0006] The present invention has been made in view of the above, and an object thereof is to provide a display image generation device that can present a peripheral image of a vehicle that more accurately represents the sense of distance between the vehicle and an object.

[0007] The present invention includes multiple means for solving the above problems, but to give one example, a display image generation device that generates a display image that includes at least a part of the surroundings of the vehicle in its display range, comprising: an image acquisition unit that acquires a plurality of captured images of the surroundings of the vehicle captured by a plurality of imaging devices provided on the vehicle; a multi-view image generation unit that generates a multi-view image from the plurality of captured images by viewpoint conversion based on the installation information of the plurality of imaging devices on the vehicle and the calculation result of the parallax of the plurality of captured images; a target object acquisition unit that selects a target object from a plurality of objects included in the plurality of captured images based on the positional relationship with the vehicle, and selects a viewpoint position of the display image including the target object based on the positional relationship between the vehicle and the target object; and a display image generation unit that generates and outputs a display image to be displayed on an image display unit provided on the vehicle based on the multi-view image and the viewpoint position.

[0008] According to the present invention, it is possible to present a vehicle's surrounding image that more accurately represents the sense of distance between the vehicle and the target.

[0009] This is a functional block diagram that schematically shows the overall configuration of the display image generation device. This is a diagram that schematically shows the appearance of a vehicle (the user's own vehicle) as an example of application of the display image generation device. This is a diagram that shows an example of a display image shown on the display device. This is a diagram that shows the situation of the user's own vehicle passing another vehicle. This is a diagram that shows the situation of obstacles being present around the user's own vehicle. This is a diagram that shows the situation of a gutter being present around the user's own vehicle. This is a diagram that shows an example of the display screen. This is a diagram that schematically shows the outlines and surfaces of objects that are expected to be recognized by the user, extracted from the overhead view shown in Figure 8.

[0010] Embodiments of the present invention will be described below with reference to the drawings.

[0011] Figure 1 is a schematic functional block diagram showing the overall configuration of the display image generation device according to this embodiment. Figure 2 is a schematic diagram showing the external appearance of a vehicle (the vehicle itself) as an example of application of the display image generation device.

[0012] In Figure 1, the display image generation device 100 generates a display image that includes at least a portion of the area around the vehicle 200 within its display range. The device includes an image acquisition unit 110 that acquires multiple captured images of the area around the vehicle 200 captured by a plurality of (for example, six) imaging devices 211, 212, 213, 214, 215, 216 installed on the vehicle 200, and a viewpoint transformation based on the installation information (camera installation information) of the plurality of imaging devices 211, ..., 216 on the vehicle 200 and the parallax calculation results of the plurality of captured images, and The system includes a multi-view image generation unit 120 that generates a multi-view image from captured images, a target object acquisition unit 130 that selects a target object from multiple objects included in multiple captured images based on its positional relationship with the vehicle 200, and selects the viewpoint position of a display image including the target object based on the positional relationship between the vehicle 200 and the target object, and a display image generation unit 140 that generates and outputs a display image to be displayed on a display device 300 (image display unit) provided on the vehicle 200 based on the multi-view image and the viewpoint position. In Figures 1 and 2, the case in which six imaging devices 211, ..., 216 are arranged on the vehicle 200 is illustrated as an example, but the system is not limited to this, and a configuration with more or fewer than six imaging devices is also possible. Furthermore, in the following description, the multiple imaging devices 211, ..., 216 may be collectively referred to as imaging device 210.

[0013] The image acquisition unit 110 includes a camera input unit 111 and an image storage unit 112.

[0014] The camera input unit 111 functions as an interface between the display image generation device 100 and the imaging device 210, acquiring images captured by the imaging device 210 and sending them to the image storage unit 112.

[0015] The multiple imaging devices 211, ..., 216 (imaging device 210) capture images of the area around the vehicle 200, and are, for example, cameras using image sensors such as CMOS or CCD. Note that the multiple imaging devices 211, ..., 216 are not limited to monocular cameras; for example, some may be compound cameras with two or more cameras, such as a stereo camera.

[0016] The image storage unit 112 temporarily stores the images acquired from the imaging device 210 by the camera input unit 111 and sends them to the multi-view image generation unit 120.

[0017] The multi-view image generation unit 120 includes a parallax generation unit 121, a point cloud integration unit 122, and a camera installation information storage unit 123.

[0018] The camera installation information storage unit 123 stores information such as the installation position and imaging direction of each of the multiple imaging devices 211, ..., 216 as camera installation information. The camera installation information is expressed, for example, in a coordinate system (vehicle body coordinate system) that is fixed relative to the vehicle 200.

[0019] Figure 2 shows the vehicle 200 with the right side of the figure as the front, and illustrates an example of the installation position and field of view of each imaging device 210 on the vehicle 200. The imaging devices 210 provided on the vehicle 200 consist of, for example, an imaging device 211 installed at the front end of the vehicle body facing forward to image the front of the vehicle body, an imaging device 212 consisting of cameras installed below the left and right side mirrors facing downward and to the side to image the area around the front wheels on the lower side of the vehicle body, an imaging device 213 consisting of multiple cameras installed above (and in the center left and right) the windshield to image the front of the vehicle body, an imaging device 214 installed on the top of the vehicle body facing rearward and to the side to image the entire rear view of the vehicle body including the side, an imaging device 215 installed on the top of the vehicle body facing forward and to the side to image the entire front view of the vehicle body including the side, and an imaging device 216 installed at the rear end of the vehicle body facing rearward to image the rear of the vehicle body. Each camera is configured to perform stereo processing and parallax generation on the overlapping regions of cameras whose imaging areas overlap, thereby enabling the calculation of the distance between feature points in each image.

[0020] The parallax generation unit 121 generates a multi-viewpoint image from multiple captured images by performing a viewpoint transformation based on the installation information (camera installation information) of the multiple imaging devices 211, ..., 216 on the vehicle 200 and the results of the parallax calculation of the multiple captured images. The camera installation information is information indicating the respective mounting positions, mounting angles, and field of view of the imaging devices 211, ..., 216 in the vehicle coordinate system set for the vehicle 200, and is stored in the camera installation information storage unit 123.

[0021] The point cloud integration unit 122 integrates the point cloud data into the multi-view image generated by the disparity generation unit 121, and outputs the multi-view image with integrated point cloud data to the display image generation unit 140.

[0022] The target acquisition unit 130 includes a monocular recognition processing unit 131, a target target setting unit 132, and a target distance notification viewpoint setting unit 133.

[0023] The monocular recognition processing unit 131 recognizes one or more targets present around the vehicle 200 from multiple images captured by multiple imaging devices 211, ..., 216. Various methods can be considered for target recognition, such as recognition by acquiring depth information from images, recognition by pattern matching of specific objects, and recognition by machine learning (including AI). The targets to be recognized can be a wide variety of things present in the environment surrounding the vehicle 200, including people and other living things passing outside the vehicle, road shapes including gutters, potholes, and road markings such as white lines, structures such as building walls, utility poles, and road signs, other moving objects (vehicles) including bicycles and other vehicles, and objects placed on the ground or fallen objects. Recognition of predetermined targets is performed as needed.

[0024] The target object setting unit 132 sets one or more specific objects as target objects from among the objects recognized by the monocular recognition processing unit 131. Objects set as target objects are, for example, objects that may affect the operation and movement of the vehicle 200, such as objects that are closest to the vehicle body of the vehicle 200, objects in positions where visibility is poor due to the driver's line of sight (such as blind spots), objects that are in the direction of movement of the vehicle 200 as determined from the steering angle of the wheels and the operation status of the shift lever, and objects that are approaching the vehicle 200.

[0025] The target distance notification viewpoint setting unit 133 sets the viewpoint from which understanding the positional relationship (distance and direction) and situation between the target object set by the target object setting unit 132 and the vehicle body of the vehicle 200 should be given higher priority as the target distance notification viewpoint (viewpoint position). A viewpoint from which understanding the situation should be given higher priority is, for example, a viewpoint from which it is possible to more reliably understand the positional relationship and situation between the vehicle 200 and a target object that is considered to have a higher probability of contact with the vehicle 200 based on the positional relationship, direction of movement, and speed of movement between the vehicle 200 and the target object.

[0026] The target distance notification viewpoint can be set by, for example, preparing in advance multiple viewpoints within the range that can be represented by the multi-view image, and multiple examples of the positional relationship between the target object and the vehicle 200 (positional relationship examples). At least one of the multiple viewpoints is then associated with each of the positional relationship examples and set up as a table. By using this table, one or more viewpoints can be selectively set as the target distance notification viewpoint from the positional relationship examples. Alternatively, the target distance notification viewpoint may be dynamically set from the multi-view image according to the operating state of the vehicle 200 (steering angle and speed).

[0027] The display image generation unit 140 generates a display image for each target distance notification viewpoint (viewpoint position) based on the multi-view image generated by the multi-view image generation unit 120 and one or more target distance notification viewpoints (viewpoint positions) set by the target target acquisition unit 130, and outputs it to the display device 300.

[0028] The display device 300 is, for example, a monitor installed so that it can be seen by the driver sitting in the driver's seat of the vehicle, as one of the devices for ensuring a field of view, and it presents the driver with an image of the area around the vehicle 200 output from the display image generation unit 140 of the display image generation device 100.

[0029] The effects and advantages of this embodiment, configured as described above, will now be explained.

[0030] Figure 3 shows an example of a display image shown on a display device in the prior art.

[0031] In the conventional technology shown in Figure 3, an example is illustrated of a vehicle with the driver's seat (including the steering wheel and pedals) located on the right side, making a left turn at an intersection. In this case, for example, display image 410 is generated as display image 400, showing the positional relationship between the vehicle 2000 and the target object from a viewpoint above and to the left rear (left rear view), and display image 420 is generated from a viewpoint above (top view). Markers 411, 412, 421, and 422 are also placed along the position that the vehicle 2000 will pass through (including a certain margin).

[0032] However, in a display device like the conventional technology described above, which presents an overhead image generated by converting a fisheye camera image (fisheye image) to an overhead view as an image of the vehicle's surroundings, the relative positional relationship of tall objects is not accurately reflected, resulting in an overhead image that does not reflect the actual sense of distance between the vehicle and the object.

[0033] Therefore, in this embodiment, a display image generation device that generates a display image that includes at least a part of the area around the vehicle within its display range acquires multiple captured images of the area around the vehicle captured by multiple imaging devices installed on the vehicle, generates a multi-view image from the multiple captured images by viewpoint transformation based on the installation information of the multiple imaging devices on the vehicle and the calculation result of the parallax of the multiple captured images, selects a target target from the multiple targets included in the multiple captured images based on the positional relationship with the vehicle, selects a viewpoint position for the display image including the target target based on the positional relationship between the vehicle and the target target, and generates and outputs a display image to be displayed on the image display unit installed on the vehicle based on the multi-view image and the viewpoint position. As a result, it is possible to present a vehicle surrounding image that more accurately represents the sense of distance between the vehicle and the target.

[0034] Figures 4 to 8 show examples of driving conditions in the vehicle.

[0035] Figure 4 shows the situation when one vehicle passes another vehicle.

[0036] Figure 4 shows a scenario where two vehicles traveling in different directions pass each other on a relatively narrow roadway. In this case, the display image generation device 100 sets the other vehicle 500 as a target object and generates display images from an overhead view and a (right) rear view that make it easy to understand the positional relationship between the own vehicle 200 and the other vehicle 500, and displays them on the display device 300 for the driver to see. In conventional technology, overhead images obtained by converting a fisheye image to an overhead view result in a shift in the relative positional relationship between the object and the own vehicle, resulting in an overhead image that does not reflect the actual sense of distance. In contrast, the present invention can present a display image of the area around the vehicle that more accurately represents the sense of distance between the own vehicle and the object.

[0037] Figure 5 shows a situation where obstacles are present around the vehicle.

[0038] Figure 5 shows the case where obstacles such as a mailbox or utility pole are present around the vehicle 200. In this case, the display image generation device 100 sets the mailbox or utility pole as a target object and generates display images from an overhead view and a forward view that make it easy to understand the positional relationship between the vehicle 200 and the obstacle, and displays them on the display device 300 for the driver to see. In addition to mailboxes and utility poles, other possible obstacles include moving objects such as pedestrians and bicycles, and installed objects such as signs, fences, and walls.

[0039] Figure 6 shows a situation where a drainage ditch is present around the vehicle.

[0040] Figure 6 shows the case where there is a gutter running along the road to the left of the vehicle 200. In this case, the display image generation device 100 sets the gutter as a target object and generates display images from an overhead view and a forward view that make it easy to understand the positional relationship between the vehicle 200 and the gutter, and displays them on the display device 300 for the driver to see.

[0041] Figure 7 shows a situation where an obstacle is present above the vehicle.

[0042] Figure 7 shows a scenario where an obstacle (such as a vehicle height restriction device indicating the height restriction under the girders of an elevated bridge) is located above the vehicle 200, which has a cargo compartment and is tall, in the direction of travel. In this case, the display image generation device 100 sets the above obstacle as a target object and generates a display image from a rearward (above) viewpoint that makes it easy to understand the positional relationship between the vehicle 200 and the obstacle, and displays it on the display device 300 for the driver to see.

[0043] Figure 8 shows an example of the display screen.

[0044] As shown in FIG. 8, for example, on a display device 300 arranged around the driver's seat of the host vehicle 200, a display image 600 generated by the display image generation device 100 is displayed. The display image 600 displayed on the display device 300 in FIG. 8 is composed of, in addition to the camera image 610, an aerial view image 620 constituted by point cloud data, an upper view image 630 (viewpoint A), and a rear view image 640 (viewpoint B). FIG. 9 is a diagram schematically showing the aerial view image shown in FIG. 8 with the outlines and surfaces of objects that are assumed to be recognized by the user (driver) extracted. As shown in FIGS. 8 and 9, it is possible to present, as a display image, a peripheral image of the vehicle that more accurately represents the sense of distance between the host vehicle and the target by means of the aerial view image based on point cloud data.

[0045] <Modification Example> In addition, various modification examples can be considered in the above-described embodiment.

[0046] For example, the behavior information of the host vehicle 200 is estimated based on at least the operation status (speed and steering angle) of the host vehicle 200, the collision risk between the host vehicle 200 and each of a plurality of target objects is calculated according to the estimated behavior information, and the calculated risks are further ranked. Among the ranked plurality of target objects, a warning display may be made on the display image for target objects higher than a predetermined rank. The warning display can be performed, for example, by highlighting the target object in the display image or by superimposing a marker or the like on the target object. By outputting the display image with the warning display to the display device 300, the driver can be prompted to be vigilant about the target object.

[0047] <Supplementary Note> The present invention is not limited to the above-described embodiment, and includes various modification examples and combinations within a range not departing from the gist thereof. Further, the present invention is not limited to those having all the configurations described in the above-described embodiment, and also includes those in which a part of the configuration is deleted. Further, each of the above-described configurations, functions, etc. may be realized by designing a part or all of them, for example, by an integrated circuit. Further, each of the above-described configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function.

[0048] 100...Display image generation device, 110...Image acquisition unit, 111...Camera input unit, 112...Image storage unit, 120...Multi-view image generation unit, 121...Parallax generation unit, 122...Point cloud integration unit, 123...Camera installation information storage unit, 130...Target object acquisition unit, 131...Monocular recognition processing unit, 132...Target object setting unit, 133...Target distance notification viewpoint setting unit, 14 0...Display image generation unit, 200...Own vehicle, 210...Imaging device, 211, 212, 213, 214, 215, 216...Imaging device, 300...Display device, 400, 410, 420...Display image, 411, 412, 421, 422...Marker, 500...Other vehicle, 600...Display image, 610...Camera image, 620...Overhead view, 630...Upper view, 640...Rear view

Claims

1. A display image generation device that generates a display image that includes at least a portion of the area around the vehicle, comprising: an image acquisition unit that acquires a plurality of captured images of the area around the vehicle captured by a plurality of imaging devices provided on the vehicle; a multi-view image generation unit that generates a multi-view image from the plurality of captured images by viewpoint conversion based on the installation information of the plurality of imaging devices on the vehicle and the result of the parallax calculation of the plurality of captured images; a target object acquisition unit that selects a target object from a plurality of objects included in the plurality of captured images based on the positional relationship with the vehicle, and selects a viewpoint position of the display image including the target object based on the positional relationship between the vehicle and the target object; and a display image generation unit that generates and outputs a display image to be displayed on an image display unit provided on the vehicle based on the multi-view image and the viewpoint position.

2. A display image generation device according to claim 1, wherein the target object acquisition unit selects a plurality of viewpoint positions, and the display image generation unit generates a plurality of display images according to the plurality of viewpoint positions selected by the target object acquisition unit.

3. A display image generation device according to claim 1, wherein the target object acquisition unit selects the viewpoint position according to the visibility of the target object due to the driver's line of sight of the vehicle.

4. A display image generation device according to claim 1, wherein the target object acquisition unit selects a plurality of target objects, ranks the collision risk of the vehicle with respect to each of the plurality of target objects according to the behavior information of the vehicle estimated based on at least the speed and steering angle of the vehicle, and the display image generation unit outputs to the image display unit such that a warning is displayed on the display image for target objects with a higher rank than a predetermined rank among the ranked plurality of target objects.

Images