Display system and display method

Abstract

Provided is a display system which, when generating a bird's-eye view image from a plurality of imaging devices disposed on the right and left side faces of a vehicle, can preferentially use a video which allows easy confirmation of the surroundings. A display system (1) according to the present disclosure comprises the plurality of imaging devices, a display device (15) that displays a bird's-eye view image in which the vehicle is virtually viewed from above, and an image generation unit (13). The image generation unit (13) generates a bird's-eye view image on the basis of a video captured by at least one imaging device among videos captured by the plurality of imaging devices. The plurality of imaging devices include two or more side face imaging devices provided on the left side face of the vehicle, or two or more side face imaging devices provided on the right side face of the vehicle. The image generation unit (13) generates a bird's-eye view image by preferentially using a video captured by, among the side face imaging devices, a high-position imaging device installed at a high position when viewed from the ground contact surface of the vehicle.

Description

Display System and Display Method

[0001] The present disclosure relates to a display system and a display method.

[0002] Patent Document 1 describes a technique for generating a bird's-eye view image around an articulated vehicle by combining a bird's-eye view image around a tractor and a bird's-eye view image around a trailer in accordance with the relative angle of a coupling portion between the tractor and the trailer, and displaying the generated bird's-eye view image.

[0003] Japanese Patent Laying-Open No. 2012-105158

[0004] However, with the technique described in Patent Document 1, even when there is an image from among the images captured by a tractor-side camera and a trailer-side camera that allows easy confirmation of the surroundings, a bird's-eye view image is generated only based on the relative angle of the coupling portion.

[0005] Therefore, when generating a bird's-eye view image, that is, an overhead view image, from a plurality of imaging devices arranged on the left and right side surfaces of a vehicle, development of a technique that preferentially uses video that allows easy confirmation of the surroundings is desired.

[0006] A display system according to the present disclosure comprises: a plurality of imaging devices arranged on a vehicle and configured to image the surroundings of the vehicle; a display device configured to display an overhead image obtained by virtually looking down on the vehicle from above; and an image generation unit configured to generate the overhead image based on video captured by at least one of the plurality of imaging devices, wherein the plurality of imaging devices include either two or more side imaging devices provided on the left side surface of the vehicle or two or more side imaging devices provided on the right side surface of the vehicle, and the image generation preferentially uses video captured by a high-position imaging device installed at a higher position as viewed from the ground contact surface of the vehicle among the side imaging devices to generate the overhead image.

[0007] The display method according to this disclosure involves a plurality of imaging devices arranged on a vehicle, which capture images of the area around the vehicle and display an overhead image of the vehicle virtually viewed from above, and an image generation unit which performs an image generation process to generate the overhead image based on the image captured by at least one of the imaging devices among the plurality of imaging devices, and the plurality of imaging devices include either two or more side imaging devices provided on the left side of the vehicle, or two or more side imaging devices provided on the right side of the vehicle, and the image generation process generates the overhead image by preferentially using the image captured by a high-position imaging device, which is installed at a high position relative to the ground contact surface of the vehicle, among the side imaging devices.

[0008] According to this disclosure, when generating an overhead view image from multiple imaging devices positioned on the left and right sides of a vehicle, it becomes possible to prioritize the use of images that make it easier to see the surroundings.

[0009] This is a block diagram illustrating an example configuration of the display system according to Embodiment 1. This is a schematic top view showing an example of a vehicle equipped with the display system of Figure 1. This is a schematic side view showing an example of a vehicle equipped with the display system of Figure 1. This is a schematic diagram illustrating an example of processing of the image generation unit of the display system of Figure 1 when the vehicle in Figures 2 and 3 turns left. This is a schematic diagram illustrating an example of processing of the image generation unit of the display system of Figure 1 when the vehicle in Figures 2 and 3 turns right. This is a flowchart illustrating an example of processing of the image generation unit in the display system according to Embodiment 2. This is a schematic diagram illustrating the synthesis priority when no abnormality is detected in the processing example of Figure 6. This is a schematic diagram showing an example of an image captured by the towing-side imaging device when an abnormality is detected in the processing example of Figure 6. This is a schematic diagram illustrating the synthesis priority when an abnormality is detected in the processing example of Figure 6.

[0010] The following describes embodiments of the invention, but the invention claimed is not limited to these embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential for solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate.

[0011] <Embodiment 1> An example configuration of the display system according to Embodiment 1 will be described using Figures 1 to 3. Figure 1 is a block diagram showing an example configuration of the display system according to Embodiment 1. Figure 2 is a schematic top view showing an example of a vehicle equipped with the display system of Figure 1. Figure 3 is a schematic side view showing an example of a vehicle equipped with the display system of Figure 1.

[0012] The display system according to this embodiment may include a control unit 10, a display device 15, a towing-side imaging device 16, and a towed-side imaging device 17, as illustrated in the display system 1 of Figure 1. The control unit 10 may also include an image acquisition processing unit 11, an angle calculation unit 12, an image generation unit 13, and a display control unit 14. Each component will be described below.

[0013] Display system 1 is a display system mounted on a vehicle 100 that includes a towed vehicle 23 and a towing vehicle 21 connected to the towed vehicle 23, as illustrated in Figures 2 and 3. This vehicle 100 is sometimes referred to as the "towing vehicle" itself, but in the following explanation, in order to distinguish between the towing vehicle and the towed vehicle, it will be referred to as the "vehicle".

[0014] In Figure 2, the towing vehicle 21 and the towed vehicle 23 are connected by a coupling part 22 that rotates around a pivot point J within a range where the towing vehicle 21 and the towed vehicle 23 do not come into contact. Of course, the configuration and shape of the vehicle 100 are not limited to this, and it is acceptable as long as the towing vehicle 21 is connected to the towed vehicle 23. The vehicle 100 can refer to, for example, a large towing truck. The towed vehicle 23 can refer to, for example, the trailer portion of a cargo truck, that is, the cargo bed portion.

[0015] The display system 1 includes a plurality of imaging devices positioned on the towed vehicle 23 and the towing vehicle 21 to capture images of the areas around the towing vehicle 21 and the towed vehicle 23. Each imaging device can be a camera that captures video. The plurality of imaging devices should cover the areas around the towing vehicle 21 and the areas around the towed vehicle 23 as their imaging range, but should be installed to cover at least the left and right sides. The plurality of imaging devices should continuously capture images while the vehicle 100 is in operation, so that the driver can always check the surroundings. The plurality of imaging devices may also capture images only when certain conditions are met, such as when the vehicle 100 is traveling at a speed below a certain level, or when the shift lever of the vehicle 100 (not shown) is in the reverse position.

[0016] The above-mentioned plurality of imaging devices include one or more side imaging devices provided on the left and right sides of the towing vehicle 21, and one or more side imaging devices provided on the left and right sides of the towed vehicle 23. Here, the imaging devices on the towing vehicle 21 side will be described as towing-side imaging devices 16, and the imaging devices on the towed vehicle 23 side will be described as towed-side imaging devices 17. At least one of each of the towing-side imaging devices 16 and towed-side imaging devices 17 will be installed on the towing vehicle 21 and the towed vehicle 23, respectively.

[0017] Furthermore, by using cameras equipped with wide-angle lenses such as fisheye lenses, the above-mentioned multiple imaging devices, particularly the towing-side imaging device 16 and the towed-side imaging device 17, can acquire wide-angle images and cover the area around the vehicle 100.

[0018] In the following examples, as illustrated in Figures 2 and 3, the towing side imaging device 16 includes side imaging devices 16L and 16R installed on the left and right sides, for example, on the side mirrors, as well as a front imaging device 16F for forward imaging. The side imaging device 16L is the left side imaging device and is installed, for example, on the left side mirror, at a position above the left front wheel 24L1. The side imaging device 16R is the right side imaging device and is installed, for example, on the right side mirror, at a position above the right front wheel 24R1.

[0019] Furthermore, the following example describes a towed vehicle imaging device 17 that includes side imaging devices 17L and 17R installed at lower positions on the left and right sides, as well as a rear imaging device 17B for rearward imaging. The side imaging device 17L is the left side imaging device and is installed, for example, on the chassis between the front left rear wheel 24L2 and the rear left wheel 24L3 of the two left rear wheels. The side imaging device 17R is the right side imaging device and is installed, for example, on the chassis between the front right rear wheel 24R2 and the rear left wheel 24R3 of the two right rear wheels.

[0020] The display system 1 may include information processing devices such as a navigation system, a mobile phone, or a smartphone. This information processing device may be referred to as a computer, or may include a computer. The display device 15 is a display device installed in the driver's seat of the towing vehicle 21. Therefore, the display device 15 may refer to a display device installed in this information processing device. For example, the display device 15 may be a display device embedded in the rearview mirror installed in the driver's seat, or a display device embedded in the windshield. Specifically, the display device 15 is a display audio, infotainment system, or center information display installed in the driver's seat. The display device 15 does not have to be installed in the vehicle 100. For example, the display device 15 may be installed in a location where there is a person who remotely operates the vehicle 100.

[0021] The control unit 10 controls the entire display system 1. The control unit 10 is responsible for functions such as acquiring video, generating display images, and display control in the display system 1. The control unit 10 may be implemented by, for example, a processor such as a CPU (Central Processing Unit), working memory, and a non-volatile storage device that stores a control program. This program may include a program that, when executed by the processor, realizes the functions of the display system 1. The control unit 10 may also be implemented using a programmable integrated circuit, such as an FPGA (field-programmable gate array) or a microcomputer.

[0022] Furthermore, a separate storage device, distinct from the above-mentioned storage device or control unit 10, stores various information used by the display system 1. The storage device is, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive). This storage device stores various setting information and also temporarily stores images captured by the towing-side imaging device 16 and the towed-side imaging device 17, as well as generated images.

[0023] The display device 15 displays an overhead view image of the towing vehicle 21 and the towed vehicle 23, virtually viewed from above. In the example shown in Figure 1, the display control unit 14 is responsible for controlling the display of the display device 15. The display device 15 may be composed of a display device such as a liquid crystal panel or an organic electroluminescent panel. The display control unit 14 controls the display device 15 to display the overhead view image generated by the image generation unit 13.

[0024] The image generation unit 13 generates an overhead view image based on the video captured by at least one of the imaging devices, which is one of the images captured by the towing imaging device 16 and the towed imaging device 17. The source video is acquired by the image acquisition processing unit 11 from the towing imaging device 16 and the towed imaging device 17. The image acquisition processing unit 11 then passes the acquired video to the image generation unit 13. Since the overhead view image is generated from the images shown in the video, such as the images of each frame, the generated overhead view image can be said to be an overhead view video.

[0025] The generated overhead images may be overhead images of only one or both sides, or they may include overhead images of at least one of the front and rear views. Depending on the scene, the overhead images may be generated from images captured by a single imaging device, or multiple types of images may be generated, such as left side, right side, front, and rear views.

[0026] The image generation unit 13 generates an overhead view image by prioritizing the use of video captured by the high-position imaging device, which is installed at a higher position relative to the ground contact surface of the vehicle 100, among the towing-side imaging device 16 and the towed-side imaging device 17.

[0027] In the example shown in Figure 3, the high-position imaging device is the side imaging device 16L when it is on the left side, and similarly, the side imaging device 16R when it is on the right side. Hereinafter, the side imaging image 16L will be referred to as the high-position imaging image 16L, and the side imaging device 16R will be referred to as the high-position imaging device 16R. On the other hand, the side imaging devices 17L and 17R are installed at a lower position than the high-position imaging devices 16L and 16R, for example, at the same height as the wheels.

[0028] In this way, the image generation unit 13 selects the video from among the multiple towing-side imaging devices 16 and towed-side imaging devices 17, prioritizing the video from the device with better imaging conditions, i.e., the one with a higher ground clearance, and generates the overhead view image.

[0029] Furthermore, the image generation unit 13 may generate an overhead view image by prioritizing the use of images captured by the high-position imaging device among the towing-side imaging device 16 and the towed-side imaging device 17, based on the connection angle between the towing vehicle 21 and the towed vehicle 23 in a plane parallel to the ground contact surface. This connection angle can be calculated by the angle calculation unit 12. In other words, the angle calculation unit 12 calculates the connection angle between the towing vehicle 21 and the towed vehicle 23 in a plane parallel to the ground contact surface.

[0030] The angle calculation unit 12 may calculate the coupling angle based on images captured by at least one of the multiple imaging devices. The angle calculation unit 12 determines how the towed vehicle 23 appears from the left and right images captured on the towing vehicle 21 side, and how the towing vehicle 21 appears from the left and right images captured on the towed vehicle 23 side. Based on these determination results, the angle calculation unit 12 calculates the angle between the towing vehicle 21 and the towed vehicle 23, that is, the coupling angle, which is the angle in which direction.

[0031] The angle calculation unit 12, although not shown, includes an analysis unit that analyzes the video acquired by the image acquisition processing unit 11 for the purpose of calculating such connected angles, or the control unit 10 may include such an analysis unit. The video analysis can be performed, for example, for each frame image. This analysis unit may also be configured to determine obstacles and other objects from the video in which the surrounding environment has been captured. The determination results can be displayed, for example, superimposed on an overhead view image.

[0032] However, the angle calculation unit 12 is not an essential component of the display system 1. For example, the coupling angle may be measured by a sensor provided on the coupling unit 22, or by a sensor provided on the towed vehicle 23 side of the towing vehicle 21, or by a sensor provided on the towing vehicle 21 side of the towed vehicle 23.

[0033] Next, using Figures 4 and 5, we will explain examples of processing by the image generation unit 13 when the vehicle 100 turns left and right, respectively. Figure 4 is a schematic diagram illustrating an example of processing by the image generation unit 13 when the vehicle 100 in Figures 2 and 3 turns left. Figure 5 is a schematic diagram illustrating an example of processing by the image generation unit 13 when the vehicle 100 in Figures 2 and 3 turns right.

[0034] First, using Figures 4 and 5, we will briefly explain the process of adjusting the image cropping position when generating an overhead view image. The image acquisition processing unit 11 acquires images captured by the front imaging device 16F and the rear imaging device 17B as the front image and rear image, respectively, and acquires images captured by the side imaging devices 16L, 16R, 17L, and 17R as the side image, and passes them to the image generation unit 13. The image acquisition processing unit 11 also passes the acquired images to the angle calculation unit 12.

[0035] In Figures 4 and 5, the imaging regions captured by the forward imaging device 16F, the rear imaging device 17B, and the side imaging devices 16L, 16R, 17L, and 17R are, for example, regions SF, SB, SL1, SR1, SL2, and SLR, respectively. Note that in Figure 4, for convenience, the entire region SR2 is not shown, but in reality it includes region SRD, which will be described later. Similarly, in Figure 5, for convenience, the entire region SL2 is not shown, but in reality it includes region SLD, which will be described later.

[0036] The image generation unit 13 synthesizes these images to generate an overhead view image to be displayed on the display device 15, but uses the linking angle θ to adjust the cropping position for synthesis. The angle calculation unit 12 calculates the linking angle θ from these images received from the image acquisition processing unit 11 and passes it to the image generation unit 13. Based on the received linking angle θ, the image generation unit 13 adjusts the cropping position of the unnecessary overlapping parts of the images in regions SF, SB, SL1, SR1, SL2, and SLR, that is, the cropping position of the necessary parts, based on geometric calculations. Then, the image generation unit 13 crops the unnecessary parts of the images at the adjusted positions and synthesizes the remaining images. Existing techniques can be used for the synthesis method.

[0037] The image generation unit 13 can use techniques such as camera calibration processing, projection transformation processing, image stitching processing, and blending processing as synthesis methods. Camera calibration processing is a process that grasps the position and angle of each imaging device and corrects distortion when generating an overhead view image to be displayed on the display device 15. Projection transformation processing is a process that converts the remaining image after cutting out the unnecessary parts of the image area acquired from each imaging device into a plane. Image stitching processing is a process that connects the remaining images after cutting out the unnecessary parts of the image area acquired from each imaging device, and performs geometric correction and optimization of overlapping areas. Blending processing is a process that smoothly connects the boundaries of each image area.

[0038] As shown in Figure 4, when vehicle 100 turns to the left, the proportion of images of the towed vehicle 23 and the towing vehicle 21 that are captured by the two side imaging devices 16L and 17L installed on the left side of vehicle 100 increases. Therefore, in this case, the image generation unit 13 adjusts the cropping angle of the left side of vehicle 100 to be acute. Here, adjusting the cropping angle to be acute means cropping the side of region SL1 that shows the towed vehicle 23 so that it becomes more acute, and cropping the side of region SL2 that shows the towing vehicle 21 so that it becomes more acute.

[0039] Similarly, when vehicle 100 turns to the left, the image generation unit 13 adjusts the cropping angle on the right side of vehicle 100 to be obtuse, because the towed vehicle 23 and the towing vehicle 21 will no longer be visible in the two side imaging devices 16R and 17R installed on the right side of vehicle 100, respectively. Here, adjusting the cropping angle to be acute means widening region SR1 to an obtuse angle as much as possible while using its image as is, and widening region SR2 to an obtuse angle as much as possible while using its image as is. In either case, the coupling angle θ is calculated based on the images captured by the side imaging devices 16L and 17L.

[0040] Furthermore, although the explanation will be omitted, as illustrated in Figure 5, even when turning to the right, the image generation unit 13 can perform adjustments using a similar process, except that the coupling angle θ is calculated based on the images captured by the side imaging devices 16R and 17R.

[0041] The image generation unit 13 basically generates an overhead image by cutting out each video based on the cropping position adjusted as described above, but in this embodiment, it generates the overhead image by appropriately using a side imaging device with good imaging conditions. In other words, when generating an overhead image, the image generation unit 13 may preferentially use the high-position side imaging devices 16L and 16R over the side imaging devices 17L and 17R, respectively.

[0042] To explain in more detail, the image generation unit 13 generates an overhead view image in the following manner when the coupling angle θ is not horizontal, that is, when the towed vehicle 23 and the towing vehicle 21 are not in a straight line.

[0043] As illustrated in Figure 4 on the left side when turning left, the image generation unit 13 generates an overhead image based only on the video captured by the high-position imaging device 16L on the side where the connection angle θ is smaller than horizontal.

[0044] On the other hand, as exemplified on the right side when turning left in FIG. 4, the image generating unit 13 generates a bird's-eye view image as follows on the side where the coupling angle θ is larger than the horizontal direction. That is, the image generating unit 13 generates the bird's-eye view image based on the image captured by the high-position imaging device 16R and the image corresponding to the blind spot area of the high-position imaging device 16R among images captured by a side imaging device 17R other than the high-position imaging device 16R. The blind spot area in FIG. 4 is the portion illustrated as area SRD.

[0045] In fact, as exemplified in FIG. 3, the installation positions of the imaging devices may have different distances from the ground contact surface due to constraints such as the physical constraints of the vehicle 100. For example, the side imaging devices 16L and 16R are installed near the side mirror at, for example, 2 m above the ground, and the side imaging devices 17L and 17R are installed at a position about the height of the wheels on the chassis portion of the towed vehicle 23, for example, at several tens of centimeters above the ground. As in the present embodiment, a wide-angle lens such as a fisheye lens having a horizontal angle of view of 180 degrees or more may be used for an imaging device that is used for capturing an image for displaying a bird's-eye view image.

[0046] When the installation position of a side imaging device provided with such a wide-angle lens is low, that is, in the case of the side imaging devices 17L and 17R, there is an increased possibility that nearby objects such as the ground and surrounding objects are captured in an extremely large size. When the distance between the subject and the imaging device is extremely short as described above, only a part of the subject may be captured. Conversely, when the installation position of a side imaging device provided with a wide-angle lens is high, that is, in the case of the side imaging devices 16L and 16R, the distance between the imaging device and subjects such as nearby objects including the ground and surrounding objects becomes appropriate, so that the entire information within the angle of view can be captured in a well-balanced manner. Therefore, compared with the side imaging devices 16L and 16R which have a higher ground clearance for surrounding imaging, the side imaging devices 17L and 17R perform imaging under unfavorable imaging conditions, that is, bad imaging conditions.

[0047] Therefore, as described above, when the connecting angle θ is smaller than horizontal, the image generation unit 13 uses only the image of the area SL1 captured under favorable imaging conditions, from among the area SL2 captured by the side imaging device 17L and the area SL1 captured by the high-position imaging device 16L. That is, in this case, the image generation unit 13 generates a bird's-eye view image based only on the image captured by the high-position imaging device 16L.

[0048] Further, as described above, when the connecting angle θ is larger than horizontal, the image generation unit 13 uses only the image of the area SL1 captured under favorable imaging conditions for the overlapping portion, from among the area SR2 captured by the side imaging device 17R and the area SR1 captured by the high-position imaging device 16R. That is, in this case, the image generation unit 13 generates a bird's-eye view image by preferentially using the image captured by the high-position imaging device 16L for the overlapping portion. Here, since the non-overlapping area SRD is a blind spot for the high-position imaging device 16R, the image generation unit 13 generates the bird's-eye view image using the image captured by the side imaging device 17R.

[0049] The above description refers to the case of a left turn, and basically the same approach is applied to generate a bird's-eye view image when turning right as well. That is, as exemplified on the right side when turning right in FIG. 5, when the connecting angle θ is smaller than horizontal, the image generation unit 13 generates a bird's-eye view image based only on the image captured by the high-position imaging device 16R, that is, based only on the image of the area SR1.

[0050] On the other hand, as exemplified on the left side when turning right in FIG. 5, when the connecting angle θ is larger than horizontal, the image generation unit 13 generates the bird's-eye view image as follows. That is, the image generation unit 13 generates the bird's-eye view image based on the image captured by the high-position imaging device 16L and the image captured by the side imaging device 17L that corresponds to the blind spot area of the high-position imaging device 16L. The blind spot area in FIG. 5 is the portion illustrated as area SLD.

[0051] The above explanation has only covered the images of the left and right sides. In practice, as described above, the image generation unit 13 can use the images captured by the front imaging device 16F and the rear imaging device 17B, respectively, for the front and rear views, and combine them with the images of both sides as described above to generate an overhead view image to be displayed on the display device 15.

[0052] Furthermore, when the connection angle θ is horizontal, as described above, the image generation unit 13 may generate an overhead image using, for example, the left side, the video captured by the side imaging device 16L and the video captured by the side imaging device 17L, and using one of the videos for the overlapping portion. However, even in this case, the video captured by the high-position imaging device 16L, which is located at a higher position, may be used preferentially, and the overhead image may be generated using this video for the overlapping portion.

[0053] Similarly, when the linking angle θ is horizontal, the image generation unit 13 may, for example, in the case of the right side, use the video captured by the side imaging device 16R and the video captured by the side imaging device 17R, and use one of the videos for the overlapping portion to generate an overhead view image. However, even in this case, the video captured by the high-position imaging device 16R, which is located at a higher position, may be used preferentially, and the overhead view image may be generated using this video for the overlapping portion.

[0054] In the above example, the explanation assumed that two side imaging devices were provided on each of the left and right sides. However, even if three or more side imaging devices are provided on each of the left and right sides, the priority of the overlapping portion of the image can be determined using the same approach. In the above example, the explanation assumed that there was one towed vehicle. However, it is not limited to this, and there may be two or more towed vehicles. In this case, one or more side imaging devices may be provided on each of the left and right sides of multiple towed vehicles. In the above example, the explanation assumed that the vehicle consisted of a towed vehicle and a towing vehicle. However, it is not limited to this, and it is sufficient to have two or more side imaging devices on the left and right sides of the vehicle. For example, in the case of a long vehicle such as a bus, it is sufficient to have two or more side imaging devices on the left side, or two or more side imaging devices on the right side. In this case, since the coupling angle θ is always horizontal, it is not necessarily required to have an angle calculation unit.

[0055] As described above, in this embodiment, the overlap state between imaging regions changes as the positional relationship between the side imaging devices changes due to the state of the towing vehicle 21 and the towed vehicle 23. Therefore, the image is extracted according to the change in this overlap state, i.e., the overlapping portion. In this embodiment, the quality of the generated overhead image can be improved by prioritizing the selection and use of the image from the side imaging devices with overlapping imaging regions that has better imaging conditions, such as a higher installation position.

[0056] In reality, there are limitations on where the imaging device can be installed in vehicle 100. For example, an imaging device near the side mirror has a higher ground clearance, better visibility, less distortion, and better imaging conditions compared to an imaging device near the wheels. Therefore, in this embodiment, the quality of the displayed overhead image can be improved by prioritizing the use of images with such good imaging conditions to generate an overhead image. In particular, in the case of vehicles such as container trailers, where the container on top of the towed vehicle 23 is frequently replaced, it is difficult to install the imaging device on the container due to wiring and other factors. Therefore, the imaging device must be placed on the chassis of the towed vehicle, which is lower than the container. Even in such vehicles, the quality of the overhead image can be improved.

[0057] In other words, in this embodiment, when generating an overhead view image from multiple imaging devices located on the left and right sides of the vehicle 100, specifically the towed vehicle 23 and the towing vehicle 21, it is possible to prioritize the use of images that allow the driver, passengers, and those remotely viewing the overhead view image, as well as the vehicle-mounted obstacle detection system, to easily see the surroundings, thereby improving the quality of the overhead view image.

[0058] Furthermore, around-view cameras are equipped on many vehicles, including automobiles, and are used as an aid during parking. There are also systems that display around-view views for vehicles equipped with towed vehicles, such as large towing trucks. However, in vehicles currently on the market, even for vehicles equipped with towed vehicles, an overhead view image is generated from side-view images captured by, for example, four cameras, and this overhead view image is superimposed onto an object representing the vehicle, which consists of a roughly rectangular image.

[0059] In this embodiment, the objects representing such vehicles may be objects that separately represent the towing vehicle 21 and the towed vehicle 23, or objects that separately represent the coupling part 22, the towing vehicle 21, and the towed vehicle 23. In this embodiment, the generated overhead image may be superimposed on these objects, or an overhead image may be generated including these objects, and displayed on the display device 15. As a result, in this embodiment, even with a vehicle 100 equipped with a towed vehicle 23, the surrounding environment can be displayed clearly on the display device 15.

[0060] <Embodiment 2> The display system according to Embodiment 2 will be described focusing on the differences from Embodiment 1, but various examples described in Embodiment 1 can be applied, including the main configuration example of the display system 1.

[0061] In Embodiment 1, an example was given in which the side imaging device with the higher ground clearance was prioritized as having better imaging conditions, but other imaging conditions may be added. In this embodiment, a predetermined condition is adopted as the other imaging condition in which the image obstructs the imaging of the surroundings.

[0062] The above-mentioned predetermined conditions refer to conditions other than good imaging conditions based on ground height, that is, conditions other than superior imaging conditions based on ground height, among the conditions for determining priority. For example, the above-mentioned predetermined conditions can be conditions related to nearby obstacles that obstruct imaging by the imaging device, or attached substances such as water droplets, fog, or other deposits. In any example, if the image generation unit 13 finds that the captured image contains an obstruction exceeding a predetermined threshold, it should use the image from the side imaging device that captured that image as a non-priority source to generate an overhead image.

[0063] In other words, the image generation unit 13 in this embodiment determines which imaging device to use preferentially for generating the overhead image, based on whether each of the images captured by the plurality of imaging devices satisfies predetermined conditions that obstruct imaging of the surroundings.

[0064] Furthermore, while it has been assumed up to this point that each imaging device has the same specifications, the specifications of the imaging device may also be a condition as a specified condition. For example, an imaging device with higher resolution or an imaging device with a wider FOV may be given priority. FOV stands for Field of View, which refers to the field of view or the area of ​​view displayed on the screen.

[0065] Furthermore, there is no priority given to which of the above-mentioned predetermined conditions and the conditions related to ground height should be adopted. For example, the image generation unit 13 may set multiple threshold levels for the above-mentioned predetermined conditions and prioritize the conditions related to ground height up to a certain level that does not interfere with imaging, in order to generate an overhead image. Alternatively, if the image generation unit 13 finds that the image captured by the high-position imaging device is an image that satisfies predetermined conditions that would interfere with the imaging of the surroundings by the high-position imaging device, it may prioritize using the image captured by a device other than the high-position imaging device to generate an overhead image.

[0066] An example of processing by the image generation unit 13 will be explained using Figures 6 to 9. Figure 6 is a flowchart illustrating an example of processing by the image generation unit in the display system according to Embodiment 2. Figure 7 is a schematic diagram illustrating the synthesis priority when no abnormality is detected in the processing example of Figure 6. Figure 8 is a schematic diagram showing an example of an image captured by the traction-side imaging device when an abnormality is detected in the processing example of Figure 6. Figure 9 is a schematic diagram illustrating the synthesis priority when an abnormality is detected in the processing example of Figure 6.

[0067] In the following example, when the imaging conditions of the side imaging device are normal, an overhead image is generated using the priority based on the ground height in Embodiment 1 as the initial priority. In this example, the display system 1 monitors for events that change the imaging conditions, such as nearby obstacles or attached objects. Only when such an event is detected does the display system 1 detect an anomaly based on the content and extent of the event, and if an anomaly is detected, it changes the priority of video usage.

[0068] First, each side imaging device takes an image, and the image acquisition processing unit 11 acquires each image (step S11). At this stage, the images may be combined using control based on the initial priority settings to generate an overhead view image, which can then be displayed on the display device 15. In the following explanation, side imaging devices 16L and 17L will be used as examples, but the same applies to side imaging devices 16R and 17R. In this case, as shown in Figure 7, the image generation unit 13 can generate an overhead view image using the image of region SL1 captured by side imaging device 16L for the portion where region SL2 overlaps with region SL1.

[0069] Next, the image acquisition processing unit 11 analyzes the images of each acquired video using an internally provided video analysis unit (not shown) (step S12). This video analysis unit may be included in the analysis unit provided in the angle calculation unit 12.

[0070] The video analysis unit performs analysis based on the contrast and brightness features of the video. The video analysis unit analyzes whether the features described above are abnormal for the imaging state of each side imaging device, based on the video acquired by each side imaging device. In other words, the video analysis unit determines whether it has detected an anomaly in any of the above-mentioned monitoring events as a result of its analysis (step S13).

[0071] If no abnormality is detected (NO in step S13), the video analysis unit waits until an abnormality is detected. When the answer to step S13 is YES, the video analysis unit estimates the content of the abnormal event, i.e., the content of the malfunction (step S14). Of course, in step S13, if a large number of events are detected, a process may be performed to narrow down the detected abnormal events to one event. Note that the processes in steps S13 and S14 may be executed simultaneously.

[0072] The analysis performed by the video analysis unit in steps S12 to S14 can be carried out as rule-based judgment and estimation, or as rule-based AI (Artificial Intelligence) judgment and estimation. The video analysis unit may also perform the analysis based on features indicating the contrast and brightness of the video, or alternatively, features indicating attached substances such as raindrops. Furthermore, the video analysis unit may perform the processing in steps S12 to S14 by AI detection using a machine learning model.

[0073] The analysis performed by the video analysis unit, as described in steps S12 to S14, should be carried out for each video, and preferably include a comparison of the analysis results between the side imaging device 16L and the side imaging device 17L.

[0074] As an example, Figure 8 shows image 16Lim extracted from the video. As shown in Figure 8, if water droplet image Wa is included in image 16Lim due to water droplets adhering to the lens of the side imaging device 16L, the imaging condition deteriorates significantly. In this case, the video analysis unit monitors the video acquired by the side imaging device 16L and, upon detecting this abnormality, estimates that the problem is due to water droplet adhesion. In the following explanation, we will assume that no abnormality was detected in the video captured by the side imaging device 17L, or that even if an abnormality was detected, the degree of the abnormality was lower than that of the side imaging device 16L.

[0075] Following step S14, the image generation unit 13 changes the synthesis priority based on the estimation results described above (step S15). In the example above, in step S15, if the estimation result for the image captured by the side imaging device 16L indicates an abnormality, and the image captured by the side imaging device 17L is normal or the degree of abnormality is lower than that of the side imaging device 16L, the priority is changed.

[0076] The control unit 10 then determines whether or not to terminate the processing of the image generation unit 13 (step S16). For example, the control unit 10 determines to terminate the processing when the vehicle 100 is no longer in operation. If the control unit 10 determines to terminate the processing (YES in step S16), it terminates the processing of the image generation unit 13. If the control unit 10 does not determine to terminate the processing (NO in step S16), it returns the process to step S11.

[0077] Thus, if the image generation unit 13 determines that the image captured by the side imaging device 16L is of lower quality than the image captured by the side imaging device 17L due to the adhesion of water droplets, as shown in the analysis result of image 16Lim, it changes the priority of the side imaging device 16L to a lower level. As a result, the image generation unit 13 uses the image captured by the side imaging device 17L preferentially for the overlapping region of images from the side imaging device 16L and the side imaging device 17L, and synthesizes the images to generate an overhead view image. For example, as shown in Figure 9, the image generation unit 13 can generate an overhead view image using the image of region SL2 captured by the side imaging device 17L for the portion where region SL2 overlaps with region SL1.

[0078] On the other hand, for example, if the imaging conditions of the side imaging device 16L are superior to those of the side imaging device 17L across the entire overlapping imaging area, the image captured by the side imaging device 16L may be used, and the image captured by the side imaging device 17L may not be selected.

[0079] The decision on whether or not to make specific changes will be explained. If the estimation result for the image captured by the side imaging device 16L indicates an abnormality, and there is an abnormality in the image captured by the side imaging device 17L, and the degree of the abnormality is higher or the same as that of the side imaging device 16L, the image generation unit 13 will not change the priority. For example, if the side imaging device 17L has deposits on the lens as well as the side imaging device 16L, and changing the priority would not improve the quality of the overhead image, the image generation unit 13 does not need to change the priority. Also, if the abnormality detection result or estimation result for the image captured by the side imaging device 16L does not detect an abnormality, the priority will not be changed. In this way, the image generation unit 13 does not need to change the priority depending on the imaging conditions of the side imaging device 17L.

[0080] Furthermore, the image generation unit 13 may decide whether or not to change the priority based on multiple imaging conditions, such as changing the priority only when an obstacle nearby that interferes with shooting is captured in the image, or when it detects an abnormality caused by attached objects such as attached substances, water droplets, or fogging.

[0081] In the example described above, we assumed that each of the left and right sides is equipped with two side imaging devices. However, even if each of the left and right sides is equipped with three or more side imaging devices, the same approach can be used to determine the priority of the overlapping images.

[0082] Furthermore, while the above example described a case where the entire FOV of the imaging conditions of the side imaging device 16L changes, it is not limited to this. For example, the video analysis unit may divide the video into predetermined regions and define them, and the image generation unit 13 may determine the priority based on the imaging conditions of each region. This means that, for example, if the imaging area of ​​the part where an obstacle is attached to the lens of the side imaging device 16L does not overlap with the imaging area of ​​the side imaging device 17L, or if changing the priority does not improve the quality of the overhead image, the image generation unit 13 will not change the priority. Also, when the video analysis unit makes a determination based on the imaging conditions of each region, it may adjust the cropping position for combining the overhead images based on the determination result for each region. For example, if the imaging area of ​​the side imaging device 16L where an obstacle is attached to the lens overlaps with the imaging area of ​​the side imaging device 17L, the cropping position is adjusted so that the area SL1 captured by the side imaging device 16L is used for the area where no obstacle is attached to the lens of the side imaging device 16L, and the area SL2 captured by the side imaging device 17L is used for the area where an obstacle is attached to the lens of the side imaging device 16L.

[0083] Furthermore, the attached material may not only be water droplets, but also condensation equivalent to fine water droplets, or other attached materials such as snow, ice, mud, leaves, insects, or dirt. In addition, nearby obstacles may be people, other vehicles, man-made objects, living organisms, or plants. The video analysis unit performs a determination as to whether or not the above predetermined conditions are met, and the image generation unit 13 decides whether or not to change the priority according to the determination result. For example, if the attached material is such as a dry leaf, where the cause of the obstacle can be immediately resolved, the image generation unit 13 does not need to change the priority.

[0084] Furthermore, the timing for switching the composite area in the generated overhead image and reflecting it in the display may be linked to the operation of the vehicle 100's shift lever or steering wheel, or it may be triggered by events such as changes in attached objects or nearby obstacles.

[0085] According to this embodiment, in addition to the effects of Embodiment 1, it is possible to generate overhead images by considering imaging conditions other than ground height, thereby enabling the generation of higher quality overhead images.

[0086] <Alternative Examples, etc.> Some or all of the processing in the display system 1 described above can be implemented as a computer program, as described as a control program. The program described above includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments when loaded into a computer. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited, of the computer-readable medium or physical storage medium include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited, of the temporary computer-readable medium or communication medium include electrical, optical, acoustic or other forms of propagating signals.

[0087] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its spirit.

[0088] This application claims priority based on Japanese Patent Application No. 2025-051009, filed on 26 March 2025, and incorporates all of its disclosures herein.

[0089] This disclosure can be suitably applied to display systems mounted on vehicles and the like.

[0090] 1 Display system 10 Control unit 11 Image acquisition processing unit 12 Angle calculation unit 13 Image generation unit 14 Display control unit 15 Display device 16 Towing side imaging device 17 Towed side imaging device

Claims

1. A display system comprising: a plurality of imaging devices arranged on a vehicle to capture images of the area around the vehicle; a display device that displays an overhead image of the vehicle virtually viewed from above; and an image generation unit that generates the overhead image based on an image captured by at least one of the imaging devices among the images captured by the plurality of imaging devices, wherein the plurality of imaging devices include either two or more side imaging devices provided on the left side of the vehicle, or two or more side imaging devices provided on the right side of the vehicle, and the image generation unit generates the overhead image by preferentially using an image captured by a high-position imaging device among the side imaging devices that is installed at a high position relative to the ground contact surface of the vehicle.

2. The display system according to claim 1, wherein the vehicle comprises a towed vehicle and a towing vehicle connected to the towed vehicle, the plurality of imaging devices include one or more side imaging devices provided on the left and right sides of the towing vehicle and one or more side imaging devices provided on the left and right sides of the towed vehicle, and further comprises an angle calculation unit for calculating the connection angle between the towing vehicle and the towed vehicle in a plane parallel to the ground contact surface, and the image generation unit generates the overhead view image by prioritizing the use of the image captured by the high-position imaging device among the side imaging devices based on the connection angle.

3. The display system according to claim 2, wherein, when the connection angle is not horizontal, the image generation unit generates the overhead view image on the side where the connection angle is greater than horizontal, based on the video captured by the high-position imaging device and the video captured by the side imaging device other than the high-position imaging device that corresponds to the blind spot area of ​​the high-position imaging device, and generates the overhead view image on the side where the connection angle is less than horizontal, based only on the video captured by the high-position imaging device.

4. The display system according to any one of claims 1 to 3, wherein the image generation unit determines which imaging device to use preferentially for generating the overhead image based on whether each of the images captured by the plurality of imaging devices satisfies predetermined conditions that obstruct imaging of the surroundings.

5. A display method comprising: a plurality of imaging devices arranged on a vehicle, which image the area around the vehicle, and display an overhead image of the vehicle virtually viewed from above; an image generation unit which performs an image generation process to generate the overhead image based on the image captured by at least one of the imaging devices among the plurality of imaging devices; the plurality of imaging devices include either two or more side imaging devices provided on the left side of the vehicle, or two or more side imaging devices provided on the right side of the vehicle; and the image generation process which generates the overhead image by preferentially using the image captured by a high-position imaging device among the side imaging devices that is installed at a high position relative to the ground contact surface of the vehicle.

Images