Vehicular image display apparatus, vehicular image display system, vehicular image display method, and vehicular image display program

Abstract

A vehicular image display apparatus 10 comprises: an image acquisition unit 20 that acquires, as an image to be displayed, an image of a vehicle interior; an identification unit 32 that identifies , as a person to be displayed, a person appearing in the image to be displayed; an information acquisition unit 33 that acquires, as person-related information, information related to the person; an image generation unit 35 that executes image processing based on the person-related information acquired in advance by the information acquisition unit on a portion in which the person to be displayed appears within the image to be displayed to generate an alternative image; and a display unit 36 that can display the image to be displayed acquired by the image acquisition unit, and displays the alternative image instead of the image to be displayed under a predetermined environment.

Description

Vehicle Image Display Device, Vehicle Image Display System, Vehicle Image Display Method, and Vehicle Image Display Program ,

[0007] , ,

[0006] Cross - reference to Related Applications

[0001] This application is based on Japanese Application No. 2025 - 004763 filed on January 14, 2025, the contents of which are incorporated herein by reference.

[0002] This disclosure relates to a vehicle image display device, a vehicle image display system, a vehicle image display method, and a vehicle image display program.

[0003] For example, Patent Document 1 discloses a technique of imaging the situation of the rear seat of a vehicle with a camera and displaying the captured image on the display of the front seat. According to this technique, the driver or the passenger in the front passenger seat of the vehicle can check the situation of a passenger such as a child sitting in the rear seat through the front display without directly looking at the rear seat.

[0004] Japanese Unexamined Patent Application Publication No. 2019 - 119416

[0005] In this type of technical field, for example, in a dark environment inside the vehicle such as at night, it is difficult to obtain a clear color image of the rear seat. Therefore, for example, in a dark environment inside the vehicle such as at night, it is considered to display an infrared image instead of a color image. However, even with an infrared image, in a dark environment inside the vehicle, the image quality is close to a monochrome image, so there is a problem that it is difficult to check the expression and situation of the passengers sitting in the rear seat. [[ID=​​​In one embodiment of the present disclosure, a vehicle image display device includes: an image acquisition unit that acquires an image of the interior of a vehicle as a display target image; an identification unit that identifies a person in the display target image as a display target person; an information acquisition unit that acquires information related to a person as person-related information; an image generation unit that generates a substitute image by performing image processing on the portion of the display target image in which the display target person is depicted, based on the person-related information acquired in advance by the information acquisition unit; and a display unit that is capable of displaying the display target image acquired by the image acquisition unit, and which displays the substitute image in place of the display target image under predetermined conditions.

[0008] In one embodiment of the present disclosure, a vehicle image display system comprises an in-vehicle device mounted on a vehicle and an external device capable of data communication with the in-vehicle device, the system comprising: an image acquisition unit that acquires an image of the interior of a vehicle as a display target image; an identification unit that identifies a person in the display target image as a display target person; an information acquisition unit that acquires information related to a person as person-related information; an image generation unit that generates a substitute image by performing image processing on the portion of the display target image in which the display target person is depicted, based on the person-related information acquired in advance by the information acquisition unit; and a display unit that is capable of displaying the display target image acquired by the image acquisition unit, and which displays the substitute image in place of the display target image under predetermined conditions.

[0009] In one aspect of the present disclosure, a vehicle image display method includes: an image acquisition step of acquiring an image of the interior of a vehicle as a display target image; an identification step of identifying a person in the display target image as a display target person; an information acquisition step of acquiring information related to the person as person-related information; an image generation step of generating a substitute image by performing image processing on the portion of the display target image in which the display target person is depicted, based on the person-related information acquired in advance by the information acquisition step; and a display step of displaying the display target image acquired by the image acquisition step, which is displayable and, under a predetermined environment, displays the substitute image instead of the display target image.

[0010] In one aspect of this disclosure, a vehicle image display program causes a control unit of a vehicle image display device that displays an image of the interior of a vehicle to execute: an image acquisition step of acquiring an image of the interior of a vehicle as a display target image; an identification step of identifying a person in the display target image as a display target person; an information acquisition step of acquiring information related to the person as person-related information; an image generation step of generating a substitute image by performing image processing on the portion of the display target image in which the display target person is depicted, based on the person-related information acquired in advance by the information acquisition step; and a display step of displaying the display target image acquired by the image acquisition step, and, under a predetermined environment, displaying the substitute image instead of the display target image.

[0011] According to this disclosure, in a predetermined environment where it is difficult to obtain a clear image of the interior of a vehicle, an alternative image processed by the image generation unit can be displayed. In other words, an image can be displayed in which the portion of the target image containing the target person has been processed based on previously acquired person-related information. This makes it possible to display an image that makes it easy to check the situation inside the vehicle, even in an environment where it is difficult to obtain a clear image of the interior of a vehicle.

[0012] The above-mentioned objectives and other objectives, features and advantages of this disclosure will become clearer from the following detailed description with reference to the attached drawings. The drawings are as follows: Figure 1 is a schematic block diagram showing an example configuration of a vehicle image display device according to the first embodiment of this disclosure; Figure 2 is a schematic flowchart showing an example of a "process for acquiring person-related information in advance" according to the first embodiment of this disclosure; Figure 3 is a schematic flowchart showing an example of a "process for displaying a colorized image as an alternative image" according to the first embodiment of this disclosure; Figure 4 is a schematic diagram showing an example of a data structure for storing person-related information, etc. according to the second embodiment of this disclosure; Figure 5 is a schematic diagram showing an example of a data structure for storing person-related information, etc. according to the third embodiment of this disclosure; Figure 6 is a schematic diagram showing an example of a data structure for storing person-related information, etc. according to the fourth embodiment of this disclosure; and Figure 7 is an example of a control flow according to the fifth embodiment of this disclosure. Figure 13 is a flowchart illustrating the above, Figure 14 is a flowchart illustrating an example of the control flow according to the sixth embodiment of the present disclosure, Figure 8 is a flowchart illustrating an example of the control flow according to the seventh embodiment of the present disclosure, Figure 9 is a flowchart illustrating an example of the control flow according to the seventh embodiment of the present disclosure, Figure 10 is a flowchart illustrating an example of the control flow according to the eighth embodiment of the present disclosure, Figure 11 is a block diagram illustrating an example of the configuration of a vehicle image display device according to the eighth embodiment of the present disclosure, Figure 12 is a block diagram illustrating an example of the configuration of a vehicle image display device according to the ninth embodiment of the present disclosure, Figure 13 is a block diagram illustrating an example of the configuration of a vehicle image display device according to the tenth embodiment of the present disclosure, and Figure 14 is a block diagram illustrating an example of the configuration of a vehicle image display system according to the eleventh embodiment of the present disclosure.

[0013] Hereinafter, several embodiments relating to the vehicle image display device, vehicle image display system, vehicle image display method, and vehicle image display program of this disclosure will be described with reference to the drawings. In addition, substantially identical components in multiple embodiments will be denoted by the same reference numerals, and their descriptions will be omitted.

[0014] (First Embodiment) The vehicle image display device 10 illustrated in Figure 1 is, for example, mounted on a vehicle and includes an imaging device 20, an on-board computer 30, a display 50, and the like.

[0015] The imaging device 20 is an example of an image acquisition unit and is capable of acquiring images of the vehicle's interior as display targets. The imaging device 20 comprises an imaging unit 21 and a light-emitting unit 22. The imaging unit 21 is composed of a sensor or imager capable of detecting visible light and near-infrared light, such as an RGB-IR image sensor, and is capable of acquiring color images and infrared images as display targets. The light-emitting unit 22 is equipped with a light source capable of emitting light with near-infrared wavelengths detectable by the imaging unit 21, for example, wavelengths of 850 to 940 nanometers.

[0016] The imaging device 20 can acquire infrared images by performing imaging processing by the imaging unit 21 while the light-emitting unit 22 emits near-infrared light. Although the infrared images acquired by the imaging device 20 are less clear or more prone to being unclear than color images, it is possible to perform image recognition processing to the extent that the contours and features of the subject in the infrared image are extracted and the subject is identified.

[0017] The imaging device 20 is installed in a location that allows imaging of the rear seats of the vehicle, such as the overhead console or surrounding area of ​​the vehicle, the rearview mirror or surrounding area of ​​the vehicle, or the headrest or surrounding area of ​​the front seats of the vehicle. An imaging device 20 installed in such a location can acquire color images or infrared images of the rear seats inside the vehicle as display targets.

[0018] The imaging device 20 can acquire a color image of the rear seats inside the vehicle by performing imaging processing by the imaging unit 21 without emitting light from the light-emitting unit 22, in environments where the illuminance inside the vehicle is above a predetermined value. Situations in which the illuminance inside the vehicle is above a predetermined value include, for example, daytime, nighttime when the interior lamps are on, and when the interior lamps are automatically turned on in conjunction with the user opening the door in a dark place such as a garage.

[0019] The imaging device 20 can acquire an infrared image of the rear seats inside the vehicle by performing imaging processing with the imaging unit 21 while emitting light from the light-emitting unit 22 in environments where the illuminance inside the vehicle is not above a predetermined value. Situations in which the illuminance inside the vehicle is lower than the predetermined value include, for example, nighttime driving, or driving on dark roads such as tunnels during the daytime.

[0020] The in-vehicle computer 30 is an example of a control unit and is composed of an ECU (Electronic Control Unit) equipped with, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The in-vehicle computer 30 may be a dedicated ECU for controlling the vehicle image display device 10, or it may be an integrated ECU capable of controlling other devices mounted on the vehicle besides the vehicle image display device 10.

[0021] The in-vehicle computer 30 includes various processing units such as a device control processing unit 31, an image recognition processing unit 32, an information acquisition processing unit 33, a storage processing unit 34, an image generation processing unit 35, a display processing unit 36, and an illumination determination processing unit 37. The various processing units 31 to 37 are virtually implemented as software or applications by the in-vehicle computer 30 executing a control program. The control program includes the vehicle image display program disclosed herein. The various processing units 31 to 37 may be implemented by hardware, or by a combination of software and hardware.

[0022] The device control processing unit 31 can perform control over the imaging process carried out by the imaging device 20. More specifically, the device control processing unit 31 can perform, for example, exposure control to control exposure by the imaging unit 21, light emission control to control light emission by the light emission unit 22, and abnormality detection processing to detect whether or not an abnormality has occurred in the imaging device 20, thereby controlling the overall operation of the imaging device 20.

[0023] The image recognition processing unit 32 is an example of an identification unit, and it can perform image recognition processing on the display target image acquired by the imaging device 20 to identify a person in the display target image as a display target person. The image recognition processing that the image recognition processing unit 32 can perform includes, for example, face detection processing to detect the face of a person in the display target image, and feature extraction processing to extract feature quantities from a person in the display target image. Feature quantities are information that indicates the characteristics of each part of a person, such as the contour of the face, the position and shape of the eyes on the face, the position and shape of the nose on the face, the position and shape of the mouth on the face, and the hairstyle. Based on the face detection processing and feature extraction processing, the image recognition processing unit 32 identifies a person in the display target image acquired by the imaging device 20 as a display target person.

[0024] The information acquisition processing unit 33 is an example of an information acquisition unit and is capable of acquiring information related to a person as person-related information. For example, the information acquisition processing unit 33 acquires the color image itself acquired by the imaging device 20 as person-related information in a bright environment such as during the daytime, that is, in an environment where the illuminance inside the vehicle is above a predetermined value.

[0025] The memory processing unit 34 is configured to store various types of data in a storage medium (not shown) provided by the vehicle, such as a flash memory. The memory processing unit 34 stores color images captured by the imaging device 20 and acquired as person-related information by the information acquisition processing unit 33 in the storage medium (not shown).

[0026] The image generation processing unit 35 is an example of an image generation unit, and is realized by incorporating an artificial intelligence algorithm such as a Generative Adversarial Network (GAN). When the imaging device 20 acquires an infrared image of the vehicle interior as the image to be displayed, the image generation processing unit 35 performs image processing based on person-related information on the infrared image to generate a colorized image. The colorized image is an example of a substitute image, and can be defined as, for example, an image in which the infrared image has been colorized or an image in which color information has been combined with the infrared image.

[0027] When an infrared image is acquired by the imaging device 20 as the image to be displayed, the image generation processing unit 35 searches the data set of color images stored by the storage processing unit 34 for a color image in which the same person as the person in the infrared image is present. This search process can be performed, for example, by the image generation processing unit 35 performing image recognition processing similar to that of the image recognition processing unit 32 to identify the person in the infrared image or the color image. Alternatively, the image generation processing unit 35 may request the image recognition processing unit 32 to perform the color image search process. When the image generation processing unit 35 finds a color image in which the same person as the person in the infrared image is present, it performs image processing based on the found color image on the portion of the infrared image in which the person to be displayed is present to generate a colorized image.

[0028] When generating a colorized image, the image generation processing unit 35 obtains person color information related to the person's color from the searched color image and performs image processing based on that person color information. More specifically, for example, in the infrared image, the image generation processing unit 35 obtains the skin color of the person from the searched color image as person color information and superimposes it on the parts where the person's skin is visible. Also, in the infrared image, the image generation processing unit 35 obtains the hair color of the person from the searched color image as person color information and superimposes it on the parts where the person's hair is visible. In this way, the image generation processing unit 35 synthesizes the person color information obtained from the color image into each part of the infrared image. As a result, the image generation processing unit 35 generates a colorized image by converting the infrared image into a color image.

[0029] When the illuminance inside the vehicle is above a predetermined value, the image generation processing unit 35 provides the acquired color image to the display processing unit 36 ​​without performing any image processing on the color image to be displayed. When the illuminance inside the vehicle is not above a predetermined value, the image generation processing unit 35 performs image processing on the acquired infrared image to be displayed to generate a colorized image. Then, the image generation processing unit 35 provides the colorized image to the display processing unit 36 ​​in place of the infrared image.

[0030] The image generation processing unit 35 stores the infrared image, which is the image to be displayed, as input data, and the generated colorized image, which is a replacement image, as output data. The image generation processing unit 35 then performs machine learning based on the stored data. By repeating this machine learning, the image generation processing unit 35 becomes able to generate clearer colorized images from the infrared image.

[0031] The display processing unit 36 ​​is an example of a display unit and outputs a color image or a colorized image received from the image generation processing unit 35 to the display unit 50. In environments where the illuminance inside the vehicle is above a predetermined value, the color image acquired by the imaging device 20 is output directly to the display unit 50. In environments where the illuminance inside the vehicle is not above a predetermined value, a colorized image generated by the image generation processing unit 35 is output to the display unit 50 instead of the infrared image acquired by the imaging device 20.

[0032] The illuminance determination processing unit 37 can determine whether the illuminance inside the vehicle is above a predetermined value. For example, if the vehicle is equipped with an illuminance sensor that detects the illuminance inside the vehicle, the illuminance determination processing unit 37 can determine whether the illuminance inside the vehicle is above a predetermined value based on the illuminance detected by the illuminance sensor. Alternatively, the illuminance determination processing unit 37 may, for example, utilize the clock function installed in the vehicle to determine that the illuminance inside the vehicle is above a predetermined value during the daytime, and that it is not above a predetermined value during the nighttime. The predetermined value that serves as the criterion for illuminance determination can be changed and set as appropriate.

[0033] The display unit 50 is capable of displaying images output from the on-board computer 30. In environments where the illumination inside the vehicle is above a predetermined value, the display unit 50 displays a color image output from the on-board computer 30. In environments where the illumination inside the vehicle is not above a predetermined value, the display unit 50 displays a colorized image output from the on-board computer 30. The display unit 50 is composed of, for example, a center display mounted in front of the front seats inside the vehicle, and is a display device that can be seen by at least the driver seated in the driver's seat of the vehicle. The display unit 50 may also be, for example, an aftermarket dedicated display installed on the dashboard of the vehicle.

[0034] Next, we will explain the control flow for the "process of acquiring person-related information in advance" and the "process of displaying a colorized image as a substitute image" that the vehicle image display device 10 can execute.

[0035] <Process for acquiring person-related information in advance> As illustrated in Figure 2, the in-vehicle computer 30 acquires a color image of the interior of the vehicle using the imaging device 20 in an environment where the illumination inside the vehicle is above a predetermined value (Step A1). Step A1 is an example of an image acquisition step. The in-vehicle computer 30 detects the faces of people in the color image acquired by the imaging device 20 (Step A2). The in-vehicle computer 30 extracts the feature quantities of people in the color image acquired by the imaging device 20 (Step A3).

[0036] The in-vehicle computer 30 identifies the person in the color image acquired by the imaging device 20 as the person to be displayed, based on the results of face detection processing (step A2) and feature extraction processing (step A3) (step A4). Step A4 is an example of the identification step. The in-vehicle computer 30 stores the color image acquired by the imaging device 20 as person-related information (step A5). Step A5 is an example of the information acquisition step. The in-vehicle computer 30 displays the color image acquired by the imaging device 20 on the display 50 in an environment where the illuminance inside the vehicle is above a predetermined value (step A6). Step A6 is an example of the display step.

[0037] <Process for displaying a colorized image as a substitute image> As illustrated in Figure 3, the in-vehicle computer 30 acquires an infrared image of the interior of the vehicle using the imaging device 20 in an environment where the illumination inside the vehicle is not above a predetermined value (step B1). Step B1 is an example of an image acquisition step. The in-vehicle computer 30 detects the faces of people in the infrared image acquired by the imaging device 20 (step B2). The in-vehicle computer 30 extracts the feature quantities of people in the infrared image acquired by the imaging device 20 (step B3).

[0038] The in-vehicle computer 30 identifies the person in the infrared image acquired by the imaging device 20 as the person to be displayed, based on the results of face detection processing (step B2) and feature extraction processing (step B3) (step B4). Step B4 is an example of the identification step. The in-vehicle computer 30 searches the data group of color images stored by the memory processing unit 34 for a color image in which the same person as the person in the infrared image is present (step B5). The in-vehicle computer 30 performs image processing on the portion of the infrared image in which the person to be displayed is present, based on the searched color image, to generate a colorized image (step B6). Step B6 is an example of the image generation step. In an environment where the illuminance inside the vehicle is not above a predetermined value, the in-vehicle computer 30 displays the colorized image generated in step B6 on the display 50 instead of the infrared image acquired by the imaging device 20 (step B7). Step B7 is an example of the display step.

[0039] According to the first embodiment, the imaging device 20 acquires a color image and an infrared image of the interior of the vehicle as display target images. The image recognition processing unit 32 identifies a person in the display target image as a display target person. The information acquisition processing unit 33 acquires information related to the person as person-related information before the image processing by the image generation processing unit 35 is performed. When an infrared image is acquired by the imaging device 20, the image generation processing unit 35 performs image processing on the portion of the infrared image in which the display target person is visible, based on the person-related information acquired in advance by the information acquisition processing unit 33, to generate a colorized image. The display processing unit 36 ​​can display the display target image acquired by the imaging device 20 on the display unit 50. Furthermore, in environments where the illumination inside the vehicle is not above a predetermined value, in other words, in environments where it is difficult to acquire a clear image of the interior of the vehicle, the display processing unit 36 ​​displays the colorized image generated by the image generation processing unit 35 on the display unit 50 instead of the infrared image acquired by the imaging device 20.

[0040] According to this configuration example, in environments where it is difficult to obtain a clear image of the interior of a vehicle, the image generation processing unit 35 can perform image processing on an alternative image, that is, it can display a colorized image in which image processing based on previously acquired person-related information has been performed on at least the portion of the infrared image in which the person to be displayed is visible. As a result, even in environments where it is difficult to obtain a clear image of the interior of a vehicle, an image can be displayed that makes it easy to confirm the situation of people present in the vehicle.

[0041] In colorized images, where various parts of a person are colored, the contours of the person's face and other features become clearer compared to infrared or monochrome images, which tend to be blurry. Therefore, it becomes easier to recognize the person's appearance and expressions. Furthermore, in colorized images, it becomes possible to grasp narrative information, such as "the person's face is red because they are angry" or "the area around their eyes is red because they have just cried," making it easier to understand the person's situation in more detail.

[0042] According to the vehicle image display device 10, the imaging device 20 acquires an image of the rear seat in the vehicle interior as a display target image. The display processing unit 36 displays the color image acquired by the imaging device 20 or the colorized image generated by the image generation processing unit 35 on the display 50 in front of the driver's seat that can be visually recognized by the driver of the vehicle. According to this configuration example, the driver facing forward in the traveling direction of the vehicle can check the state of the person sitting in the rear seat via the front display 50 without directly looking at the rear seat. Note that the imaging device 20 may be provided not only at a position where it can image the rear seat in the vehicle interior but also at a position where it can image an area that cannot be directly seen by the driver facing forward in the traveling direction of the vehicle.

[0043] According to the vehicle image display device 10, the image generation processing unit 35 can execute machine learning based on a data group in which an infrared image, which is a display target image, is used as input data and a colorized image, which is a generated alternative image, is used as output data. By repeating such machine learning, it becomes possible to generate a clearer colorized image and display an image that makes it easier to check the situation of the people present in the vehicle interior.

[0044] According to the vehicle image display device 10, the information acquisition processing unit 33 acquires the color image itself in which the display target person is captured as person-related information. According to this configuration example, the information source itself that contains various information related to the display target person can be utilized when generating the colorized image. Therefore, for example, it is possible to avoid a situation where there is insufficient information required for colorizing the infrared image and generate a clearer colorized image.

[0045] According to the vehicle image display device 10, the information acquisition processing unit 33 acquires the color image acquired by the imaging device 20 as person-related information in an environment where the illuminance in the vehicle interior is a predetermined value or more. According to this configuration example, a colorful color image can be utilized when generating the colorized image. Therefore, a clearer colorized image can be generated.

[0046] (Second Embodiment) According to the second embodiment, the information acquisition processing unit 33 is configured to acquire person feature quantity information as person-related information from the color image acquired by the imaging device 20. The person feature quantity information is, for example, information indicating the contour of the face of the person shown in the color image, information indicating the position and shape of the eyes on the face, information indicating the position and shape of the nose on the face, information indicating the position and shape of the mouth on the face, information indicating the hairstyle, and other information indicating the features of each part of the human body of the person.

[0047] Further, the information acquisition processing unit 33 is configured to acquire person color information as person-related information from the color image acquired by the imaging device 20. The person color information is, for example, information indicating the skin color of the person shown in the color image, information indicating the color of the eyes, information indicating the color of the lips, information indicating the color of the hair, and other information indicating the colors of each part of the human body of the person.

[0048] As illustrated in FIG. 4, the storage processing unit 34 stores the color image acquired by the imaging device 20 as person-related information, and also stores the person feature quantity information and person color information acquired from the color image in association with each color image. When searching for a color image in which the same person as the display target person shown in the infrared image appears, the image generation processing unit 35 can also utilize the person feature quantity information acquired and stored in advance by the information acquisition processing unit 33. That is, the image generation processing unit 35 can speed up the search process by searching for a color image in which the same feature quantity or an approximate feature quantity as the feature quantity of the display target person shown in the infrared image is stored. In addition, the image generation processing unit 35 performs image processing based on the person color information acquired and stored in advance by the information acquisition processing unit 33 for the display target person shown in the infrared image. That is, the image generation processing unit 35 generates a colorized image obtained by colorizing the infrared image based on the person color information acquired and stored in advance by the information acquisition processing unit 33 as a substitute image.

[0049] By acquiring and storing person-related information using the data structure illustrated in Figure 4, the image generation processing unit 35 can search for color images containing the same person as the person in the infrared image from the data set of color images stored by the storage processing unit 34, while referring to the person feature information stored by the storage processing unit 34. Therefore, it becomes easier to find color images containing the same person as the person in the infrared image.

[0050] Furthermore, when the image generation processing unit 35 performs image processing based on the searched color image to generate a colorized image, it can colorize the infrared image based on already stored person color information without having to acquire person color information again from the color image. Therefore, the processing load when generating a colorized image can be reduced.

[0051] Furthermore, the information acquisition processing unit 33 may be configured to discard the color image after acquiring feature information and person color information as person-related information from the color image acquired by the imaging device 20. In other words, if the minimum information necessary to convert an infrared image to a color image has been acquired, it is possible to convert the infrared image to a color image even if the color image, which is the source of that information, is discarded. Therefore, if the minimum information necessary to convert an infrared image to a color image has been acquired from the color image, discarding that color image can prevent situations where limited memory resources run out.

[0052] The information acquisition processing unit 33 may be configured to acquire person feature information and person color information extracted from color images acquired by the imaging device 20 as person-related information, or, instead of acquiring person feature information and person color information extracted from color images acquired by the imaging device 20 as person-related information, to acquire and store person feature information and person color information that have been pre-entered by the user as person-related information.

[0053] (Third Embodiment) According to the third embodiment, the information acquisition processing unit 33 is configured to acquire clothing-related information from the color image acquired by the imaging device 20. Clothing-related information is, for example, information indicating the characteristic quantities and colors of the clothing worn by the person in the color image. Clothing characteristic quantities are, for example, information indicating the shape of the clothing and the patterns applied to the clothing.

[0054] As illustrated in Figure 5, the memory processing unit 34 stores the color images acquired by the imaging device 20 as person-related information, and also stores clothing-related information associated with each color image. The image generation processing unit 35 performs image processing on the target person in the infrared image based on the person color information that has been previously acquired and stored by the information acquisition processing unit 33. Furthermore, the image generation processing unit 35 performs image processing on the clothing worn by the target person in the infrared image based on the clothing-related information that has been previously acquired and stored by the information acquisition processing unit 33. In other words, the image generation processing unit 35 generates a colorized image of the person and clothing in the infrared image as a substitute image, based on the person color information and clothing-related information that have been previously acquired and stored by the information acquisition processing unit 33.

[0055] If the image generation processing unit 35 determines that the clothing worn by a person identified by the image recognition processing unit 32 in the infrared image acquired by the imaging device 20 matches the clothing indicated by the clothing-related information previously acquired by the information acquisition processing unit 33, it will colorize not only the person in the infrared image but also the clothing.

[0056] If the clothing worn by a person identified by the image recognition processing unit 32 in the infrared image acquired by the imaging device 20 does not match the clothing indicated by the clothing-related information previously acquired by the information acquisition processing unit 33, the image generation processing unit 35 excludes the portion of the infrared image acquired by the imaging device 20 that shows the clothing of the person identified by the image recognition processing unit 32 from the image processing.

[0057] According to this configuration example, if the clothing captured in the infrared image matches the clothing indicated by pre-acquired clothing-related information, that clothing can also be colorized. By colorizing not only the person but also the clothing in the infrared image, it becomes possible to display a colorized image that makes it easier to confirm the situation of the person inside the vehicle.

[0058] On the other hand, if the clothing captured in the infrared image does not match the clothing indicated by the previously acquired clothing-related information, the clothing cannot be colorized, and even if it were colorized, it is not guaranteed that a colorized image reflecting the actual color of the clothing could be generated. Therefore, if the clothing captured in the infrared image does not match the clothing indicated by the previously acquired clothing-related information, it is advisable to exclude the portion of the infrared image containing the clothing from image processing. This prevents the display of a colorized image that does not reflect the actual color of the clothing.

[0059] Furthermore, clothing-related information may be displayed on the screen of the display unit 50 in a selectable format, such as the colors of multiple types of clothing, the shapes of multiple types of clothing, and the coordination patterns of multiple types of clothing. The information selected by the user may be acquired and stored in advance as clothing-related information.

[0060] Clothing-related information may be acquired and stored when clothing with the same features and colors, or clothing with similar features and colors, is recognized from multiple images taken at different times. This allows information related to the clothing frequently worn by the person being displayed to be stored as clothing-related information and utilized when converting infrared images to color images.

[0061] (Fourth Embodiment) According to the fourth embodiment, the information acquisition processing unit 33 is configured to acquire non-person-related information from the color image acquired by the imaging device 20. Non-person-related information is information that indicates the characteristics and colors of subjects other than people, such as a child seat. The characteristics of subjects other than people are information that indicates the shape of the subject, for example.

[0062] As illustrated in Figure 6, the memory processing unit 34 stores the color images acquired by the imaging device 20 as person-related information, and also stores non-person-related information associated with each color image. The image generation processing unit 35 performs image processing on the target person in the infrared image based on the person color information that has been previously acquired and stored by the information acquisition processing unit 33. Furthermore, the image generation processing unit 35 performs image processing on subjects other than people in the infrared image based on the non-person-related information that has been previously acquired and stored by the information acquisition processing unit 33. That is, the image generation processing unit 35 generates a colorized image as a substitute image, in which the people and non-person subjects in the infrared image are colorized, based on the person color information and non-person-related information that have been previously acquired and stored by the information acquisition processing unit 33.

[0063] According to this configuration example, it is possible to colorize not only the portion of the infrared image that contains the person being displayed, but also the portion containing other subjects. Therefore, not only people in the infrared image, but also other subjects such as child seats can be colorized, resulting in a colorized image that makes it easier to check the situation inside the car.

[0064] (Fifth Embodiment) As illustrated in Figure 7, according to the fifth embodiment, the image generation processing unit 35 determines whether or not the infrared image acquired by the imaging device 20 includes scenery outside the vehicle (step C1). Whether or not the infrared image includes scenery outside the vehicle can be determined by performing image recognition processing, for example, whether or not feature quantities indicating the shape of the vehicle's windows can be extracted from the infrared image, or whether or not feature quantities of objects existing outside the vehicle, such as traffic lights, can be extracted from the infrared image.

[0065] If the image generation processing unit 35 determines that the infrared image contains scenery outside the vehicle (step C1: YES), it excludes the portion of the infrared image containing the scenery outside the vehicle from the image processing (step C2) and proceeds to step C3. If the image generation processing unit 35 determines that the infrared image does not contain scenery outside the vehicle (step C1: NO), it proceeds to step C3 without performing the process of excluding a portion of the infrared image from the image processing (step C2).

[0066] The image generation processing unit 35, having moved to step C3, appropriately colorizes the portion of the infrared image that has not been excluded from image processing to generate a colorized image. At this time, the image generation processing unit 35 may colorize the entire portion of the infrared image that has not been excluded from image processing, or it may colorize only a portion of the portion of the infrared image that has not been excluded from image processing, for example, only the portion in which a person is pictured.

[0067] The portion of the infrared image that shows the scenery outside the vehicle does not contain the person to be displayed, and therefore there is little need to colorize such portions. According to the fifth embodiment, if the infrared image contains the scenery outside the vehicle, the portion of the infrared image that contains the scenery outside the vehicle is excluded from image processing. This prevents the image from being colorized even in portions of the infrared image that do not contain the person to be displayed, thereby reducing the load when performing image processing.

[0068] (Sixth Embodiment) According to the sixth embodiment, the image generation processing unit 35 is configured to perform image processing when conditions set by the user are met. That is, as illustrated in Figure 8, the image generation processing unit 35 determines whether or not a person set by the user is captured in the infrared image acquired by the imaging device 20 (step D1), and whether or not a person corresponding to a set attribute set by the user is captured in the infrared image acquired by the imaging device 20 (step D2).

[0069] Then, if the image recognition processing unit 32 identifies a person set by the user in the infrared image acquired by the imaging device 20 (Step D1: YES), or if the image recognition processing unit 32 identifies a person corresponding to a set attribute set by the user in the infrared image acquired by the imaging device 20 (Step D2: YES), the image generation processing unit 35 performs image processing to convert the infrared image into a color image (Step D3).

[0070] On the other hand, if the image recognition processing unit 32 does not identify a person set by the user in the infrared image acquired by the imaging device 20 (Step D1: NO), and the image recognition processing unit 32 does not identify a person corresponding to the attributes set by the user in the infrared image acquired by the imaging device 20 (Step D2: NO), the image generation processing unit 35 does not perform image processing to convert the infrared image into a color image.

[0071] For example, if the user, the driver, has set a condition that image processing should be performed when a designated person, such as "eldest son," is seated in the back seat, the image generation processing unit 35 will perform image processing to convert the infrared image into a color image when the image recognition processing unit 32 identifies "eldest son," as set by the user, in the infrared image acquired by the imaging device 20.

[0072] Furthermore, for example, if the user (driver) has set a condition that image processing should be performed only when the attribute of a person sitting in the back seat is "child", the image generation processing unit 35 will perform image processing to convert the infrared image into a color image when the image recognition processing unit 32 identifies a person with the attribute set by the user, in this case "child," in the infrared image acquired by the imaging device 20.

[0073] Information indicating which person is the "eldest son" and which person is a "child" can be pre-input into the vehicle image display device 10. Specifically, the in-vehicle computer 30 extracts face images of people exhibiting the same or similar feature quantities from the color image data stored by the memory processing unit 34. The in-vehicle computer 30 then displays the extracted face image on the display 50 and also displays an inquiry message such as "Who is this person?" on the display 50. The in-vehicle computer 30 also displays an answer input field on the display 50 for the user to enter an answer. The user can enter information such as "eldest son" or "child" in this answer input field. The in-vehicle computer 30 stores the entered information in association with the color image.

[0074] By pre-storing this information, the in-vehicle computer 30 can identify who the person in the color image is, or what their attributes are. The in-vehicle computer 30 then searches for color images containing the same or similar features as the person in the infrared image, and by referring to the information stored in association with the searched color image, it can identify who the person in the infrared image is, or what their attributes are.

[0075] Furthermore, it is possible to set conditions for executing image processing, such as "when the same person is on board on the same day." For example, if the same person is on board during the day and at night on the same day, a color image of that person taken during the day will be stored, so the infrared image taken at night can be adequately converted to color. On the other hand, if, for example, the person in the nighttime infrared image was not on board during the day on the same day, a color image of that person taken during the day will not be stored, so the infrared image taken at night cannot be converted to color. In cases where there is a possibility that color conversion may not be possible, it is advisable to set the system to not perform color conversion in advance.

[0076] Furthermore, even if the same person rides the vehicle multiple times during the same night, if a color image is acquired at least once while the interior lights are on, it is possible to convert subsequent infrared images into color images.

[0077] Furthermore, even if the condition "when the same person is in the vehicle on the same day" is not set, the on-board computer 30 may automatically perform color image processing if it determines through image recognition that "the same person was in the vehicle on the same day," and may not automatically perform color image processing if it determines that "the same person was not in the vehicle on the same day."

[0078] (Seventh Embodiment) As illustrated in Figure 9, according to the seventh embodiment, the image generation processing unit 35 compares multiple infrared images acquired by the imaging device 20, that is, multiple infrared images acquired at different timings (step E1). Then, based on the comparison results of the multiple infrared images, the image generation processing unit 35 determines whether or not a person has moved inside the vehicle (step E2).

[0079] The image generation processing unit 35 determines that a person has moved inside the vehicle if the posture of the person in the multiple infrared images is different (Step E2: YES). If the image generation processing unit 35 determines that a person has moved inside the vehicle, it performs image processing to convert the infrared images into color images (Step E3). On the other hand, if the posture of the person in the multiple infrared images is the same, the image generation processing unit 35 determines that the person has not moved inside the vehicle (Step E2: NO). If the image generation processing unit 35 does not determine that a person has moved inside the vehicle, it does not perform image processing to convert the infrared images into color images.

[0080] According to the seventh embodiment, the image generation processing unit 35 can detect the movement of a person inside the vehicle based on a plurality of infrared images acquired by the imaging device 20. When the image generation processing unit 35 detects the movement of a person inside the vehicle, it performs image processing to convert the infrared images into color images. When it does not detect the movement of a person inside the vehicle, it does not perform image processing to convert the infrared images into color images. This configuration avoids unnecessary image processing even when there is no movement of a person inside the vehicle, that is, when there is little need to generate a new colorized image, thereby reducing the load when performing image processing.

[0081] (Eighth Embodiment) According to the eighth embodiment, the image generation processing unit 35 is configured to repeatedly perform image processing to convert an infrared image into a color image at predetermined time intervals. The image generation processing unit 35 is configured to change the time interval at which the image processing is repeated based on the magnitude of the processing load associated with performing the image processing to convert an infrared image into a color image.

[0082] More specifically, the image generation processing unit 35 shares resources such as the processor and memory provided in the vehicle with devices other than the vehicle image display device 10. Therefore, as illustrated in Figure 10, the image generation processing unit 35 determines whether the processing load associated with performing image processing exceeds a predetermined standard (step F1). This standard can be changed and set as appropriate.

[0083] If the processing load associated with performing image processing exceeds a predetermined standard (Step F1: YES), the image generation processing unit 35 increases the time interval at which it repeatedly performs image processing (Step F2). On the other hand, if the processing load associated with performing image processing does not exceed a predetermined standard (Step F1: NO), the image generation processing unit 35 does not change the time interval at which it repeatedly performs image processing and maintains the same time interval (Step F3).

[0084] As illustrated in Figure 11, the vehicle image display device 10 further includes a notification processing unit 38. The notification processing unit 38 is virtually implemented as software or an application by the in-vehicle computer 30 executing a control program. The notification processing unit 38 may be implemented by hardware, or by a combination of software and hardware.

[0085] The notification processing unit 38 is an example of a notification unit, and when the time interval at which the image generation processing unit 35 repeatedly performs image processing is changed, it notifies the system of the time interval change information. The time interval change information is information that notifies the system that the time interval at which the image generation processing unit 35 repeatedly performs image processing has been changed. The notification processing unit 38 may, for example, output the time interval change information as text information on the screen of the display unit 50, or it may output the time interval change information as audio information from a speaker (not shown) installed in the vehicle.

[0086] According to the eighth embodiment, the image processing by the image generation processing unit 35 is repeatedly performed at predetermined time intervals. This allows the colorized image displayed on the display unit 50 to always be updated to show the latest situation, and for example, the driver can check the situation inside the vehicle in real time via the colorized image displayed on the display unit 50.

[0087] The image generation processing unit 35 changes the time interval based on the processing load associated with performing image processing. In this configuration example, if the processing load associated with performing image processing becomes excessive, the time interval between repeated image processing executions is lengthened to reduce the number of image processing executions, thereby allowing the image processing to continue while sharing limited resources with other devices. On the other hand, if the processing load associated with performing image processing is small and there are ample resources to share with other devices, it is preferable to shorten the time interval between repeated image processing executions to increase the number of image processing executions. This increases the update frequency of the colorized image displayed on the display unit 50, enabling the display of an image that reflects the conditions inside the vehicle in real time.

[0088] The notification processing unit 38 notifies the user of the change in the time interval at which the image generation processing unit 35 repeatedly performs image processing. With this configuration example, the user can be informed of how often the colorized image displayed on the display unit 50 is being updated. As a result, the user can understand whether the colorized image displayed on the display unit 50 represents real-time or near-real-time conditions.

[0089] (Ninth Embodiment) According to the ninth embodiment, the image generation processing unit 35 is configured to repeatedly perform image processing to convert an infrared image into a color image at predetermined time intervals. The imaging device 20 is configured to acquire the display target image inside the vehicle at a time interval shorter than the time interval at which the image generation processing unit 35 performs image processing.

[0090] As illustrated in Figure 12, the vehicle image display device 10 further includes an estimation processing unit 39. The estimation processing unit 39 is virtually implemented as software or an application by the in-vehicle computer 30 executing a control program. The estimation processing unit 39 may be implemented by hardware, or by a combination of software and hardware.

[0091] The estimation processing unit 39 is an example of an estimation unit, and estimates the state of a person inside the vehicle based on multiple display target images acquired by the imaging device 20 at time intervals shorter than the time intervals at which the image generation processing unit 35 performs image processing.

[0092] The estimation processing unit 39 estimates that a person is asleep if, for example, the posture of a person in multiple display target images is exactly the same or almost the same. The estimation processing unit 39 estimates that a person is awake if, for example, the posture of a person in multiple display target images is significantly different.

[0093] The display output unit 25 outputs status information indicating the state of a person estimated by the estimation processing unit 39 onto the screen of the display unit 50. When the interior of the vehicle is bright, such as during the day, the display unit 50 displays the status information together with the display target image of the interior of the vehicle acquired by the imaging device 20. When the interior of the vehicle is dark, such as at night, the display unit 50 displays the status information together with a colorized image, which is a substitute image generated by the image generation processing unit 35.

[0094] As status information, it is preferable to display text information indicating a person's status, such as "sleeping" or "awake," superimposed on the target image or colorized image on the screen of the display unit 50. In addition, status information may be displayed as text information on the screen of the display unit 50, or, instead of displaying it as text information on the screen of the display unit 50, it may be output as audio information from a speaker (not shown) installed in the vehicle.

[0095] According to the ninth embodiment, status information indicating the state of the person being displayed is added to the display image or colorized image in which the person being displayed is pictured. Therefore, for example, even if the display image or colorized image shows the person with their eyes closed due to blinking or other reasons, even though the person being displayed is awake, the status information allows for confirmation that the person is awake.

[0096] (Tenth Embodiment) According to the tenth embodiment, the image generation processing unit 35 is configured to repeatedly perform image processing to convert an infrared image into a color image at predetermined time intervals. The imaging device 20 is configured to acquire the display target image inside the vehicle at a time interval shorter than the time interval at which the image generation processing unit 35 performs image processing.

[0097] As illustrated in Figure 13, the vehicle image display device 10 further includes a determination processing unit 40. The determination processing unit 40 is virtually implemented as software or an application by the in-vehicle computer 30 executing a control program. The determination processing unit 40 may be implemented by hardware, or by a combination of software and hardware.

[0098] The determination processing unit 40 is an example of a determination unit, and it determines whether the eyes of a person inside the vehicle are open or closed based on multiple display target images acquired by the imaging device 20 at time intervals shorter than the time intervals in which the image generation processing unit 35 performs image processing.

[0099] The determination processing unit 40 determines, for example, that a person's eyes are open if the eyes of the person in all of the multiple display target images are open. However, even if there are multiple display target images in which a person's eyes are closed, the determination processing unit 40 may also determine that the person's eyes are open if the number of such images is small enough to be judged as merely a temporary closure, such as blinking.

[0100] The determination processing unit 40 determines, for example, that a person's eyes are closed if all of the eyes of the person depicted in multiple display target images are closed. However, even if there are display target images in which a person's eyes are open among the multiple display target images, the determination processing unit 40 may also determine that the person's eyes are closed if the number of such images is small enough to be judged as merely a temporary opening of the eyes due to being awake.

[0101] If the determination processing unit 40 determines that the eyes of a person inside the vehicle are open, the image generation processing unit 35 identifies the infrared image with the greatest degree of eye opening among multiple infrared images. The image generation processing unit 35 then performs image processing on the identified infrared image to generate a colorized image. This prevents the colorization from being based on an image where the person inside the vehicle is actually awake, but whose eyes are half-open or closed, for example due to blinking. It also prevents the colorization from being based on an image where the person inside the vehicle is actually asleep, but whose eyes are half-open, for example due to temporary wakefulness.

[0102] Furthermore, if the determination processing unit 40 determines that the eyes of a person inside the vehicle are open, the image generation processing unit 35 may overlay text information or icon information indicating that the person is awake on the screen of the display unit 50. This allows the system to indicate that the person inside the vehicle is awake, even if the color image is created based on an image where the eyes are half-open or closed due to blinking or other reasons. Also, if the determination processing unit 40 determines that the eyes of a person inside the vehicle are closed, the system may overlay text information or icon information indicating that the person is asleep on the screen of the display unit 50. This allows the system to indicate that the person inside the vehicle is asleep, even if the color image is created based on an image where the person is temporarily awake during sleep and their eyes are half-open or open.

[0103] (11th Embodiment) The vehicle image display system 100 illustrated in Figure 14 comprises an on-board device 101 mounted on a vehicle and an external device 102 capable of wireless data communication with the on-board device 101.

[0104] The in-vehicle device 101 includes an imaging device 20 and a display 50. The in-vehicle device 101 also includes an in-vehicle computer 30. The in-vehicle computer 30 includes a device control processing unit 31, a display processing unit 36, an illumination determination processing unit 37, and the like. The external device 102 includes components of the in-vehicle computer 30 in the above-described embodiment, such as an image recognition processing unit 32, an information acquisition processing unit 33, a storage processing unit 34, and an image generation processing unit 35. The vehicle image display system 100 is configured in which the functions of the in-vehicle computer 30 in the above-described embodiment are distributed between the in-vehicle device 101 and the external device 102.

[0105] According to the vehicle image display system 100, when the illuminance inside the vehicle, as detected by the illuminance determination processing unit 37, is above a predetermined value, the in-vehicle device 101 provides the color image acquired by the imaging device 20 to the display processing unit 36. The display processing unit 36 ​​then displays the provided color image on the screen of the display unit 50. Furthermore, when the illuminance inside the vehicle, as detected by the illuminance determination processing unit 37, is above a predetermined value, the in-vehicle device 101 also provides the color image acquired by the imaging device 20 to the external device 102 via wireless data communication. On the other hand, when the illuminance inside the vehicle, as detected by the illuminance determination processing unit 37, is not above a predetermined value, the in-vehicle device 101 provides the infrared image acquired by the imaging device 20 to the external device 102 via wireless data communication.

[0106] When the external device 102 receives a color image from the in-vehicle device 101, the image recognition processing unit 32 identifies the person in the color image received from the in-vehicle device 101 as the person to be displayed. The information acquisition processing unit 33 of the external device 102 acquires the color image provided by the in-vehicle device 101 as person-related information. The storage processing unit 34 of the external device 102 stores the color image provided by the in-vehicle device 101 as person-related information. The external device 102 may also store feature information and color information acquired from the color image.

[0107] When the external device 102 receives an infrared image from the in-vehicle device 101, the image generation processing unit 35 performs image processing on the portion of the infrared image received from the in-vehicle device 101 that contains the person to be displayed, based on information such as a color image that has been acquired in advance, to generate a colorized image from the infrared image. The external device 102 then provides the generated colorized image to the in-vehicle device 101 via wireless data communication.

[0108] Upon receiving a colorized image from the external device 102, the in-vehicle device 101 outputs the colorized image to the display unit 50 via the display processing unit 36. As a result, the screen of the display unit 50 of the in-vehicle device 101 displays the colorized image generated by the external device 102 instead of the infrared image.

[0109] Even with the vehicle image display system 100, in environments where it is difficult to obtain a clear image of the interior of the vehicle, the image generation processing unit 35 can display an alternative image, that is, a colorized image in which image processing has been performed on the portion of the infrared image in which the person to be displayed is shown, based on information such as a pre-acquired color image. As a result, even in environments where it is difficult to obtain a clear image of the interior of the vehicle, a colorized image can be displayed that makes it easier to confirm the situation of people inside the vehicle.

[0110] Furthermore, according to the vehicle image display system 100, relatively resource-intensive processes involving image recognition and image processing, such as image recognition processing, information acquisition processing, storage processing, and image generation processing, are performed by the external device 102 rather than the in-vehicle device 101. Therefore, the processing load on the in-vehicle device 101 can be reduced. Data communication between the in-vehicle device 101 and the external device 102 may be wired communication, or a configuration in which wireless and wired data communication are mixed.

[0111] (Other Embodiments) This disclosure is not limited to the embodiments described above, and various modifications and extensions can be made without departing from the spirit of the disclosure. For example, a vehicle image display device or vehicle image display system may be configured by appropriately selecting and combining the embodiments described above.

[0112] The information acquired in advance by the information acquisition processing unit 33 before the image generation processing is executed by the image generation processing unit 35 is not limited to the various types of information exemplified in the multiple embodiments described above, but can be applied to a variety of information, as long as it is information that can be used when converting an infrared image into a color image.

[0113] The color images stored by the memory processing unit 34 should be configured to be updated with the most recent color images possible. For example, a child's appearance may change significantly even in a relatively short period of time, and even an adult's hairstyle or hair color may change significantly in a relatively short period of time. Therefore, by keeping the most recent color images as person-related information, it becomes easier to generate color images that reflect the person's current state.

[0114] The imaging device 20 may be configured with a dedicated imager capable of acquiring at least color images and infrared images. Alternatively, the imaging device 20 may be configured with a high-sensitivity imager capable of acquiring images while suppressing noise even with low light levels. Even with such a high-sensitivity imager, it may be difficult to acquire a clear image in environments where the illuminance inside the vehicle is not above a predetermined value. Therefore, even when the imaging device 20 is configured with such a high-sensitivity imager, applying this disclosure makes it possible to display an image that makes it easy to check the conditions inside the vehicle.

[0115] While this disclosure is described in accordance with the embodiments, it is understood that this disclosure is not limited to those embodiments or structures. This disclosure also encompasses various modifications and variations within the equivalence. In addition, various combinations and forms, as well as other combinations and forms that include only one, more, or fewer of those elements, fall within the scope and concept of this disclosure.

[0116] Furthermore, the control unit and its method described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the control unit and its method described herein may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the control unit and its method described herein may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. The computer program may also be stored in a computer-readable non-transitional tangible storage medium as instructions to be executed by the computer.

[0117] Furthermore, this disclosure includes, in addition to the inventions described in the claims, the following inventions:

[0118] [1] A vehicle image display device comprising: an image acquisition unit (20) that acquires an image of the interior of a vehicle as a display target image; an identification unit (32) that identifies a person in the display target image as a display target person; an information acquisition unit (33) that acquires information related to a person as person-related information; an image generation unit (35) that generates a substitute image by performing image processing on the portion of the display target image in which the display target person is visible, based on the person-related information acquired in advance by the information acquisition unit; and a display unit (36) that is capable of displaying the display target image acquired by the image acquisition unit, and which displays the substitute image in place of the display target image under predetermined conditions.

[0119] [2] The vehicle image display device according to [1], wherein the image acquisition unit acquires an image of the rear seats inside the vehicle as the image to be displayed, and the display unit displays the image to be displayed or the alternative image on a display device (50) that can be seen by the driver of the vehicle.

[0120] [3] The vehicle image display device according to [1] or [2], wherein the image generation unit is capable of performing machine learning based on a data set in which the image to be displayed is input data and the alternative image is output data.

[0121] [4] The vehicle image display device according to any one of [1] to [3], wherein the image acquisition unit acquires a color image of the interior of a vehicle as the display target image, the information acquisition unit acquires person color information related to the color of a person from the color image as the person-related information, and the image generation unit performs image processing on the display target person who is pictured in the display target image based on the person color information previously acquired by the information acquisition unit.

[0122] [5] The vehicle image display device according to [4], wherein the image acquisition unit acquires an infrared image of the interior of the vehicle as the image to be displayed, and the image generation unit generates a colorized image of the infrared image colorized based on the person color information acquired in advance by the information acquisition unit as the substitute image.

[0123] [6] The information acquisition unit further acquires clothing-related information relating to the clothing worn by the person to be displayed in the image to be displayed as person-related information, and the image generation unit excludes the portion of the image to be displayed in which the clothing of the person to be displayed is visible from the image processing if the clothing of the person to be displayed in the image to be displayed does not match the clothing indicated by the clothing-related information previously acquired by the information acquisition unit. [4] or [5] The vehicle image display device according to [4] or [5].

[0124] [7] The vehicle image display device according to any one of [1] to [6], wherein the information acquisition unit acquires the display target image in which the display target person is depicted as the person-related information.

[0125] [8] The vehicle image display device according to [7], wherein the information acquisition unit acquires the display target image acquired by the image acquisition unit as the person-related information in an environment where the illuminance inside the vehicle is above a predetermined value.

[0126] [9] The vehicle image display device according to any one of [1] to [8], wherein the image generation unit is capable of performing the image processing on parts of the display target image other than the part in which the display target person is depicted.

[0127]

[10] The vehicle image display device according to [9], wherein the image generation unit excludes the portion of the image to be displayed that contains the scenery outside the vehicle from the image processing if the image to be displayed contains the scenery outside the vehicle.

[0128]

[11] The vehicle image display device according to any one of [1] to

[10] , wherein the image generation unit performs the image processing when the conditions set by the user are met.

[0129]

[12] The vehicle image display device according to any one of [1] to

[11] , wherein the image generation unit performs the image processing when a person set by the user is identified by the identification unit in the display target image, or when a person corresponding to an attribute set by the user is identified by the identification unit in the display target image.

[0130]

[13] The vehicle image display device according to any one of [1] to

[12] , wherein the image generation unit is capable of detecting the movement of a person inside the vehicle based on a plurality of display target images acquired by the image acquisition unit, and when the movement of a person inside the vehicle is detected, the image processing is performed.

[0131]

[14] The image generation unit repeatedly performs the image processing at predetermined time intervals, as described in any one of [1] to

[13] , for use as an image display device for vehicles.

[0132]

[15] The vehicle image display device according to

[14] , wherein the image generation unit changes the time interval based on the magnitude of the processing load associated with performing the image processing.

[0133]

[16] The vehicle image display device according to

[15] , further comprising a notification unit (38) that notifies that the time interval has been changed when the image generation unit changes the time interval.

[0134]

[17] The vehicle image display device according to any one of

[14] to

[16] , further comprising an estimation unit (39) that estimates the state of a person inside the vehicle based on a plurality of display target images acquired by the image acquisition unit at time intervals shorter than the time interval, wherein the display unit displays state information indicating the state of the person estimated by the estimation unit together with the alternative image.

[0135]

[18] The vehicle image display device according to any one of

[14] to

[17] , further comprising a determination unit (40) that determines whether the eyes of a person inside the vehicle are open or closed based on a plurality of display target images acquired by the image acquisition unit at time intervals shorter than the time interval, wherein the image generation unit performs the image processing on the display target image among the plurality of display target images that shows the greatest degree of eye opening of the person's eyes when the determination unit determines that the eyes of a person inside the vehicle are open.

[0136]

[19] A vehicle image display system (100) comprising an in-vehicle device (101) mounted on a vehicle and an external device (102) capable of data communication with the in-vehicle device, comprising: an image acquisition unit (20) that acquires an image of the interior of a vehicle as a display target image; an identification unit (32) that identifies a person in the display target image as a display target person; an information acquisition unit (33) that acquires information related to a person as person-related information; an image generation unit (35) that generates a substitute image by performing image processing on the portion of the display target image in which the display target person is depicted, based on the person-related information acquired in advance by the information acquisition unit; and a display unit (36) that is capable of displaying the display target image acquired by the image acquisition unit, and which displays the substitute image in place of the display target image under predetermined conditions.

[0137]

[20] A vehicle image display method comprising: an image acquisition step of acquiring an image of the interior of a vehicle as a display target image; an identification step of identifying a person in the display target image as a display target person; an information acquisition step of acquiring information related to a person as person-related information; an image generation step of generating a substitute image by performing image processing on the portion of the display target image in which the display target person is pictured, based on the person-related information acquired in advance by the information acquisition step; and a display step of displaying the display target image acquired by the image acquisition step, which is displayable and, under a predetermined environment, displays the substitute image instead of the display target image.

[0138]

[21] A vehicle image display program that causes the control unit (30) of a vehicle image display device (10) that displays an image of the interior of a vehicle to execute: an image acquisition step of acquiring an image of the interior of a vehicle as a display target image; an identification step of identifying a person in the display target image as a display target person; an information acquisition step of acquiring information related to a person as person-related information; an image generation step of generating a substitute image by performing image processing on the portion of the display target image in which the display target person is shown, based on the person-related information acquired in advance by the information acquisition step; and a display step of displaying the display target image acquired by the image acquisition step, and displaying the substitute image in place of the display target image under a predetermined environment.

Claims

1. A vehicle image display device comprising: an image acquisition unit (20) that acquires an image of the interior of a vehicle as a display target image; an identification unit (32) that identifies a person in the display target image as a display target person; an information acquisition unit (33) that acquires information related to a person as person-related information; an image generation unit (35) that generates a substitute image by performing image processing on the portion of the display target image in which the display target person is visible, based on the person-related information acquired in advance by the information acquisition unit; and a display unit (36) that is capable of displaying the display target image acquired by the image acquisition unit, and which displays the substitute image in place of the display target image under predetermined conditions.

2. The vehicle image display device according to claim 1, wherein the image acquisition unit acquires an image of the rear seats inside the vehicle as the image to be displayed, and the display unit displays the image to be displayed or the alternative image on a display device (50) that can be seen by the driver of the vehicle.

3. The vehicle image display device according to claim 1, wherein the image generation unit is capable of performing machine learning based on a data set in which the image to be displayed is input data and the alternative image is output data.

4. The vehicle image display device according to claim 1, wherein the image acquisition unit acquires a color image of the interior of a vehicle as the display target image, the information acquisition unit acquires person color information related to the color of a person from the color image as the person-related information, and the image generation unit performs image processing on the display target person depicted in the display target image based on the person color information previously acquired by the information acquisition unit.

5. The vehicle image display device according to claim 4, wherein the image acquisition unit acquires an infrared image of the interior of a vehicle as the image to be displayed, and the image generation unit generates a colorized image of the infrared image, which is obtained by colorizing the infrared image based on the person color information acquired in advance by the information acquisition unit, as the substitute image.

6. The vehicle image display device according to claim 4, wherein the information acquisition unit further acquires clothing-related information relating to the clothing worn by the person to be displayed in the image to be displayed, and the image generation unit, if the clothing of the person to be displayed shown in the image to be displayed does not match the clothing indicated by the clothing-related information previously acquired by the information acquisition unit, excludes the portion of the image to be displayed in which the clothing of the person to be displayed is shown from the image processing.

7. The vehicle image display device according to claim 1, wherein the information acquisition unit acquires the display target image in which the display target person is depicted as the person-related information.

8. The vehicle image display device according to claim 7, wherein the information acquisition unit acquires the display target image acquired by the image acquisition unit as the person-related information in an environment where the illuminance inside the vehicle interior is above a predetermined value.

9. The vehicle image display device according to claim 1, wherein the image generation unit is capable of performing the image processing on parts of the display target image other than the part in which the display target person is depicted.

10. The vehicle image display device according to claim 9, wherein if the image to be displayed includes scenery outside the vehicle, the portion of the image to be displayed that includes scenery outside the vehicle is excluded from the image processing.

11. The vehicle image display device according to claim 1, wherein the image generation unit performs the image processing when the conditions set by the user are met.

12. The vehicle image display device according to claim 1, wherein the image generation unit performs the image processing when a person set by the user is identified by the identification unit in the display target image, or when a person corresponding to an attribute set by the user is identified by the identification unit in the display target image.

13. The vehicle image display device according to claim 1, wherein the image generation unit is capable of detecting the movement of a person inside the vehicle based on a plurality of display target images acquired by the image acquisition unit, and performs the image processing when the movement of a person inside the vehicle is detected.

14. The image generation unit repeatedly performs the image processing at predetermined time intervals as described in claim 1 for a vehicle image display device.

15. The vehicle image display device according to claim 14, wherein the image generation unit changes the time interval based on the magnitude of the processing load associated with performing the image processing.

16. The vehicle image display device according to claim 15, further comprising a notification unit (38) that notifies that the time interval has been changed when the image generation unit changes the time interval.

17. The vehicle image display device according to claim 14, further comprising an estimation unit (39) that estimates the state of a person inside a vehicle based on a plurality of display target images acquired by the image acquisition unit at time intervals shorter than the time interval, wherein the display unit displays state information indicating the state of the person estimated by the estimation unit together with the alternative image.

18. The vehicle image display device according to claim 14, further comprising a determination unit (40) that determines whether the eyes of a person inside the vehicle are open or closed based on a plurality of display target images acquired by the image acquisition unit at time intervals shorter than the time interval, wherein the image generation unit performs the image processing on the display target image among the plurality of display target images that shows the greatest degree of eye opening of the person.

19. A vehicle image display system (100) comprising an in-vehicle device (101) mounted on a vehicle and an external device (102) capable of data communication with the in-vehicle device, the system comprising: an image acquisition unit (20) that acquires an image of the interior of a vehicle as a display target image; an identification unit (32) that identifies a person in the display target image as a display target person; an information acquisition unit (33) that acquires information related to a person as person-related information; an image generation unit (35) that generates a substitute image by performing image processing on the portion of the display target image in which the display target person is depicted, based on the person-related information acquired in advance by the information acquisition unit; and a display unit (36) that is capable of displaying the display target image acquired by the image acquisition unit, and which displays the substitute image in place of the display target image under predetermined conditions.

20. A vehicle image display method comprising: an image acquisition step of acquiring an image of the interior of a vehicle as a display target image; an identification step of identifying a person in the display target image as a display target person; an information acquisition step of acquiring information related to a person as person-related information; an image generation step of generating a substitute image by performing image processing on the portion of the display target image in which the display target person is visible, based on the person-related information acquired in advance by the information acquisition step; and a display step of displaying the display target image acquired by the image acquisition step, which is displayable and, under a predetermined environment, displays the substitute image instead of the display target image.

21. A vehicle image display program that causes the control unit (30) of a vehicle image display device (10) that displays images of the interior of a vehicle to execute: an image acquisition step of acquiring an image of the interior of a vehicle as a display target image; an identification step of identifying a person in the display target image as a display target person; an information acquisition step of acquiring information related to a person as person-related information; an image generation step of generating a substitute image by performing image processing on the portion of the display target image in which the display target person is shown, based on the person-related information acquired in advance by the information acquisition step; and a display step of displaying the display target image acquired by the image acquisition step, and displaying the substitute image in place of the display target image under predetermined conditions.

Images