Information processing device, information processing method, and information processing system

Abstract

The present invention performs, for example, prediction related to a failure of a sensor.　This information processing device comprises a failure prediction unit that receives, as an input, evaluation information related to normality of a sensor, and outputs a prediction result related to a failure of the sensor.

Description

Information Processing Apparatus, Information Processing Method, and Information Processing System

[0001] The present technology relates to an information processing apparatus, an information processing method, and an information processing system.

[0002] When a defect (abnormality) occurs in a part of a solid-state imaging device, a defect appears in the captured image due to the defect. For example, in Patent Document 1 below, it is described that when a defect such as a disconnection occurs in a pixel signal line vsl of a solid-state imaging device, a line defect occurs in the captured image.

[0003] International Publication No. 2020 / 158400

[0004] The technology described in Patent Document 1 can detect a failure of a solid-state imaging device based on a captured image. However, in the technology described in Patent Document 1, the solid-state imaging device has failed when a line defect of the imaging device is detected. That is, there is room for improvement in that it is not possible to predict in advance the timing when a failure occurs, rather than the failure of the solid-state imaging device.

[0005] One object of the present technology is to provide an information processing apparatus, an information processing method, and an information processing system that can predict a failure of a sensor such as an imaging sensor.

[0006] The present technology is, for example, an information processing apparatus including a failure prediction unit that takes evaluation information regarding the normality of a sensor as an input and outputs a prediction result regarding a failure of the sensor.

[0007] The present technology is, for example, an information processing method in which a failure prediction unit takes evaluation information regarding the normality of a sensor as an input and outputs a prediction result regarding a failure of the sensor.

[0008] The present technology is, for example, an information processing system including a sensor and an application processor connected to the sensor, and the application processor includes a failure prediction unit that takes evaluation information regarding the normality of the sensor as an input and outputs a prediction result regarding a failure of the sensor.

[0009] This is a block diagram illustrating an example configuration of an information processing system according to the first embodiment. This diagram is referenced when explaining an example of a learning method for the learning model used in the anomaly detection unit according to the embodiment. This diagram is referenced when explaining the process by which the anomaly detection unit according to the embodiment generates evaluation image data. This diagram illustrates an example of the history of evaluation values. This diagram is referenced when explaining the process performed by the failure prediction unit according to the embodiment. This diagram is referenced when explaining the process performed by the defect location classification unit according to the embodiment. This diagram illustrates that a defect location classification model is prepared for each type of CIS. This is a flowchart illustrating the flow of processing performed by the information processing system according to the first embodiment. A to D are diagrams referenced when explaining the flow of processing performed by the information processing system according to the first embodiment. A and B are diagrams illustrating examples of information presentation. This diagram illustrates an example of mode switching. This is a block diagram illustrating an example configuration of an information processing system according to the second embodiment. A to C are diagrams illustrating an example of classification results according to the second embodiment. This diagram illustrates an application example. This diagram illustrates an application example. This diagram illustrates an application example. This diagram illustrates an application example.

[0010] The embodiments of this technology will be described below with reference to the drawings. The description will proceed in the following order: <First Embodiment> <Second Embodiment> <Modified Examples> <Application Examples>

[0011] <First Embodiment> [Example of Information Processing System Configuration] (Overview) Figure 1 shows an example of the configuration of an information processing system according to the first embodiment (hereinafter referred to as information processing system 1A as appropriate). Information processing system 1A includes at least a CIS (CMOS Image Sensor) 10 and an application processor (AP) 20 connected to the CIS 10. Information processing system 1A according to this embodiment further includes a notification unit 30. Information processing system 1A can be configured, for example, as an in-vehicle sensor. For example, based on the image data captured by the CIS 10, objects such as people or other vehicles are detected, and vehicle control is performed according to the detection result.

[0012] The imaging sensor CIS 10 is an example of a sensor or imaging sensor in this technology. However, this technology can also be applied to sensors other than imaging sensors. The application processor 20 is connected downstream of the CIS 10. The application processor 20 is an example of an information processing device in this technology. The information processing device in this technology may include not only the application processor 20 but also a notification unit 30. In general terms, the application processor 20 predicts a failure of the CIS 10 based on the image data obtained by the CIS 10. Here, a failure of the CIS 10 means that even if signal processing correction is performed, the CIS 10 cannot perform its function, in other words, the CIS 10 has failed to the extent that image data usable by an in-vehicle sensor cannot be obtained, and the CIS 10 needs to be repaired or replaced. The notification unit 30 is connected downstream of the application processor 20.

[0013] (CIS) Details of each component constituting the information processing system 1A will be described. First, the details of the CIS 10 will be described. The CIS 10 includes, for example, a pixel unit 11, an ADC (Analog to Digital Converter) 12 connected downstream of the pixel unit 11, an ISP (Image Signal Processor) 13 connected downstream of the ADC 12, and an output unit 14.

[0014] An imaging optical system (not shown) guides light from the subject (incident light), and the light is focused onto the light-receiving surface of the pixel section 11. The imaging optical system includes lenses such as a cover lens, a zoom lens, and a focus lens, as well as an iris mechanism. This imaging optical system may also include actuators and drive circuits for each of these zoom lenses, focus lenses, and iris mechanisms.

[0015] The pixel unit 11 comprises a pixel array unit in which multiple pixels are arranged two-dimensionally, and a readout circuit. Each pixel includes a photoelectric conversion element such as a photodiode (PD) or a floating diffusion (FD). The readout circuit reads out the electrical signal obtained by photoelectric conversion from each pixel in the pixel array unit. A known circuit configuration can be applied to the pixel unit 11. The electrical signal obtained in the pixel unit 11 is output from the pixel unit 11 as image data.

[0016] The captured image data output from the pixel unit 11 is subjected to processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) by a readout circuit (not shown).

[0017] The ADC12 performs A / D (Analog / Digital) conversion processing to convert analog image data, which has undergone CDS processing and other treatments, into digital image data. The digitally converted image data is supplied from the ADC12 to the ISP13.

[0018] ISP13 is an image processing processor that performs various image signal processing on captured image data supplied from ADC12. ISP13 performs known signal processing on captured image data as digital data, such as preprocessing, syncing, YC generation, and resolution conversion.

[0019] In the preprocessing stage, clamping is performed on the captured image data to clamp the R, G, and B black levels to predetermined levels, and correction processing is performed between the R, G, and B color channels. In the simulcasting stage, color separation processing is performed so that the image data for each pixel contains all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaicing is performed as the color separation process. In the YC generation stage, luminance (Y) signals and color (C) signals are generated (separated) from the R, G, and B image data. In the resolution conversion stage, resolution conversion processing is performed on the image data that has undergone various signal processing steps.

[0020] Image data that has undergone image signal processing is supplied from ISP 13 to output unit 14. ISP 13 may perform other known image signal processing besides the processing described above. For example, ISP 13 may perform codec processing to generate files in formats known as video file formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264, or files known as still image file formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format). These image signal processing processes may also be AI image processing performed by applying a trained model.

[0021] The output unit 14 is an interface that outputs the captured image data, which has undergone image signal processing by the ISP 13, to the outside of the CIS 10. The captured image data, as digital data, is output to, for example, the application processor 20 via the output unit 14. The captured image data output via the output unit 14 is also output as appropriate to a processor (not shown) that performs the functions of an in-vehicle sensor, such as the aforementioned person and object detection.

[0022] The CIS 10 may output the captured image data for all frames to the application processor 20, or it may output the captured image data to the application processor 20 every few frames (for example, every 100 frames).

[0023] (Application Processor) Next, the details of the application processor 20 will be described. The application processor 20 includes, for example, an input unit 21, an anomaly detection unit 22 (an example of a detection unit), a memory 23, a failure prediction unit 24, an output unit 25, and a defect location classification unit 26. The anomaly detection unit 22 is connected downstream of the input unit 21. The memory 23 is connected downstream of the anomaly detection unit 22. The failure prediction unit 24 is connected downstream of the memory 23. The output unit 25 is connected downstream of the failure prediction unit 24. In addition, the defect location classification unit 26 is connected downstream of the anomaly detection unit 22. The output unit 25 is also connected downstream of the defect location classification unit 26.

[0024] The input unit 21 is an interface for receiving data supplied to the application processor 20. For example, the input unit 21 receives captured image data output from the output unit 14 of the CIS 10. The input unit 21 supplies the captured image data output from the output unit 14 of the CIS 10 to the anomaly detection unit 22 as input image data.

[0025] The anomaly detection unit 22 applies a learning model to the captured image data input from the input unit 21 to generate evaluation image data that includes at least the defective parts caused by the defective parts of the CIS 10. This evaluation image data is supplied to the defective part classification unit 26.

[0026] Here, the defective parts of the CIS10 refer to defects in some floating diffusion, disconnections in some vertical signal lines (hereinafter referred to as vertical signal lines VSL or VSL as appropriate), and defects in some reset transistors (hereinafter referred to as reset transistors RST or RST as appropriate). The defective parts that appear in the evaluation image data due to these defective parts are of a degree that cannot be corrected by image signal processing by ISP13. However, if the defective parts are below a certain level, there is no problem with using it as an in-vehicle sensor. However, if the defective parts exceed a certain level, it will be difficult to use it as an in-vehicle sensor, and repair or replacement of the CIS10 will be necessary. In other words, if the defective parts exceed a certain level, the CIS10 will be in a malfunction state.

[0027] The anomaly detection unit 22 includes an evaluation value generation unit 22A as a processing block. The evaluation value generation unit 22A of the anomaly detection unit 22 generates evaluation values ​​based on evaluation image data. The evaluation value generation unit 22A supplies the generated evaluation values ​​to the memory 23.

[0028] Memory 23 stores the history of evaluation values ​​generated by the evaluation value generation unit 22A. The history of evaluation values ​​refers to, for example, the temporal changes in evaluation values. Examples of memory 23 include HDD (Hard Disk Drive) and SSD (Solid State Drive). Memory 23 may also be a removable storage medium such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0029] The history of evaluation values ​​is input to the failure prediction unit 24 as evaluation information regarding the health of the CIS 10. The failure prediction unit 24 may read the history of evaluation values ​​stored in the memory 23. Alternatively, the anomaly detection unit 22 may have the memory function of the memory 23. In this case, the anomaly detection unit 22 may output the history of evaluation values ​​to the failure prediction unit 24 at an appropriate timing.

[0030] The failure prediction unit 24 takes evaluation information regarding the health of the CIS 10 as input and outputs a prediction result regarding the failure of the CIS 10. This process is performed, for example, using a learning model that takes evaluation information regarding the health of the CIS 10 as input and outputs a prediction result regarding the failure of the CIS 10. The prediction result from the failure prediction unit 24 is supplied to the output unit 25.

[0031] The output unit 25 is an interface that outputs the prediction results from the fault prediction unit 24 to the outside of the application processor 20. For example, the prediction results from the fault prediction unit 24 are output to the notification unit 30 via the output unit 25.

[0032] The defect location classification unit 26 applies a learning model to the evaluation image data supplied from the anomaly detection unit 22, and outputs a classification result that classifies the defective locations of the CIS 10. The classification result from the defect location classification unit 26 is output to the notification unit 30 via the output unit 25.

[0033] The components of the application processor 20 can be changed as appropriate. For example, the functions of the anomaly detection unit 22 and the defective location classification unit 26 may be integrated. A single component with integrated functions may generate evaluation image data, evaluation values, and classification results by applying a predetermined learning model to the captured image data.

[0034] (Notification Unit) The notification unit 30 is a general term for devices that notify users of various types of information, for example, of the information processing system 1A. The notification unit 30 converts the prediction results input from the application processor 20 into information based on the prediction results that the user can perceive and presents it. The notification unit 30 also converts the classification results input from the application processor 20 into information based on the classification results that the user can perceive and presents it.

[0035] Examples of notification units 30 include display devices and audio output devices such as speakers. Examples of display devices include LCDs (Liquid Crystal Displays) and organic EL (Electro-Luminescence) panels. When the information processing system 1A is applied as an in-vehicle sensor, an example of a notification unit 30 is a display installed in the center cluster inside the vehicle.

[0036] [Regarding Evaluation Image Data] Next, an example of evaluation image data will be described. First, referring to Figure 2, the learning method of the learning model (AI learning model) used in the anomaly detection unit 22 will be explained.

[0037] As shown in Figure 2, during the learning phase, a set of input image data is prepared as training data. Preferably, each image data constituting the input image data set is an image of a real scene, and a large amount of it is prepared. Image data can be acquired via the internet. For example, normal image data of a real scene can be acquired by utilizing satellite images posted on a specific website or video from a dashcam. Furthermore, if location information such as GPS (Global Positioning System) is associated with the image data, the acquired image data can be classified by driving area based on this location information. Note that image data may also be acquired without using the internet. During the learning phase, only image data of real scenes that does not contain defects caused by faulty parts of the imaging sensor is used for training.

[0038] The learning model used in the anomaly detection unit 22 (hereinafter referred to as the image restoration model as appropriate) compresses the training image data and then learns to accurately restore the image data from it. As a result, even if the input image data contains defective parts caused by a faulty part of the imaging sensor, the image restoration model can output normal image data that does not contain the defective parts as the restored image data.

[0039] The anomaly detection unit 22 generates evaluation image data using the image reconstruction model constructed through the learning process described above. As shown in Figure 3, an example of input image data, the input image data IMA, is input to the anomaly detection unit 22 via the input unit 21.

[0040] The input image data IMA is the image data supplied from the CIS 10. The input image data IMA includes a defective portion A1 caused by a fault in the CIS 10. The defective portion A1 is a defect that cannot be removed by image signal processing by the ISP 13. If the area of ​​the defective portion A1 is small, it does not impair the function of the in-vehicle sensor. However, if the area of ​​the defective portion A1 exceeds a certain level, in other words, if the number of faulty locations causing the defective portion A1 or the output value of the faulty locations exceeds a certain level, it becomes difficult to accurately detect people or objects using the input image data IMA. In other words, the CIS 10 malfunctions.

[0041] The anomaly detection unit 22 applies a pre-trained image restoration model to the input image data IMA. This results in restored image data IMB, which is the restored version of the input image data IMA. As described above, the image restoration model can restore the input image data IMA with high accuracy. Therefore, the restored image data IMB is image data in which the defective portion A1 has been removed and restored.

[0042] Next, the anomaly detection unit 22 generates image data IMC by calculating the difference between the input image data IMA and the reconstructed image data IMB. This image data IMC is image data in which only the defective portion A1, which is the difference between the two image data, remains. The anomaly detection unit 22 outputs at least this image data IMC as evaluation image data. Hereinafter, the image data IMC will be appropriately referred to as evaluation image data IMC. In this embodiment, the anomaly detection unit 22 supplies the evaluation image data IMC to the defect location classification unit 26.

[0043] The method for generating evaluation image data IMC is not limited to the method described above. For example, evaluation image data IMC may be generated directly from input image data IMA using a learning model that has learned about defects themselves, such as abrupt changes in pixel values ​​or the characteristics of the defective parts (details of which will be described later).

[0044] [Regarding the evaluation value] As shown in FIG. 3, the evaluation value generation unit 22A generates an evaluation value based on the evaluation image data IMC. The evaluation value is, for example, a value obtained by evaluating by converting the defective portion A1 included in the evaluation image data IMC. In other words, it can also be said to be a characteristic value indicating the characteristics of the defective portion A1. For example, the number of pixels included in the defective portion A1, the integrated value of the pixel values of the pixels included in the defective portion A1, the square mean value of the pixel values of the pixels included in the defective portion A1, the absolute mean value of the pixel values of the pixels included in the defective portion A1, combinations of these, etc. can be used as the evaluation value.

[0045] The evaluation value generation unit 22A writes the generated evaluation value into the memory 23. As described above, the memory 23 stores the history of the evaluation values. The history of the evaluation values is, for example, the temporal change of the evaluation values.

[0046] FIG. 4 shows an example of the history of the evaluation values stored in the memory 23. The horizontal axis in FIG. 4 indicates time, and the vertical axis indicates the evaluation value. Generally, the CIS 10 undergoes aging deterioration due to long-term use, and the number and area of defective portions included in the captured image data increase. Therefore, when the evaluation value is a value obtained by converting the defective portion as in this embodiment, as shown in FIG. 4, the evaluation value tends to increase with the passage of time. Of course, the temporal change of the evaluation value shown in FIG. 4 is an example, and there may be a period during which the evaluation value hardly changes or a period during which it increases rapidly depending on the case.

[0047] [Processing of the failure prediction unit] Next, an example of the processing performed by the failure prediction unit 24 will be described. As shown in FIG. 5, the failure prediction unit 24 sets two threshold values for the above-described history of the evaluation values. The two threshold values are a failure threshold value and a prediction threshold value set to a value smaller than the failure threshold value. The two threshold values are stored in the memory 23, for example, and the failure prediction unit 24 can appropriately read these threshold values from the memory 23. The failure threshold value is a threshold value for determining that the CIS 10 has failed because the evaluation value is large, that is, the defective portion is large and there are many defective locations in the CIS 10. When the evaluation value reaches the failure threshold value, it is determined that the CIS 10 has failed.

[0048] The prediction threshold is a threshold for the failure prediction unit 24 to output a prediction result. When the evaluation value exceeds the prediction threshold, the failure prediction unit 24 outputs the prediction result to the notification unit 30 via the output unit 25. The case where the evaluation value exceeds the prediction threshold may be either the case where the evaluation value is equal to or greater than the prediction threshold or the case where the evaluation value is greater than the prediction threshold. The case where the evaluation value exceeds the prediction threshold indicates that although it cannot be said that the CIS 10 has failed, it is predicted that the failure will occur in a relatively short time. Therefore, the failure prediction unit 24 outputs, as a prediction result, that the CIS 10 will soon fail. The failure prediction unit 24 may perform an operation based on the width from the prediction threshold to the failure threshold and the degree of change in the evaluation value to obtain the specific time until the CIS 10 fails. Then, the failure prediction unit 24 may output, as a prediction result, the specific time (e.g., one week later, one month later, or specific date and time, etc.) until the CIS 10 fails.

[0049] Note that the failure threshold and the prediction threshold may be obtained from a cloud server or the like, for example, by the application processor 20 performing communication. Also, the application processor 20 may communicate with the CIS 10, and the application processor 20 may obtain the failure threshold and the prediction threshold from the CIS 10.

[0050] [Processing of the defective part classification unit] Next, referring to FIG. 6, a processing example performed by the defective part classification unit 26 will be described. As shown in FIG. 6, restored image data IMB is obtained by applying an image restoration model to the input image data IMA. Then, evaluation image data IMC is generated by taking the difference between the input image data IMA and the restored image data IMB. The processing up to this point is performed by the abnormality detection unit 22 as described above. The evaluation image data IMC is supplied from the abnormality detection unit 22 to the defective part classification unit 26.

[0051] The defect location classification unit 26 applies a learned model to the evaluation image data IMC. This learned model (hereinafter referred to as the defect location classification model) takes the evaluation image data IMC as input and outputs the defect locations of the CIS 10. The defect location classification model detects the presence or absence of defective parts in the input evaluation image data IMC, and if a defective part such as defective part A1 exists in the evaluation image data IMC, it classifies the defective part according to the characteristics of the abnormality. For example, the defective part is classified into "FD defect," "VSL defect," and "RST defect" according to the characteristics of the defective part.

[0052] Incidentally, the characteristics of the defective parts in the evaluation image data IMC are related to the defective locations in the CIS 10, for example, in the pixel area 11. In other words, the defective locations in the CIS 10 can be estimated based on the characteristics of the defective parts. The defective location classification model estimates the defective locations in the CIS 10.

[0053] For example, if the characteristic of a defective area in the evaluation image data IMC is a white dot-shaped "FD defect," then it is often the case that a defect has occurred in some of the floating diffusion of the pixel portion 11. Therefore, if the defective area included in the evaluation image data IMC is an "FD defect," the floating diffusion is estimated to be the defective area. And, since a defect has occurred in some of the floating diffusion at present, as shown in Figure 6, the floating diffusion is estimated to be the specific location of the failure expected in the future.

[0054] For example, if the characteristic of a defect in the evaluation image data IMC is a vertical linear defect called a "VSL defect," then it is often the case that some of the vertical signal lines VSLs in the pixel section 11 have defects such as disconnections. Therefore, if the defect in the evaluation image data IMC is a "VSL defect," the vertical signal lines VSLs are estimated to be the defective areas. And, since some of the vertical signal lines VSLs are currently defective, as shown in Figure 6, the vertical signal lines VSLs are estimated to be the specific areas where failures are expected in the future.

[0055] For example, if the characteristic of a defect in the evaluation image data IMC is a horizontal linear defect called an "RST defect," then it is often the case that some of the reset transistors RST in the pixel section 11 are faulty. Therefore, if the defect in the evaluation image data IMC is an "RST defect," the reset transistors RST are estimated to be the faulty location. And, since some of the reset transistors RST are currently faulty, as shown in Figure 6, the reset transistors RST are estimated to be the specific location of the failure expected in the future.

[0056] The fault location classification unit 26 outputs the classification result, which indicates the specific location of the failure expected in the future, to the notification unit 30 via the output unit 25.

[0057] Furthermore, the detection and classification of defective parts, and the estimation of defective locations, may be performed in a processing block different from the defective location classification unit 26. Also, the detection and classification of defective parts and the estimation of defective locations may be performed using separate learning models.

[0058] Furthermore, the characteristics of the defective areas may vary depending on structural differences such as the size of the CIS10. For example, in one type of sensor CIS10, the size of the "FD defect" may be 2 x 4 pixels, while in another type of sensor CIS10, the size of the "FD defect" may be 2 x 2 pixels. The characteristics of "VSL defects" and "RST defects" may also vary depending on the sensor type of the CIS10.

[0059] Therefore, a defect location classification model may be prepared for each sensor type of CIS10. For example, as shown in Figure 7, if there are sensor types TA, TB, and TC for CIS10, a defect location classification model MA corresponding to sensor type TA, a defect location classification model MB corresponding to sensor type TB, and a defect location classification model MC corresponding to sensor type TC may be prepared.

[0060] The defect location classification model MA is a model that has learned the defect locations that may be included in the image data captured by the CIS10, which has a sensor type of sensor type TA. The defect location classification model MB is a model that has learned the defect locations that may be included in the image data captured by the CIS10, which has a sensor type of sensor TB. The defect location classification model MC is a model that has learned the defect locations that may be included in the image data captured by the CIS10, which has a sensor type of sensor TC.

[0061] The user may be able to select which of the three defect location classification models to use. Alternatively, the application processor 20 may communicate with the CIS 10 to obtain the sensor type of the CIS 10 and automatically select a defect location classification model corresponding to the obtained sensor type. In this case, the application processor 20 may download the defect location classification model corresponding to the sensor type of the CIS 10 from an external device such as a cloud server. The downloaded defect location classification model is then deployed to the defect location classification unit 26 as appropriate.

[0062] [Processing flow in the information processing system] Next, the processing flow in the information processing system 1A will be explained with reference to the flowchart in Figure 8 and Figures 9A to 9D.

[0063] In step ST1, the CIS 10 performs image data acquisition processing. For example, light captured by the imaging optical system is imaged onto the light-receiving surface of the pixel unit 11, and the light is converted into an electrical signal to acquire image data. After CDS processing, AGC processing, etc. are performed on the image data, A / D conversion processing by the ADC 12 yields digital image data. The digital image data is the RAW data schematically shown in Figure 9A. Then the processing proceeds to step ST2.

[0064] In step ST2, the ISP 13 performs image signal processing on the digital image data (RAW data). An example of image signal processing has been described above, so a redundant explanation will be omitted. This yields the image data (developed image) schematically shown in Figure 9B. The image data is output from the output unit 14 and input to the application processor 20 via the input unit 21. The image data is then input to the anomaly detection unit 22. The process then proceeds to step ST3.

[0065] In step ST3, the anomaly detection unit 22 performs an anomaly detection process. During the anomaly detection process, the anomaly detection unit 22 generates evaluation image data IMC. As a result of the anomaly detection process, evaluation image data IMC including the horizontal linear defect portion A2, as schematically shown in Figure 9C, is generated. Then, the process proceeds to step ST4.

[0066] In step ST4, the evaluation value generation unit 22A performs the evaluation value generation process. The evaluation value generation unit 22A generates evaluation values ​​based on the evaluation image data IMC. The evaluation value generation unit 22A stores the generated evaluation values ​​in the memory 23. The memory 23 stores the history of evaluation values. Then, the process proceeds to step ST5.

[0067] In step ST5, the defect location classification unit 26 performs defect location classification processing. The defect location classification unit 26 detects and classifies defective parts based on the evaluation image data IMC. The defect location classification unit 26 then estimates the defective location of the CIS 10 corresponding to each classified defective part and generates a classification result corresponding to the estimated defect result. For example, the defect location classification unit 26 generates a classification result that includes information that the estimated defective location is a failure location that is predicted (expected) to occur in the future. The defect location classification unit 26 supplies the generated classification result to the output unit 25. Then the process proceeds to step ST6.

[0068] In step ST6, the failure prediction unit 24 performs failure prediction processing. Failure prediction processing is a process in which the failure prediction unit 24 generates prediction results regarding the failure of the CIS 10. For example, the failure prediction unit 24 uses the current evaluation value and the evaluation value corresponding to the failure threshold to generate a prediction result of the failure time at which the CIS 10 is expected to fail. Then the process proceeds to step ST7.

[0069] In step ST7, the failure prediction unit 24 determines whether or not to output a prediction result. For example, as shown in Figure 9D, if the latest evaluation value in the evaluation value history (the evaluation value generated in the evaluation value generation process in step ST4) exceeds the prediction threshold, the failure prediction unit 24 determines to output a prediction result.

[0070] If the evaluation value does not exceed the predicted threshold (i.e., the result of step ST7 is No), the process returns to step ST1. If the evaluation value exceeds the predicted threshold (i.e., the result of step ST7 is Yes), the process proceeds to step ST8.

[0071] In step ST8, the prediction result output process is performed. In the prediction result output process, the fault prediction unit 24 outputs the prediction result. The prediction result is output to the notification unit 30 via the output unit 25. The notification unit 30 notifies the information based on the prediction result by display or sound.

[0072] Although not shown in Figure 8, the classification results generated by the defect classification unit 26 may be output to the notification unit 30 via the output unit 25. The notification unit 30 may then notify the information based on the classification results by display or sound.

[0073] The processing performed by the anomaly detection unit 22 and the processing performed by the defect location classification unit 26 may be performed in parallel rather than in series, as shown in Figure 8.

[0074] The failure prediction unit 24 may generate a prediction result only when the evaluation value exceeds the prediction threshold. Furthermore, the output unit 25 may output a classification result along with the prediction result only when a prediction result has been output.

[0075] [Example of Information Presentation] Next, an example of information presentation by the notification unit 30 will be described with reference to Figures 10A and 10B. In this example, the notification unit 30 will be described as a display device.

[0076] The notification unit 30 displays information based on the prediction results of the failure prediction unit 24. This information includes, for example, information about the timing of the failure and information about advice. When the failure prediction unit 24 outputs a prediction result, it indicates that the failure of the CIS 10 is relatively imminent. For example, as shown in Figure 10A, the notification unit 30 displays the message "The on-board sensor will fail soon" as information about the timing of the failure. The notification unit 30 also displays the message "We recommend you get your vehicle inspected" as information about advice. If the prediction result is a specific time when the failure is expected to occur, the advice may include a specific time, such as "We recommend you get your vehicle inspected by [year] [month]."

[0077] Furthermore, information based on classification results may be displayed in addition to information based on prediction results. Classification results may include, for example, information (specific locations) of locations where failures are predicted to occur in the future, based on the locations of defects at present. The name of the location where a failure is predicted to occur in the future (in the near future) may be announced, but in many cases, ordinary users do not know the specific location of the failure corresponding to that name. Therefore, an error code corresponding to the location where a failure is predicted to occur in the future may be announced. For example, as shown in Figure 10B, the error code corresponding to the location where a failure is predicted to occur in the future may be displayed as, "The error code is "xxx"...". Users can communicate the location where a failure is predicted to occur to the vehicle inspection company, etc., simply by telling them the error code when they go for a vehicle inspection.

[0078] [About the Modes] The information processing system 1A may be configured to allow the setting of a normal drive mode and a fault prediction mode. The normal drive mode is a mode in which the captured image data output from the CIS 10 is used only to perform the normal functions of the in-vehicle sensor (object detection, person detection, etc.). In normal drive mode, the captured image data obtained by the CIS 10 is not output to the application processor 20, but is output to a processing block that performs the normal functions of the in-vehicle sensor.

[0079] The failure prediction mode is a mode in which the captured image data output from the CIS 10 is used solely for executing the functions of the application processor 20 (functions performed by the anomaly detection unit 22, failure prediction unit 24, and fault location classification unit 26, etc., as described above). In failure prediction mode, the captured image data obtained by the CIS 10 is output to the application processor 20 instead of to the processing block that performs the normal functions of the in-vehicle sensor.

[0080] For example, as shown in Figure 11, the normal drive mode and the failure prediction mode are switched for each frame. That is, one frame of captured image data is input to the application processor 20 once every two frames. Based on the input captured image data, the anomaly detection unit 22 generates evaluation image data (IMC). In this way, the anomaly detection unit 22 intermittently generates evaluation image data (IMC).

[0081] The failure prediction mode can be set as appropriate, such as once every three frames instead of once every two frames. Alternatively, the captured image data for one frame may be branched and input to the processing block that performs the normal functions of the on-board sensor and to the application processor 20. Furthermore, the processing for each mode may be performed in parallel. In addition, the processing related to the failure prediction mode may be performed only at appropriate timings, such as when the vehicle equipped with the on-board sensor is started.

[0082] As described above, this embodiment makes it possible to predict sensor failures. Users can recognize when a sensor is about to fail.

[0083] <Second Embodiment> Next, a second embodiment will be described. In the description of the second embodiment, the same or identical components as described above will be denoted by the same reference numerals, and redundant descriptions will be omitted as appropriate. Unless otherwise specified, the matters described in the first embodiment can be applied to the second embodiment.

[0084] Figure 12 shows an example configuration of an information processing system according to the second embodiment (hereinafter referred to as information processing system 1B). The following explanation will focus on the differences between information processing system 1B and information processing system 1A.

[0085] The anomaly detection unit 22 of the application processor 20 does not have an evaluation value generation unit 22A. Furthermore, no memory 23 is connected downstream of the anomaly detection unit 22; instead, a defect location classification unit 26 is connected. A memory 23 is connected downstream of the defect location classification unit 26, rather than an output unit 25.

[0086] Similar to the first embodiment, the anomaly detection unit 22 generates evaluation image data IMC. The anomaly detection unit 22 then supplies the evaluation image data IMC to the defect location classification unit 26.

[0087] The defect location classification unit 26 generates classification results by applying a defect location classification model to the evaluation image data IMC. According to the defect location classification model in this embodiment, the classification results include the predicted fault locations and the number of defective locations. The defect location classification unit 26 then writes the generated classification results to the memory 23.

[0088] As described in the first embodiment, the defect classification model can detect defective parts and then classify them into "FD defects," "VSL defects," and "RST defects." Furthermore, it can estimate the location of the defective parts in the CIS 10 corresponding to each type of defect.

[0089] For example, if an "FD defect" is detected in the evaluation image data IMC, it can be estimated that the defective area is a portion of floating diffusion. Also, if there are three "FD defects" in the evaluation image data IMC, it can be estimated that there are three floating diffusion defects. If the number of floating diffusion defects is below a certain number, the missing pixels can be corrected by image signal processing, etc. However, if the number of floating diffusion defects is above a certain number, it cannot be corrected by image signal processing, etc. In other words, the function of the in-vehicle sensor cannot be realized, and the CIS10 will have failed.

[0090] For example, if a "VSL defect" is detected in the evaluation image data IMC, it can be estimated that the defective area is a portion of the vertical signal line VSL. Also, if there are three "VSL defects" in the evaluation image data IMC, it can be estimated that there are three defective vertical signal line VSLs. If the number of defective vertical signal line VSLs is below a certain number, the missing pixels can be corrected by image signal processing, etc. However, if the number of defective vertical signal line VSLs is above a certain number, it cannot be corrected even by image signal processing, etc. In other words, the function of the in-vehicle sensor cannot be realized, and the CIS10 will have failed.

[0091] For example, if an "RST defect" is detected in the evaluation image data IMC, it can be estimated that the faulty area is one of the reset transistors (RST). Also, if there are three "RST defects" in the evaluation image data IMC, it can be estimated that there are three faulty reset transistors (RST). If the number of faulty reset transistors (RST) is below a certain number, the missing pixels can be corrected by image signal processing, etc. However, if the number of faulty reset transistors (RST) is above a certain number, it cannot be corrected by image signal processing, etc. In other words, the function of the in-vehicle sensor cannot be realized, and the CIS10 will have failed.

[0092] Thus, according to the defect classification model of this embodiment, the number of defects for each defect location can be output by considering the number of defective parts included in the evaluation image data IMC.

[0093] Memory 23 stores the classification results, specifically the specific locations where failures are expected to occur in the future according to the faulty locations, and the history of those faulty locations. More specifically, the history of faulty locations is the number of faulty locations. As the processing of the faulty location classification unit 26 is repeated, the temporal change in the number of faulty locations is stored in memory 23 for each faulty location at each time point. These classification results are input to the failure prediction unit 24 as evaluation information regarding normality.

[0094] Figures 13A to 13C show an example of the classification results stored in the memory 23 according to this embodiment. Figure 13A shows the temporal change in the number of floating diffusion defects. Generally, as CIS 10 deteriorates over time, the number of floating diffusion defects increases over time. Figure 13B shows the temporal change in the number of vertical signal line VSL defects. Similar to floating diffusion, as CIS 10 deteriorates over time, the number of vertical signal line VSL defects increases over time. Figure 13C shows the temporal change in the number of reset transistor RST defects. Similar to floating diffusion, as CIS 10 deteriorates over time, the number of reset transistor RST defects increases over time.

[0095] Similar to the first embodiment, a failure threshold and a prediction threshold are set for the number of defects. The failure threshold and prediction threshold may be the same for all defective locations, or they may differ for each defective location. For example, if the number of defects in the floating diffusion reaches the failure threshold, the CIS 10 will no longer be able to perform its function. Also, if the number of defects in the floating diffusion exceeds the prediction threshold, the CIS 10 will no longer be able to perform its function in the near future. The same applies to the vertical signal line VSL and the reset transistor RST.

[0096] The failure prediction unit 24 inputs the history of the number of defects for each component included in the classification result, and outputs a prediction result when the number of defects for any component exceeds the prediction threshold. The prediction result can be the same as in the first embodiment. The prediction result is notified by the notification unit 30 as appropriate. For example, the prediction result may display the message, "The on-board sensor will fail soon. We recommend that you take your vehicle for inspection." Thus, the information based on the prediction result may include prediction results regarding floating diffusion (see Figure 13A) where the number of defective parts exceeds the prediction threshold. A user who recognizes such notification may, for example, go to a vehicle inspection shop and request replacement or repair of the CIS 10.

[0097] The same effects as those of the first embodiment can be obtained with this embodiment as well. Furthermore, according to this embodiment, when a failure of an in-vehicle sensor is predicted, the user can also be notified of the component causing the failure.

[0098] <Modifications> Although several embodiments of this technology have been described in detail above, the content of this technology is not limited to the embodiments described above, and various modifications are possible based on the technical concept of this technology. Modifications will be described below.

[0099] In the embodiment described above, image data processed by the ISP 13 was output from the output unit 14 as captured image data. However, RAW data may also be output from the output unit 14. The captured image data in RAW format may then be input to the application processor 20, and the processing described in the embodiment may be performed on the captured image data. Captured image data in RAW format is generated, for example, by performing CDS processing, AGC processing, etc., on the captured image data obtained by the pixel unit 11, followed by A / D conversion processing by the ADC 12.

[0100] In the embodiments described above, the evaluation value was explained as a value obtained by transforming the defective portion included in the evaluation image data, in other words, a characteristic value indicating the characteristics of the defective portion. However, the evaluation value may also be a value obtained by transforming the normal portion included in the evaluation image data, in other words, a characteristic value indicating the characteristics of the normal portion. This evaluation value may also be evaluation information regarding the normality of the sensor. The history of such evaluation values ​​may also be evaluation information regarding the normality of the sensor. In this example, contrary to the embodiments, for example, if the evaluation value falls below the prediction threshold, the failure prediction unit 24 outputs a prediction result.

[0101] In the configuration examples of information processing systems 1A and 1B, the notification unit 30 may not be included. Also, some or all of the functions of the application processor 20 may be implemented by external devices such as a cloud server. Furthermore, the CIS 10 and the application processor 20 may be integrated. That is, the functions of the application processor 20 may be implemented within the CIS 10.

[0102] In the above-described embodiment, if the evaluation value or the number of failures is smaller than the prediction threshold, the failure prediction unit 24 did not output a prediction result. However, it may output a prediction result indicating that there is sufficient time before failure occurs.

[0103] In the embodiments described above, the history of evaluation values ​​and the history of the number of defects are stored, but it is also possible to update them each time and store the latest evaluation value and the latest number of defects. The processing using the learning model described in the embodiments may also be rule-based processing. Furthermore, the location of the failure is not limited to the location described in the embodiments, etc., but may be other locations such as photodiodes.

[0104] For example, the configuration, methods, processes, shapes, materials, and numerical values ​​of the embodiments described above can be combined or replaced with each other, as long as they do not deviate from the spirit of this technology. It is also possible to divide one thing into two or more parts, and to omit parts. Furthermore, the matters described in the embodiments and modifications can be combined with each other. In addition, each of the processes described above may be performed in a distributed manner by multiple information processing devices.

[0105] The effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0106] Furthermore, this technology can also be configured as follows: (1) An information processing device comprising a failure prediction unit that takes evaluation information regarding the normality of a sensor as input and outputs a prediction result regarding the failure of the sensor. (2) The information processing device according to (1), comprising a detection unit that generates evaluation image data including at least a defective portion caused by a faulty part of the sensor by applying a learning model to input image data. (3) The information processing device according to (2), wherein the detection unit generates the evaluation image data by calculating the difference between the input image data and the restored image data obtained by reconstructing the input image data by applying the learning model to the input image data. (4) The information processing device according to (2) or (3), wherein the detection unit acquires an evaluation value based on the evaluation image data, and the evaluation value is input to the failure prediction unit as evaluation information regarding the normality of the sensor. (5) The information processing device according to (4), wherein the failure prediction unit outputs the prediction result to a notification unit when the evaluation value exceeds a predetermined threshold. (6) The information processing device according to (5), wherein the notification unit notifies information based on the prediction result. (7) An information processing device according to (5) or (6), further comprising a defect location classification unit that generates a classification result including the number of defective locations of the sensor by applying a learning model to the evaluation image data and outputs the generated classification result to the notification unit. (8) An information processing device according to (7), wherein the notification unit notifies information based on the classification result. (9) An information processing device according to any one of (5) to (8), further comprising the notification unit. (10) An information processing device according to (2), further comprising a defect location classification unit that generates a classification result including the number of defective locations of the sensor by applying a learning model to the evaluation image data supplied from the detection unit. (11) An information processing device according to (10), wherein the classification result is input to the failure prediction unit as evaluation information regarding normality. (12) An information processing device according to (11), wherein the classification result is the temporal change in the number of defective locations.(13) The information processing device according to (11) or (12), wherein the fault prediction unit outputs the prediction result when the number of predetermined defective locations exceeds a predetermined threshold. (14) The information processing device according to (13), further comprising a notification unit for notifying information based on the prediction result. (15) The information processing device according to (14), wherein the information based on the prediction result includes prediction results regarding defective locations where the number of defective locations exceeds the predetermined threshold. (16) The information processing device according to any one of (2) to (15), wherein the sensor is an imaging sensor, and the input image data is RAW data obtained by the imaging sensor, or image data obtained by the imaging sensor from which predetermined image signal processing has been performed within the imaging sensor. (17) The information processing device according to any one of (2) to (15), wherein the detection unit intermittently generates the evaluation image data. (18) The information processing device according to any one of (7) to (9), wherein the learning model is a model learned according to the sensor type of the sensor. (19) An information processing method wherein a failure prediction unit takes evaluation information regarding the normality of a sensor as input and outputs a prediction result regarding the failure of the sensor. (20) An information processing system comprising a sensor and an application processor connected to the sensor, wherein the application processor includes a failure prediction unit that takes evaluation information regarding the normality of the sensor as input and outputs a prediction result regarding the failure of the sensor.

[0107] <Application Examples> Next, we will explain some application examples of this technology. Note that the content of this technology is not limited to the application examples described below.

[0108] [Application Example 1] This application example demonstrates the application of this technology to mobile devices. Specifically, this application example demonstrates the application of this technology to authenticity detection of images captured by mobile devices. Authenticity detection means detecting whether an image that has undergone image signal processing on a mobile device, etc. (hereinafter referred to as the processed image as appropriate) deviates from the image before image signal processing (hereinafter referred to as the original image as appropriate). For example, if the original image deviates to a certain extent from the processed image, it is determined to be not authentic.

[0109] Figure 14 shows an example configuration of an information processing system (hereinafter referred to as information processing system 1C) according to this application example. Similar to the embodiment, information processing system 1C has a CIS 10 and an application processor 20. The application processor 20 according to this application example has an authenticity detection unit 40 instead of a fault prediction unit 24. The authenticity detection unit 40 is connected downstream of the memory 23. An output unit 25 is connected downstream of the authenticity detection unit 40.

[0110] As shown in Figure 15, the failure prediction mode is executed for a predetermined number of frames (for example, the Nth frame) after the camera function of the mobile device is activated. The period from the activation of the camera function to the predetermined number of frames is also called the preview mode. In this application example, the failure prediction mode is a mode that performs image authenticity detection by applying the processing described in the embodiment. After the failure prediction mode, that is, from frame (N+1), the mode transitions to the normal drive mode, in other words, the mode in which the camera function of the mobile device is executed.

[0111] In preview mode, correction processing is performed on the original image to adjust brightness, color tone, etc. For example, as shown in Figure 16, RAW image data is obtained first. This image data (an example of the original image) is then subjected to a process that gradually increases the strength of the correction. For example, in the Nth frame image shown in Figure 16, the correction processing has been applied quite strongly, and the discrepancy with the image data of the first frame is large.

[0112] The anomaly detection unit 22 in this application example detects anomalies using an image restoration model. The image restoration model in this example is a model that restores the input image to an image as if it had been processed with default parameters. For a RAW image, the image processed with default parameters is the first frame image (= normal image), and the image processed with excessive correction parameters is the Nth frame image (= abnormal image). The evaluation value generation unit 22A generates evaluation values. For example, if the image is normal, the difference before and after the restoration process (an example of an evaluation value) will be small, and if the image is abnormal, the difference will be large.

[0113] For example, since the image data for the first frame is generated by processing with default parameters, the evaluation value will be small (0 or nearly 0), as shown at point P1 in Figure 17.

[0114] For the second, third, and so on, processed image data with progressively stronger correction processing is input. As the correction processing becomes stronger, the difference from the image data of the first frame increases, so the evaluation value generated by the evaluation value generation unit 22A increases as the number of frames increases, as shown in Figure 17.

[0115] A genuineness detection threshold is set for changes in the evaluation value shown in Figure 17. When the evaluation value exceeds the genuineness detection threshold (point P2 in Figure 17), it is determined that the processed image data lacks genuineness. The genuineness detection unit 40 outputs the detection result. The detection result is reported as appropriate by the notification unit 30.

[0116] As explained above, this technology can also be applied to authenticity detection of images and other data. Furthermore, the processing in this application example may be performed during periods other than preview mode, and can be performed on electronic devices other than mobile devices. Additionally, the detection result may output a degree of authenticity rather than simply whether or not authenticity was detected (a binary result).

[0117] [Application Example 2] The prediction results and classification results described in the embodiment may be sent to the maintenance company's server. The maintenance company can use the prediction results and classification results to optimize the timing of maintenance notifications, publicize recalls, improve products, etc.

[0118] 1A, 1B, 1C... Information processing system 10... CIS 11... Pixel unit 13... ISP 20... Application processor 22... Anomaly detection unit 24... Failure prediction unit 26... Defect location classification unit 30... Notification unit IMA... Input image data IMB... Restored image data IMC... Evaluation image data

Claims

1. An information processing device comprising a failure prediction unit that takes evaluation information regarding the normality of a sensor as input and outputs a prediction result regarding the failure of the sensor.

2. The information processing apparatus according to claim 1, further comprising a detection unit that generates evaluation image data including at least defective portions caused by the faulty parts of the sensor by applying a learning model to input image data.

3. The information processing apparatus according to claim 2, wherein the detection unit generates the evaluation image data by determining the difference between the input image data and the restored image data obtained by applying the learning model to the input image data.

4. The information processing apparatus according to claim 2, wherein the detection unit acquires an evaluation value based on the evaluation image data, and the evaluation value is input to the failure prediction unit as evaluation information regarding the normality of the sensor.

5. The information processing apparatus according to claim 4, wherein the fault prediction unit outputs the prediction result to the notification unit when the evaluation value exceeds a predetermined threshold.

6. The information processing apparatus according to claim 5, wherein the notification unit notifies information based on the prediction result.

7. The information processing apparatus according to claim 5, further comprising a defect location classification unit that generates a classification result including the defective location of the sensor by applying a learning model to the evaluation image data, and outputs the generated classification result to the notification unit.

8. The information processing apparatus according to claim 7, wherein the notification unit notifies information based on the classification result.

9. The information processing apparatus according to claim 5, comprising the notification unit.

10. The information processing apparatus according to claim 2, further comprising a defect location classification unit that generates a classification result including the number of defective locations of the sensor by applying a learning model to the evaluation image data supplied from the detection unit.

11. The information processing apparatus according to claim 10, wherein the classification result is input to the failure prediction unit as evaluation information regarding normality.

12. The information processing apparatus according to claim 11, wherein the classification result is the change in the number of defective locations over time.

13. The information processing apparatus according to claim 11, wherein the failure prediction unit outputs the prediction result when the number of predetermined defective locations exceeds a predetermined threshold.

14. The information processing apparatus according to claim 13, further comprising a notification unit for notifying information based on the prediction results.

15. The information processing apparatus according to claim 14, wherein the information based on the prediction results includes prediction results regarding defective locations where the number of defective locations exceeds the predetermined threshold.

16. The information processing apparatus according to claim 2, wherein the sensor is an imaging sensor, and the input image data is RAW data obtained by the imaging sensor, or image data obtained by performing predetermined image signal processing on the RAW data obtained by the imaging sensor within the imaging sensor.

17. The information processing apparatus according to claim 2, wherein the detection unit intermittently generates the evaluation image data.

18. The information processing apparatus according to claim 7, wherein the learning model is a model learned according to the sensor type of the sensor.

19. An information processing method in which a failure prediction unit takes evaluation information regarding the normality of a sensor as input and outputs a prediction result regarding the failure of the sensor.

20. An information processing system comprising a sensor and an application processor connected to the sensor, wherein the application processor includes a failure prediction unit that takes evaluation information regarding the normality of the sensor as input and outputs a prediction result regarding the failure of the sensor.

Images