Camera architectures for safety related applications and methods related thereto

Abstract

Method and systems for checking the integrity of a camera system are provided. In preferred embodiments, the method comprises providing reference data with a known integrity value to a camera sensor interface. Providing the known integrity value to a comparator. Passing the reference data through the camera sensor interface and through an image pipeline. Generating a second integrity value for the reference data after the image pipeline. Providing the second integrity value to the comparator and comparing the known integrity value to the second integrity value.

Description

PATENT CAMERA ARCHITECTURES FOR SAFETY RELATED APPLICATIONS AND METHODS RELATED THERETOFIELD OF THE INVENTION

[0001] The present invention pertains to the field of camera architectures for safety applications and methods related thereto. In particular, die present invention pertains to the field of camera archi tectures for use in aircrafts.BACKGROUND

[0002] Applicant lias produced cameras for the aerospace industry for over twenty-five years. The cameras have evolved in design and application over this time to adapt to new technology and markets. The present application is related to architectures for use with the next generation of cameras.

[0003] Applicant and other competitors in die aerospace camera market currently offer cameras intended for "entertainment only" applications. This has been the state of the industry for many years. These cameras provide video to various aircraft computers or displays but are not certified to high Design Assurance Levels due to the high cost and workload needed to certify the entire camera system. Additionally, camera manufacturers have been reluctant to utilize state of the art technology in safety related aerospace applications due to minimal service history of new technology. Accordingly, cameras are not used as sensors for safety related applications in aircraft because the system would have to meet higher Design Assurance Level ("‘DAL”) compliance.

[0004] How to guarantee the operation of the cameras to die high reliability levels required by high DAL has not been solved in the past. The general approach was to keep the cameras at a low DAL and only use the cameras in applications that do not require a higher DAI... However, die autonomous air market is looking for higher DAL cameras for use as a sensor for navigating, landing and general orientation. This invention solves the problem of allowing a path to higher DAL certification.SUMMARYPATENT

[0005] The present invention provides methods and systems for checking the integrity of a camera system. In preferred embodiments of the method, the method comprises the following steps, providing reference data with a known integrity value, for example a checksum, from non-volatile memory to a camera sensor interface. Sending the known integrity value to a comparator like a checksum comparator for example. Passing the reference data through the camera sensor interface and through an image pipeline. Generating a second integrity value, such as a second checksum, for the reference data after the image pipeline. Sending the second integrity value to the comparator and comparing the known integrity value to the second integrity value to determine the health of the camera system.

[0006] In preferred embodiments, the known integrity value and the second integrity value are both checksums. However, any type of data check may be used for the integrity values such as a hash or other data check. When checksums are used, then the comparator is a checksum comparator.

[0007] In some embodiments, more than one data check may be performed at different locations along the camera / image system. In some embodiments, a third integrity value i s generated for the reference data after image post processing. The third integrity value is also sent to die comparator and compared io the known integrity value to determine the health of the camera system.

[0008] In preferred embodiments, the camera sensor interface, image pipeline and image post processing all reside on an FPGA. However, in other embodiments a dedicated processor or ASIC may be used.

[0009] In another aspect of the invention, a camera system architecture with built in safety checks is provided. The camera system comprises an image sensor and a processing unit comprising a sensor interface, image pipeline and image post processing module. The camera system also comprises non-volatile memory including reference data with a corresponding known integrity value. The cameras system also comprises a comparator.

[0010] hi such systems, the non-volatile memory is designed to pass the reference data to the image sensor interface and the known integrity value to the comparator. The processing unit is configured to generate a second integrity value from the reference data after the reference data has been processed by the image pipeline. The processing unit is further configured to provide the second integrity value to the comparator forPATENT comparison with the known integrity value to determine the integrity of the camera system.

[0011] In preferred embodiments of the system, the processing unit is a FPGA. Also in preferred embodiments, the comparator is a checksum comparator.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Fig. 1 illustrates a schematic view of a new camera architecture for achieving higher DAI, levels on aircraft.DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] The present invention relates to methods and systems for using cameras on aircraft as sensors while being able to maintain high levels of compliance, particularly with DAI. standards.

[0014] Fig. 1 illustrates a schematic view of a new camera architecture 20 for achieving higher DAI., levels on aircraft. As may be seen in Fig. 1, fhe architecture comprises an image sensor 2 and a processing unit 3. The processing unit 3 contains a sensor interface 4, image pipeline 5 and post image processing 6 implemented in firmware.

[0015] As one skilled in the art will appreciate, processing unit 3 may be any type of processing unit including a Field Programmable Gate Array (FPGA). Other processing type chips may be used including but limited to a dedicated central processing unit (" CPU") or Application Specific Integrated Circuit (“ASIC”), just to name a few.

[0016] In operation, like typical cameras systems, this system of hardware and firmware gathers images from the sensor, adjusts fhe raw image, gathers any needed image data and outputs them in a specific format out to the aircraft.

[0017] However, the systems 20 taught herein have additional components and take additional steps. In addition to these basic operating steps, the system 20 includes a number of steps to insure the health of the camera system and integrity of the data.

[0018] In preferred embodiments, the system includes non-volatile memory 7. Types of non-volatile memory may include hut are not limited to Read-Only Memory (ROM), Flash Memory, Solid-State Drives (SSD). Magnetic Storage, Optical Discs, Ferroelectric RAM (FeRAM), Phase Change Memory (PCM), Resistive RAM (ReRAM), Magnetoresistive RAM (MRAM), Non-Volatile RAM (NVRAM).PATENT

[0019] In order to check the integrity of the various components in the system, the architectures taught herein use a reference image with a known data integrity value, for example a known hash or known checksum or other known data integrity check.

[0020] The known data integrity result is then compared to data integrity check results at various locations in the system by a comparator 8 a.k.a data integrity check 8. If all die results of each checksum / hash are consistent, then the system is functioning properly. If there are any discrepancies during the data integrity check, a fault signal may be sent out. The various components and steps of the system and architecture will be described in more detail below,

[0021] In preferred embodiments, reference data is stored in non-volatile memory 7. The reference data may be comprised of test pattern images, mock sensor health data, or other data that can be used to check the health of the system. The reference data may be any type of data as long as i creates a predictable output for the data integrity check 8.

[0022] In operation, the reference data is loaded from the non-volatile memory 7 into the sensor interface on the FPGA 4 in the same manner as data would be loaded from the image sensor 2, As the reference data is processed through the image pipeline 5 a checksum is calculated for the reference data. Tills checksum is then sent out of the FPGA 3 to an integrity check module 8.

[0023] The data integrity check module 8, which may be referred to as the Checksum Comparison Module 8 if checksums are used, compares the calculated checksum value 9 with the stored checksum value for that specific set of reference data 10. If the two checksums do not match, this indicates the image pipeline is not functioning correctly and a fault is indicated to the aircraft 11.

[0024] In preferred embodiments, the data integrity check module 8 may be a flight heritage integrated circuit, CPU, FPGA, ASIC or other programable chip.

[0025] An important aspect of the invention is that the camera system 15 is separated and apart, from the additional components required for monitoring the integrity of the camera system. As may be appreciated, the cameras system 15 includes the image sensor 2, sensor interlace 4, image pipeline 5 and image post processing 6. Separate and apart from the camera system 15 are the elements required for monitoring the integrity of the camera system, namely, the non-volatile memory 7 including the reference images and die integrity check module 8.PATENT

[0026] Having the components required for monitoring the health and integrity of the camera system separate and apart from the actual camera system allows for the camera system to be developed independently from die monitoring system. This reduces costs and allows for more modularity.

[0027] Typically, checksums or other data integrity checks are always done as part of the system itself. However, Applicant believes removing these types of tasks allows for the advantages discussed herein. The only modification of the camera system to interface with the monitoring system is to modify the firmware to allow receiving the reference images and exporting the integrity data to the integrity check module 8 for comparison.

[0028] In operation, a reference data set is injected into the camera system. The camera system processes the reference data to generate and export image data. The comparator 8 evaluates the reference data set versus the processed data to produce a checksum value, which is an external process from the camera system.

[0029] Tills system architecture provides a simple way of determining if the camera is functioning properly, enabling the system to report faults with a high reliability. This architecture can be easily expanded for verifying functionality at ah stages in a camera system. For example, as may be seen in Fig. 1, another checksum comparison 12 is added at the output of the image post processing block 6 to verify if the postprocessing is producing expected results. To this end, not only can the system detect a fault, but the sy stem can pinpoint exactly where in the architecture the fault is occurring.

[0030] As may be appreciated, although Fig. 1 only shows two checksums being generated by the system, the first after the image pipeline and the second after post processing, in more complex systems, additional checksums may be used.

[0031] Accordingly, the architecture described herein, allows for the state-of-the-art technology to be used in the camera pipeline even in safety related applications because the high reliability and relevant service history is built into the monitoring circuitry rather than the camera pipeline..

Claims

PATENT CLAIMSWhat is claimed is:

1. A method of checking the integrity of a camera system comprising:providing reference data with a known integrity value to a camera sensor interface;providing the known integrity value to a comparator;passing the reference data through the camera sensor interface and through an image piperine:generating a second integrity value for the reference data after the image pipeline;providing die second integrity value to the comparator; and comparing the known integrity value to the second integrity value.

2. The method of claim 1, wherein the known integrity value and the second integrity value are both checksums.

3. The method of claim 2, wherein the comparator is a checksum comparator.

4. The method of claim 1 further comprising:receiving the reference data at an image post processor; generating a third integrity value for the reference data after image post processor;providing the third integrity value to the comparator; and comparing the third integrity value to the known integrity value.

5. The method of claim 4, wherein the camera sensor interface, image pipeline and image post processor all reside on an FPGA.

6. The method of claim 4, wherein the camera sensor interface, the image pipeline and the image post processor are physically separate from the comparator and only in data communication.

7. The method of claim 5, wherein the comparator is not on the FPGA with any of the camera sensor interface, image pipehne and image post processor.

8. The method of claim 1, feather comprising the step of transmitting a camera health signal from the comparator based on a result of the comparison of the second integrity value and the known integrity value.

9. A camera system architecture with built in safety checks comprising:an image sensor;PATENT a processing unit comprising a sensor interface, image pipeline and image post processor;non-volatile memory including reference data with a corresponding known integrity value;a comparator;wherein the non-volatile memory is configured to pass the reference data to the image sensor interface and the known integrity value to the comparator; andwherein the processing unit is configured to generate a second integrity value from the reference data after the reference data has been processed by the image pipeline; andwherein the processing unit is configured to provide the second integrity value to the comparator for comparison with the known integrity value.

10. The system of claim 9, wherein the processing unit is a FPGA.

11. The system of claim 9, wherein the comparator is a checksum comparator.

12. The system of claim 9, further comprising:an image post processor configured to receive the reference data from the image pipeline and generate a third integrity value for the reference data after image post processing and send the third integrity value to the comparator; andwherein the comparator is configured to compare the third integrity value to the known integrity value.

13. The method of claim 12, wherein the camera sensor interface, image pipeline and image post processor all reside on an FPGA.

14. The method of claim 12, wherein the camera sensor interface, the image pipeline and the image post processor are physically separate from the comparator and only in data communication.

15. The method of claim 14, wherein the comparator is not on the FPGA with any of the camera sensor interface, image pipeline and image post processor, 16. A method of checking the integrity of a camera system comprising:receiving reference data with a known integrity value at a camera sensor interface;PATENT receiving the known integrity value at a comparator:passing the reference data through the camera sensor interface and through an image pipeline;generating a second integrity value for the reference data after the image pipeline;receiving the second integrity value at the comparator; comparing the known integrity value to the second integrity value; receiving the reference data at an image post processor: generating a third integrity value for the reference data after image post processor;receiving the third integrity value at the comparator;comparing the third integrity value to the known integrity value; and transmitting a camera health signal from the comparator based on a result of the comparison of the second integrity value and the known integrity value and the third integrity value and the known integrity value.

17. The method of claim 16, wherein the known integrity value, the second integrity value and the third integrity value are all checksums.18, The method of claim 17, wherein the comparator is a checksum comparator, 19, The method of claim 16, wherein the camera sensor interface, image pipeline and image post processor all reside on a single FPGA.

20. The method of claim 16, wherein die camera sensor interface, the image pipeline and the image post processor are physically separate from the comparator and only in data communication.

Images