Method, device, and program for integrally processing exposure adjustment for multichannel image sensors

Abstract

Disclosed, according to various embodiments of the present invention, is a method for integrally processing exposure adjustment for multichannel image sensors. The method comprises the steps of: acquiring an image signal processor (ISP) interrupt signal from any one ISP among ISPs corresponding to respective multichannel image sensors; upon receiving the ISP interrupt signal, acquiring the current luminance value calculated by the any one ISP; calculating an auto exposure (AE) value corresponding to the current luminance value on the basis of a single micro controller unit (MCU); and applying the AE value to the image sensor that is connected to the any one ISP. The processes of acquiring the current luminance value and calculating the AE value, and applying the AE value to the image sensor are repeatedly performed for each of the multichannel image sensors, and thus the exposure of each of the multichannel image sensors may be integrally adjusted.

Description

Method, device, and program for integrated processing of exposure adjustment of a multi-channel image sensor

[0001] The present invention relates to a method, apparatus, and program for integrated processing of exposure adjustment of a multi-channel image sensor. Specifically, it relates to a method, apparatus, and program for integrated processing of exposure adjustment of a multi-channel image sensor by configuring the MCU of an Image Signal Processor (ISP) connected to each of the multi-channel image sensors into a single unit.

[0002] Image sensors are a core technology utilized in various applications by converting optical signals into digital data. This technology is essential in a wide range of industries, including digital cameras, surveillance systems, medical imaging systems, autonomous vehicles, and factory automation; its importance is particularly highlighted in environments requiring high-resolution images and real-time processing. Multi-channel image sensor systems, which combine multiple such sensors, have evolved to capture complex environments or provide high-precision data. Multi-channel image sensor systems offer the advantage of overcoming the limitations of single sensors and providing richer data, three-dimensional information, and high-resolution images.

[0003] Multichannel image sensors are designed so that each channel independently captures and processes images. For example, stereo camera systems use two image sensors to extract depth information or reconstruct stereoscopic scenes, while multispectral image sensors utilize multiple sensors to analyze light of various wavelengths, making them applicable in agriculture, medicine, and remote sensing. Furthermore, in manufacturing processes, multichannel image sensors contribute to detecting product defects or improving the precision of assembly processes. As these technologies are essential for ensuring high reliability and efficiency in modern industries, the importance of multichannel image sensor systems is steadily increasing.

[0004] However, multi-channel image sensor systems present several significant challenges during the design and operation process. For instance, because each channel operates independently, it is often difficult to maintain the timing and synchronization of data generated by each image sensor. In particular, in applications requiring high-speed image processing or real-time video analysis, these synchronization issues can lead to system performance degradation. Furthermore, since multi-channel image sensor systems often deploy separate MCUs and ISPs for each channel, this redundant hardware configuration increases system design complexity and raises manufacturing costs. For instance, the presence of multiple MCUs performing the same function unnecessarily increases memory usage and power consumption, which can act as a particularly critical constraint in battery-powered mobile or embedded systems.

[0005] Therefore, a technical approach is required to improve the synchronization and data processing efficiency of multi-channel image sensors, minimize redundant hardware configurations, and reduce the complexity of system design and operation. In this regard, Korean Published Patent No. 10-2011-0048922 discloses an integrated noise modeling method for an image sensor and a noise reduction method using the same.

[0006] The present invention, devised in response to the aforementioned background technology, aims to provide a method, apparatus, and program for integrated processing of exposure adjustment of a multi-channel image sensor.

[0007] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below.

[0008] According to an embodiment of the present invention for solving the problem described above, an integrated processing method for exposure adjustment of a multi-channel image sensor is disclosed. The method comprises: a step of obtaining an ISP interrupt signal from one of the Image Signal Processors (ISPs) corresponding to each of the multi-channel image sensors; a step of obtaining a current luminance value calculated by the one of the ISPs upon receiving the ISP interrupt signal; a step of calculating an Auto Exposure (AE) value corresponding to the current luminance value based on a single Micro Controller Unit (MCU); and a step of applying the AE value to an image sensor connected to the one of the ISPs; and by repeatedly performing the process of obtaining the current luminance value, calculating the AE value, and applying the AE value to the image sensor for each of the multi-channel image sensors, the exposure of each of the multi-channel image sensors can be integratedly adjusted.

[0009] In an alternative embodiment, the step of obtaining an ISP interrupt signal from any one of the Image Signal Processors (ISPs) corresponding to each of the multi-channel image sensors may include: a step of recognizing an identifier of the ISP based on a VSYNC interrupt signal; and a step of recognizing a specific ISP that provided the ISP interrupt signal based on the identifier.

[0010] In an alternative embodiment, the current luminance value calculated by any one of the ISPs includes the current luminance value per block calculated by any one of the ISPs for each of the multiple blocks after the ISPs divide the Bayer image acquired from the image sensor into multiple blocks, and the ISPs generate an ISP interrupt signal when the calculation for the current luminance value per block is completed, and the single MCU can acquire the block-specific luminance value in conjunction with the generation of the interrupt signal.

[0011] In an alternative embodiment, the step of calculating an Auto Exposure (AE) value corresponding to the luminance value based on the single Micro Controller Unit (MCU) may include: calculating the difference between the target luminance value for each image sensor and the current luminance value; determining whether to change the exposure parameter based on the difference; and, if it is determined to change the exposure parameter, calculating a new exposure parameter and updating the AE value.

[0012] In an alternative embodiment, the step of applying the AE value to an image sensor connected to any one of the ISPs may include the step of updating the exposure setting register of the image sensor according to the AE value.

[0013] In an alternative embodiment, the single MCU determines the ISP to receive data among the multi-channel image sensors whenever a new frame starts through a VSYNC interrupt service routine (VSYNC ISR), receives the block-by-block luminance value calculated by the ISP through an ISP interrupt service routine (ISP ISR), and calculates an AE value based on the block-by-block luminance value through a main loop and applies it to the image sensor register.

[0014] In an alternative embodiment, the single MCU monitors the VSYNC interrupt signal and the ISP interrupt signal provided by a plurality of ISPs, respectively, and processes the ISP interrupt signal based on the ISP identifier recognized through the VSYNC interrupt signal to perform inter-frame synchronization of the multi-channel image sensor.

[0015] In an alternative embodiment, the single MCU sequentially references ISP-specific luminance data, uses a common buffer that temporarily stores block-specific luminance values ​​calculated per ISP, and can sequentially process the update of exposure setting registers for each multi-channel image sensor.

[0016] According to one embodiment of the present invention for solving the above-described problem, an apparatus is disclosed. The apparatus comprises: a memory for storing one or more instructions; and a processor for executing the one or more instructions stored in the memory, and the processor can perform the above-described methods by executing the one or more instructions.

[0017] According to one embodiment of the present invention for solving the above-described problem, a computer program stored on a recording medium readable by a computer is disclosed, which is combined with a computer, which is hardware, to perform the above-described methods.

[0018] Other specific details of the present invention are included in the detailed description and drawings.

[0019] The present invention can reduce system complexity and cost, and improve power efficiency and synchronization performance by integrally coordinating the exposure of each multi-channel image sensor based on a single MCU. In addition, the single MCU structure allows for a reduction in the size of the ISP chip and lowers manufacturing costs.

[0020] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0021] FIG. 1 is a drawing illustrating a system according to one embodiment of the present invention.

[0022] FIG. 2 is a hardware configuration diagram of a computing device according to one embodiment of the present invention.

[0023] Figure 3 illustrates an example of a system configuration diagram and a chip configuration diagram related to a conventional multi-channel image sensor.

[0024] FIGS. 4 to 6 are drawings for explaining an integrated processing method for exposure adjustment of a multi-channel image sensor according to an embodiment of the present invention.

[0025] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to facilitate an understanding of the invention. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0026] As used herein, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computing device and the computing device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0027] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0028] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this specification and claims should generally be interpreted to mean “one or more.”

[0029] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be interpreted as moving out of the scope of the invention.

[0030] The description of the presented embodiments is provided to enable those skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present invention. Thus, the present invention is not limited to the embodiments presented herein. The present invention should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0031] In this specification, the term "computer" refers to any type of hardware device comprising at least one processor, and may be understood to include software configurations operating on said hardware device according to the embodiments. For example, the term "computer" may be understood to include smartphones, tablet PCs, desktops, laptops, and user clients and applications running on each of these devices, but is not limited thereto.

[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0033] Each step described in this specification is described as being performed by a computer, but the subject of each step is not limited thereto, and depending on the embodiment, at least some of each step may be performed on different devices.

[0034]

[0035] FIG. 1 is a drawing illustrating a system according to one embodiment of the present invention.

[0036] Referring to FIG. 1, a system according to one embodiment of the present invention may include a computing device (100), a user terminal (200), and an external server (300). The system illustrated in FIG. 1 is according to one embodiment, and its components are not limited to the embodiment illustrated in FIG. 1 and may be added, changed, or deleted as needed.

[0037] In the following description, for convenience of explanation, it is assumed that a single MCU is included in the computing device (100), but it may also be implemented in a physically separated form. When a single MCU is included in the computing device (100), the single MCU calculates an AE value based on a luminous value received from an ISP and applies it to each of the multi-channel image sensors. Meanwhile, when the single MCU acts as a separate control device, the single MCU can process various operations of the present invention through interaction (e.g., data bus, interrupt) with the computing device (100).

[0038] In one embodiment, the computing device (100) can perform integrated processing of exposure adjustment for a multi-channel image sensor. For example, the computing device (100) can adjust the exposure integrally by integrating the Micro Controller Units (MCUs) that adjust the exposure of each multi-channel image sensor into one.

[0039] Specifically, the computing device (100) can obtain an ISP interrupt signal from one of the Image Signal Processors (ISPs) corresponding to each of the multi-channel image sensors. Additionally, upon receiving the ISP interrupt signal, the computing device (100) can obtain a current luminance value calculated by one of the ISPs. Furthermore, the computing device (100) can calculate an Auto Exposure (AE) value corresponding to the current luminance value based on a single Micro Controller Unit (MCU). And, the computing device (100) can apply the AE value to the image sensor connected to one of the ISPs.

[0040] Meanwhile, the computing device (100) can obtain a current luminous value, calculate an AE value, and repeat the process of applying the AE value to the image sensor for each of the multi-channel image sensors, thereby enabling integrated adjustment of the exposure of each of the multi-channel image sensors.

[0041] Accordingly, the computing device (100) of the present invention can improve the efficiency and performance of a multi-channel image sensor system.

[0042] Hereinafter, an example of a method in which a computing device (100) performs integrated exposure adjustment processing of a multi-channel image sensor is described with reference to FIGS. 3 to 6.

[0043] In various embodiments, the computing device (100) may provide Web or Application-based services. However, it is not limited thereto.

[0044] The computing device (100) may include any type of computer system or computer device, such as, for example, a microprocessor, a mainframe computer, a digital processor, a portable device, and a device controller. However, it is not limited thereto.

[0045] Hereinafter, the hardware configuration of the computing device (100) will be described with reference to FIG. 2.

[0046] Meanwhile, the user terminal (200) may be connected to the computing device (100) via a network (400) and may be a user terminal related to the integrated processing method for exposure adjustment of a multi-channel image sensor performed by the computing device (100).

[0047] For example, the user terminal (200) may include a terminal used by a user who designs or develops a multi-channel image sensor. Additionally, the user terminal (200) may be used as a terminal for monitoring exposure adjustments or changing settings of the multi-channel image sensor, and such a terminal may be utilized by a developer, a researcher, or a user who performs tasks to optimize the performance of the multi-channel image sensor system.

[0048] Additionally, the user terminal (200) can support controlling the exposure adjustment process of the multi-channel image sensor, changing setting values, and checking the status of the system through a connection with the computing device (100).

[0049] Here, the user terminal (200) may include, for example, various types of computer devices. Specifically, for example, the user terminal (200) may refer to various terminal devices such as smartphones, tablet PCs, desktops, and laptops.

[0050] The user terminal (200) includes a display on at least a part of the terminal and may include an operating system for running applications or extension-based services provided by the computing device (100). For example, the user terminal (200) may be a smartphone, but is not limited thereto, and the user terminal (200) may include all kinds of handheld-based wireless communication devices such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartpad, tablet PC, etc., as wireless communication devices that ensure portability and mobility.

[0051] An external server (300) can be connected to a computing device (100) via a network (400), and can transmit and receive various information / data necessary for the computing device (100) to perform integrated exposure adjustment processing of a multi-channel image sensor, and can store and manage various information / data generated as the computing device (100) performs integrated exposure adjustment processing of a multi-channel image sensor.

[0052] For example, the external server (300) may be a database server that stores information used in the exposure adjustment integrated processing method of a multi-channel image sensor. As another example, the external server (300) may be a server that provides information used in the exposure adjustment integrated processing method of a multi-channel image sensor.

[0053] The network (400) may refer to a connection structure capable of exchanging information between each node, such as computing devices, multiple terminals, and servers. For example, the network (400) includes a Local Area Network (LAN), a Wide Area Network (WAN), the World Wide Web (WWW), a wired and wireless data network, a telephone network, a wired and wireless television network, etc.

[0054] Wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc.

[0055]

[0056] FIG. 2 is a hardware configuration diagram of a computing device according to one embodiment of the present invention.

[0057] Referring to FIG. 2, a computing device (100) according to one embodiment of the present invention may include one or more processors (110), a memory (120) for loading a computer program (151) executed by the processor (110), a bus (130), a communication interface (140), and a storage (150) for storing the computer program (151). Here, FIG. 2 illustrates only the components related to the embodiment of the present invention. Therefore, a person skilled in the art to which the present invention pertains will understand that other general-purpose components may be included in addition to the components illustrated in FIG. 2.

[0058] The processor (110) controls the overall operation of each component of the computing device (100). The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU) of the computing device. Alternatively, it may be configured to include any type of processor well known in the art of the present invention.

[0059] Additionally, the processor (110) can perform operations for at least one application or program for executing the method according to embodiments of the present invention, and the computing device (100) may have one or more processors.

[0060] In various embodiments, the processor (110) may further include RAM (Random Access Memory, not shown) and ROM (Read-Only Memory, not shown) for temporarily and / or permanently storing signals (or data) processed within the processor (110). Additionally, the processor (110) may be implemented in the form of a system-on-chip (SoC) comprising at least one of a graphics processing unit, RAM, and ROM.

[0061] Memory (120) stores various data, instructions and / or information. Memory (120) may load a computer program (151) from storage (150) to execute a method / operation according to various embodiments of the present invention. When the computer program (151) is loaded into memory (120), the processor (110) may perform the method / operation by executing one or more instructions constituting the computer program (151). Memory (120) may be implemented as volatile memory such as RAM, but the technical scope of the present invention is not limited thereto.

[0062] The bus (130) provides communication functions between components of the computing device (100). The bus (130) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0063] The communication interface (140) supports wired and wireless internet communication of the computing device (100). Additionally, the communication interface (140) may support various communication methods other than internet communication. To this end, the communication interface (140) may be configured to include a communication module well known in the art of the present invention. In some embodiments, the communication interface (140) may be omitted.

[0064] Storage (150) can store a computer program (151) non-temporarily. When performing a process according to an embodiment of the present invention through a computing device (100), storage (150) can store various information necessary to perform a method according to the disclosed embodiment or to provide a service.

[0065] The storage (150) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which the present invention belongs.

[0066] A computer program (151) may include one or more instructions that cause a processor (110) to perform a method / operation according to various embodiments of the present invention when loaded into memory (120). That is, the processor (110) may perform the method / operation according to various embodiments of the present invention by executing the one or more instructions.

[0067] In one embodiment, the computer program (151) may include one or more instructions to perform various methods related to various tasks related to learning a neural network model.

[0068] The steps of the method or algorithm described in connection with embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, a hard disk, a removable disk, a CD-ROM, or any form of computer-readable recording medium well known in the art to which the present invention belongs.

[0069] The components of the present invention may be implemented as a program (or application) and stored on a medium to be executed in combination with a computer, which is hardware. The components of the present invention may be implemented as software programming or software elements, and similarly, embodiments may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, etc., including various algorithms implemented as combinations of data structures, processes, routines, or other programming configurations. Functional aspects may be implemented as algorithms executed on one or more processors.

[0070]

[0071] Figure 3 illustrates an example of a system configuration diagram and a chip configuration diagram related to a conventional multi-channel image sensor.

[0072] Specifically, Figure 3 (A) shows an example of a conventional system configuration in which an Image Signal Processor (ISP) and a Micro Controller Unit (MCU) are independently assigned to each of the multi-channel image sensors, and Figure 3 (B) shows how these ISPs and MCUs are configured within a single chip.

[0073] First, referring to Fig. 3 (A), four image sensors, #0 through #3, each generate a Bayer pattern image, and an ISP connected 1:1 to each sensor receives and processes this image data based on VSYNC for every frame. At this time, each ISP is designed with identical functionality and size, and internally, it divides each frame into 16×16 blocks using a Hardware Statistic block and calculates the average luminance value for each block. Once this process is complete, an ISP Interrupt signal is generated and notified to the MCU running the Interrupt Service Routine (ISR). The block-specific luminance values ​​calculated by the ISP are subsequently used for software Auto Exposure (AE) operations.

[0074] As shown in (A) of Fig. 3, the MCUs connected to each ISP are also configured to have the same specifications and functions, so that they operate in the form of an ISR that processes the AE calculation process in the main loop and updates the calculated exposure value back into the image sensor register.

[0075] For example, when the operation of the last block of a frame is completed, the ISP transmits an interrupt signal to the MCU; upon receiving this, the MCU ISR internally calculates new exposure parameters or applies already calculated values ​​to the image sensor registers. However, in a structure where the number of ISPs and MCUs increases proportionally with the number of sensors, an unnecessarily large number of MCUs are placed within the chip, inevitably leading to increased memory usage and a larger chip size. In particular, in embedded environments or mobile devices where low-power design is critical, such additional MCUs can cause issues with power consumption and heat generation.

[0076] Referring to Figure 3 (B), the analog logic area and the digital logic area are separated within the actual chip, with a data bus located between them. In the digital logic area, four MCUs, from MCU_0 to MCU_3, are placed along with an ISP block, each allocated 32KB of memory. In other words, the ISPs corresponding to sensors #0 through #3 collect and process image data, and the MCUs connected one-to-one to each ISP perform necessary control functions, such as AE operations and sensor register updates. If the MCUs corresponding to each ISP are configured to have the same performance, stable operation can be maintained even when processing multiple frames simultaneously in a multi-channel image sensor; however, this increases the area of ​​the digital logic region within the entire chip. Consequently, design complexity increases, there is a high possibility of increased costs in the semiconductor manufacturing process, and memory may be used unnecessarily excessively.

[0077] In other words, as explained with reference to Fig. 3, the conventional method for processing multi-channel image sensors was to deploy an equal number of ISPs and MCUs corresponding to the number of sensors. That is, each ISP calculates a luminance value through a hardware statistical block every frame, and when an MCU interrupt occurs based on this result, the MCU connected to the ISP operates by calculating a new exposure value in the main loop or updating registers via an ISR. While this distributed structure has the advantage of increasing independence per channel, the existence of multiple MCUs with the same specifications leads to redundant memory consumption and a corresponding increase in chip size.

[0078] As an improvement measure for this, the present invention integrates the processing of luminous information from multiple ISPs using a single MCU. Specifically, the present invention proposes a method that reduces the number of MCUs within the chip and simultaneously reduces memory usage and chip area. This will be explained in more detail with reference to FIGS. 4 to 6.

[0079]

[0080] FIGS. 4 to 6 are drawings for explaining an integrated processing method for exposure adjustment of a multi-channel image sensor according to an embodiment of the present invention.

[0081] Figure 4 illustrates an example of a system configuration diagram and a chip configuration diagram in which a single MCU is applied for multi-channel image sensor data integration processing of the present invention.

[0082] Referring to (A) of FIG. 4, it can be seen that the MCUs, which were previously assigned one per image sensor, are integrated into a single MCU. Additionally, (B) of FIG. 4 shows the internal configuration of the system chip integrated into a single MCU.

[0083] In the conventional method, Bayer-format images generated by each sensor were transmitted to their respective ISPs, and the MCU assigned to that ISP would receive the hardware statistical results, calculate the exposure value, and update the sensor registers. While this distributed structure is simple in terms of channel independence, it presented problems such as increased cost due to higher memory usage and expanded digital logic area within the ISP chip as the number of MCUs increased proportionally to the number of sensors. In particular, since multiple MCUs operated independently despite performing the same function, design complexity and power consumption were inevitably excessive.

[0084] Meanwhile, to solve the problems of the existing method, the present invention has a single MCU that integrally controls all ISPs.

[0085] In the present invention, ISP Index information (i.e., ISP identifier) ​​including a sensor device ID is maintained for each ISP Block, and the MCU can identify which sensor (or which ISP) is currently receiving image data by collecting the corresponding ISP Index through a VSYNC Interrupt Service Routine (VSYNC ISR). This is made possible by additionally monitoring the VSYNC Interrupt Flag, rather than merely referencing the ISP Interrupt Flag as in the conventional method to indicate that the hardware statistical operation has finished, so that the ISP Index can be received directly as an internal variable of the MCU during the VSYNC ISR stage.

[0086] When the MCU obtains an ISP Index, it can execute an Auto Exposure (AE) algorithm to update the newly calculated exposure value in the sensor register corresponding to the index. Here, the Auto Exposure (AE) algorithm is an algorithm that optimizes the brightness of an image by comparing the Luminance Value collected from the image sensor with a set Target Luminance and adjusting exposure parameters such as Exposure Time and Gain.

[0087] As a result, a single MCU can handle the real-time adjustment of exposure for all ISPs through VSYNC and ISP interrupt management and the execution of AE algorithms, thereby mitigating the issues of increased chip size and memory waste that occurred in the existing structure where a separate MCU was required for each sensor. This structure provides high efficiency in the design and operation of multi-channel image sensor systems, and by utilizing a single MCU to integrally control multiple ISPs, it is possible to simultaneously achieve system cost reduction, reduced power consumption, and optimization of hardware resources.

[0088] In addition, a single MCU design facilitates the improvement or update of AE algorithms, allowing for flexible response to future software changes.

[0089] That is, as explained with reference to FIG. 4, the present invention effectively unifies the MCU configuration, which becomes exponentially more complex as the number of sensors increases, and can perform integrated exposure control of multiple sensors by rapidly identifying the ISP Index through VSYNC ISR.

[0090]

[0091] FIG. 5 is a flowchart illustrating an integrated processing method for exposure adjustment of a multi-channel image sensor according to an embodiment of the present invention.

[0092] Referring to FIG. 5, the computing device (100) can obtain an ISP interrupt signal from one of the Image Signal Processors (ISPs) corresponding to each of the multi-channel image sensors (S110). Here, the ISP interrupt signal is a signal indicating that image data processing is completed at the ISP, and is generated at the time when hardware statistical operations (e.g., calculation of luminous values ​​per block) are finished. Additionally, the ISP interrupt signal can serve as a trigger to enable a single MCU to safely receive data from the ISP and execute an Auto Exposure (AE) algorithm.

[0093] Specifically, the computing device (100) can recognize the identifier of an ISP based on a VSYNC interrupt signal. And, the computing device (100) can recognize a specific ISP that provided the ISP interrupt signal based on the identifier. Here, the VSYNC interrupt signal is a signal for frame synchronization that indicates that new frame data acquisition has started at each image sensor. The VSYNC interrupt signal can be used to align the timing of frames being processed by each ISP in a multi-channel environment and to maintain a synchronization state.

[0094] More specifically, the computing device (100) records ISP Index information in an internal variable whenever a frame synchronization signal occurs through a VSYNC ISR (Interrupt Service Routine), and can accurately identify the ISP that calculates the luminous value of the current frame by referring to this information. Here, the VSYNC ISR is an interrupt processing routine that is executed when a frame synchronization signal (VSYNC Interrupt) occurs, and it plays the role of detecting which ISP the signal occurred from and preparing the necessary data so that the MCU can process it internally. That is, the VSYNC ISR can determine the ISP Index (identifier) ​​and store the value so that it can be used for the execution of the AE (Auto Exposure) algorithm in a subsequent stage.

[0095] For example, the computing device (100) manages unique identifiers for each ISP in a memory table and can quickly determine which image sensor the ISP that actually generated the interrupt is mapped to by querying the corresponding identifier in the VSYNC ISR. Through this, a single MCU can accurately recognize the ISP currently processing data even in a multi-channel environment and obtain a luminous value for performing the AE algorithm in a timely manner. In this way, the VSYNC ISR can support the MCU in efficiently managing the status and task order of each ISP without interference in the data flow between ISPs.

[0096] According to one embodiment, the computing device (100) can obtain a current luminous value calculated by one of the ISPs upon receiving an ISP interrupt signal (S120).

[0097] Here, the current luminance value calculated by one ISP may include the current luminance value per block calculated by one ISP dividing the Bayer image acquired from the image sensor into multiple blocks and calculating the value per block for each of the multiple blocks. These block-specific luminance values ​​are used as data to precisely analyze the brightness state of the entire image and can be used as input values ​​for the AE (Auto Exposure) algorithm.

[0098] In addition, any one ISP can generate an ISP interrupt signal when the calculation for the current luminance value for each block is completed. That is, any one ISP can divide the Bayer image acquired from the image sensor into multiple blocks, calculate the luminance value for each block, and then generate an ISP interrupt signal when the calculation for all blocks is completed. Here, the ISP interrupt signal acts as a trigger to notify a single MCU that the luminance value is ready and can define the starting point for the execution of the AE algorithm.

[0099] In addition, a hardware statistical block exists within the ISP to calculate these luminance values. Here, the hardware statistical block divides image data into blocks and calculates the average brightness value or the average value per RGB channel for each block. Furthermore, when the calculation for the last block is completed, the hardware statistical block sets an interrupt flag so that the MCU can detect the data readiness status. In other words, the hardware statistical block generates a signal that informs the MCU of the completion status of the block-by-block luminance calculation task.

[0100] In this case, a single MCU can obtain block-by-block luminance values ​​in conjunction with the occurrence of an ISP interrupt signal. Specifically, the single MCU refers to ISP Index information to read block-by-block luminance values ​​from the memory buffer of the corresponding ISP and uses them as input values ​​for the AE algorithm. By utilizing block-by-block data in this way, the AE algorithm can finely reflect changes in brightness or illuminance distribution in specific areas, enabling more precise exposure control.

[0101] In various embodiments, when a computing device (100) or a single MCU recognizes that a hardware statistical block has generated an interrupt, it can receive block-specific luminous data prepared by the ISP in batches and input it into a software AE algorithm.

[0102] For example, the computing device (100) can check a memory buffer that matches the ISP Index, and then load the luminous value stored in the buffer into an internal register or memory of the MCU to use for subsequent processing.

[0103] According to one embodiment, the computing device (100) can calculate an Auto Exposure (AE) value corresponding to the current luminous value based on a single Micro Controller Unit (MCU) (S130).

[0104] Specifically, the computing device (100) can calculate the difference between the target luminance value and the current luminance value for each image sensor. And, the computing device (100) can determine whether to change the exposure parameter based on the difference.

[0105] More specifically, the computing device (100) can determine whether it is necessary to adjust the exposure parameter by calculating the difference between the target luminance value per image sensor (e.g., user-set brightness level or a standard predefined by the system) and the current luminance value.

[0106] For example, the computing device (100) can calculate a Global Luminance Value for the entire frame by summing the average brightness of each block, or estimate the error from the Target Luminance by assigning weights to specific areas (e.g., the central area). In this process, assigning weights to the central area enables efficient exposure adjustment when the center of the image contains more important information (e.g., portraits, key areas of a manufacturing process). For example, by assigning high weights to blocks in the central area and low weights to surrounding areas, the target brightness can be achieved more precisely.

[0107] Additionally, if the computing device (100) determines that the error exceeds a preset threshold (e.g., ±10% brightness deviation), it decides to change exposure parameters such as exposure time or gain value. The threshold set here is a standard for controlling the sensitivity of the AE algorithm; if set to a value too small, frequent exposure adjustments may occur, causing the brightness of the image to become unstable, and if set to a value too large, the exposure adjustment response may become slow.

[0108] The computing device (100) can finely adjust the exposure time per frame according to the magnitude and direction of the error to gradually converge the brightness level to a target value, or rapidly adjust the exposure value to a large value when a large error is detected to achieve appropriate brightness. For example, the computing device (100) finely adjusts the exposure time to about 5ms when the error is within ±5%, but can significantly adjust the exposure time to about 50ms at once when the error is ±20% or more.

[0109] Meanwhile, if the computing device (100) decides to change the exposure parameter, it can calculate a new exposure parameter and update the AE value.

[0110] Specifically, the computing device (100) can calculate an AE value to be applied in a future frame based on the changed exposure time or gain value and store it in an AE variable inside the MCU.

[0111] For example, the computing device (100) can dynamically control the range of change of exposure parameters to prevent excessive changes in image brightness in response to an environment where the light intensity changes rapidly (e.g., moving from indoors to outdoors), or perform rapid exposure adjustment in steps when rapid adaptation is required.

[0112] According to one embodiment, a computing device (100) can apply an AE value to an image sensor connected to one of the ISPs (S140). Here, the AE value is an exposure parameter calculated based on the difference between the target luminance and the current luminance, and is a value for controlling the amount of light collected by the image sensor. The AE value may mainly include parameters such as exposure time, gain, and, under certain conditions, aperture. Through this, the AE value is designed to optimize the brightness of the image and automatically adjust according to the lighting environment.

[0113] Specifically, the computing device (100) can update the exposure setting register of the image sensor according to the AE value. That is, the computing device (100) can make the previously calculated AE value apply to the image frame to be received thereafter by writing it to the exposure setting register of the image sensor.

[0114] More specifically, the computing device (100) can identify the image sensor to be updated by referring to the ISP Index, look up the control register address of the sensor, and then record the calculated new exposure parameters (e.g., exposure time, gain value, etc.). This process is performed independently for each sensor connected to the ISP, and allows each sensor to accurately reflect the corresponding AE value.

[0115] For example, the computing device (100) can call an AE update routine within the MCU ISR to access the register map of the image sensor and update values ​​such as exposure time or analog and digital gains so that the changed exposure conditions are applied from the next frame.

[0116] For example, if the image sensor receives a result indicating that the current luminous value is lower than the target brightness, the computing device (100) can update the settings to increase the exposure time by 10 ms and amplify the gain value by 1 dB. These changes are reflected immediately, allowing the image to be captured with the adjusted brightness starting from the next frame.

[0117] Accordingly, the computing device (100) can provide stable and optimized image quality even in environments where lighting changes rapidly by adjusting the exposure settings of the image sensor in real time based on the AE value. This enables precise exposure adjustment that reflects the characteristics of each sensor even in a multi-channel image sensor system, and can ensure consistency and quality of image data.

[0118] According to one embodiment of the present invention, a computing device (100) can obtain a current luminous value, calculate an AE value, and repeat the process of applying the AE value to an image sensor for each of the multi-channel image sensors, thereby enabling integrated adjustment of the exposure of each of the multi-channel image sensors.

[0119] Specifically, the computing device (100) can sequentially identify the ISP Index of each channel by utilizing the VSYNC ISR and ISP ISR in common regardless of the number of sensors, and process the luminous value calculated independently for each image sensor in a single MCU.

[0120] For example, the computing device (100) can centrally control the exposure conditions of all channels by updating the AE based on the luminous value of sensor #0 and then cyclically executing the same process on other sensors such as sensor #1, sensor #2, and sensor #3. This can significantly reduce memory usage and hardware resource consumption of the entire ISP chip, as it does not require placing a separate MCU corresponding to each sensor as in the existing structure.

[0121] Accordingly, the computing device (100) identifies the ISP Index by linking VSYNC and ISP interrupt signals to integrally control a multi-channel image sensor with a single MCU, and dynamically adjusts exposure parameters using luminous values ​​calculated for each block, thereby ensuring stable exposure control performance while minimizing memory usage and chip area as well as reducing system costs. This is particularly effective in cases where real-time image processing is required in various environments or in high-performance camera systems equipped with multiple sensors, and can be implemented in an advantageous manner even in low-power environments such as mobile devices or embedded systems.

[0122]

[0123] FIG. 6 is a diagram illustrating a single MCU-based multi-channel image sensor control process according to an embodiment of the present invention, visually showing how VSYNC interrupts and ISP interrupts are linked to adjust exposure.

[0124] In one embodiment, a single MCU according to the present invention can determine an ISP to receive data among multi-channel image sensors whenever a new frame starts through a VSYNC interrupt service routine (VSYNC ISR). Additionally, the single MCU can receive a block-by-block luminance value calculated by the ISP through an ISP interrupt service routine (ISP ISR), and calculate an AE value based on the block-by-block luminance value through a main loop and apply it to an image sensor register.

[0125] Specifically, as illustrated in (A) of FIG. 6, when the frame synchronization signal VSYNC occurs, the single MCU enters the VSYNC ISR to identify which ISP is processing the current frame data. During this process, if the VSYNC Interrupt Flag is set, the ISP Index (identifier) ​​is checked and recorded in an internal MCU variable. Additionally, when the VSYNC ISR ends, the single MCU clears the interrupt flag and operates in a structure that waits for the next interrupt.

[0126] For example, if a single MCU identifies in the VSYNC ISR that ISP #0 is receiving new image data in time with one frame cycle, it can then be prepared to receive the result of the luminous value calculation corresponding to ISP #0 in the next step.

[0127] Therefore, the VSYNC Interrupt ISR path (refer to the red arrow) in the flowchart shown in (B) of Fig. 6 is executed to set the 'Current ISP Index' inside the MCU, so that when an ISP Interrupt ISR occurs later, it becomes possible to accurately determine which ISP's luminous value needs to be updated.

[0128] In one embodiment, a single MCU according to the present invention can monitor VSYNC interrupt signals and ISP interrupt signals provided by a plurality of ISPs, respectively. Additionally, the single MCU can perform inter-frame synchronization of a multi-channel image sensor by processing the ISP interrupt signals separately based on the ISP identifier recognized through the VSYNC interrupt signal.

[0129] Specifically, when the ISP Index is determined in the VSYNC ISR, if the ISP Interrupt ISR path indicated by the green arrow in (B) of Fig. 6 is executed, the corresponding ISP Index is checked again, and then the block-by-block luminance value for which the hardware statistical calculation is completed can be received.

[0130] In addition, when the ISP Interrupt Flag is set, a single MCU can retrieve block-by-block average brightness information stored in the ISP-specific memory space and prepare it for use in the main loop.

[0131] For example, when ISP #0 generates an ISP Interrupt at the time when it has calculated all the luminous data divided into 16×16 blocks for one frame, a single MCU collects the block data based on the information “Current ISP Index = #0” and uses it for AE calculation.

[0132] Therefore, by separating and monitoring VSYNC and ISP interrupts occurring per frame in a multi-channel image sensor environment, the present invention enables a single MCU to accurately synchronize and process data flows coming in with independent timing for each sensor.

[0133] In one embodiment, a single MCU according to the present invention can sequentially reference luminous data per ISP. Additionally, the single MCU uses a common buffer that temporarily stores block-by-block luminous values ​​calculated per ISP, and can sequentially process the update of exposure setting registers for each multi-channel image sensor.

[0134] Specifically, in the main loop region illustrated in (B) of FIG. 6, after obtaining luminance data per ISP in the “Get Hardware Statistic data” step, if the “Check AE Enable” option is turned on, the AE algorithm can be performed to compare “Target & Current Y Avg.” Additionally, a single MCU can calculate a new exposure value based on the difference between the collected target brightness and the current luminance value, set it to “Update Flag = 1,” and when this flag is set, the corresponding value can be updated in the image sensor register in the ISP Interrupt ISR.

[0135] For example, if the average block brightness value of ISP #0 stored in the common buffer is significantly lower than the target brightness, the MCU can be instructed to increase the gain value or exposure time and then update the registers in the ISP Interrupt ISR.

[0136] Therefore, the present invention can simplify the overall ISP chip structure and reduce memory usage by efficiently managing ISP-specific data through a shared buffer and having a single MCU sequentially update the exposure parameters required for each channel.

[0137]

[0138] According to an additional embodiment of the present invention, a computing device (100) can perform an integrated AE algorithm that dynamically corrects exposure deviations between sensors by comparing luminous information extracted from each of the multi-channel image sensors.

[0139] Specifically, the computing device (100) can maintain uniform image quality across all channels by collectively collecting block-by-block luminance values ​​provided by multiple ISPs in a single MCU, modeling the average brightness per sensor, and reflecting this in overall exposure control.

[0140] More specifically, the computing device (100) can calculate the representative brightness for each sensor by mathematically integrating the block-by-block luminous average value calculated by each ISP or by calculating a weighted sum. Subsequently, the computing device (100) can compare and analyze the deviations between the representative brightnesses for each sensor (e.g., maximum and minimum brightness intervals, standard deviation, or difference between specific areas) and calculate a correction parameter by referring to the exposure data of other channels if a specific channel is excessively dark or bright.

[0141] For example, the computing device (100) may define a “brightness balance index between channels” when the brightness deviation between channels exceeds a certain threshold, and may dynamically adjust the exposure time or gain of the problematic channel when this index exceeds the threshold. This correction process may go beyond simple average comparison and additionally assign block-specific regional weights (e.g., important area weight, central area weight) to reflect local characteristics of ambient lighting or object reflectance for each channel.

[0142] Additionally, the computing device (100) has a “channel-to-channel correction routine” built into a single MCU, so that when it detects that the average brightness measured by sensor #0 is lower than other sensors (e.g., #1, #2, #3) by more than a certain range, it can automatically adjust the exposure amount by increasing the exposure time of the corresponding channel by x% or increasing the gain by y dB.

[0143] For example, the computing device (100) can remeasure and recalculate the luminous value for each frame while maintaining sensor-specific frame synchronization, and can track and reflect the error between each channel in real time based on the brightness balance between channels.

[0144] In this way, the computing device (100) can maximize image consistency between channels by individually correcting the exposure amount for each sensor in real time, even in environments where the illumination level changes rapidly or in situations where only some sensors have different lighting conditions.

[0145] For example, when sensors capturing the side and top of a product, respectively, on a process automation line have different lighting conditions, the multi-channel correction algorithm of the present invention can reduce the brightness gap between sensors and minimize the quality variation of the resulting image. In particular, in systems where multiple image sensors must be operated simultaneously, such as multi-view cameras, stereo vision, and multispectral analysis, minimizing the exposure variation between sensors can improve the accuracy of 3D reconstruction or spectral data analysis.

[0146] Accordingly, the computing device (100) of the present invention, unlike the conventional single-channel-centered AE approach, can enable high-quality and high-precision image processing by comparing and analyzing brightness data produced by multiple sensors and calculating dynamic correction parameters that reflect the characteristics of each sensor. This can also perform the function of automatically offsetting problems such as individual sensor degradation or offset drift that may occur as the system operation time increases, thereby ensuring excellent flexibility and stability across the entire multi-channel imaging field.

[0147]

[0148] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive.

Claims

1. A method performed by a computing device comprising at least one processor, A step of obtaining an ISP interrupt signal from any one of the Image Signal Processors (ISPs) corresponding to each of the multi-channel image sensors; A step of obtaining a current luminous value calculated by any one of the ISPs upon receiving the above ISP interrupt signal; A step of calculating an Auto Exposure (AE) value corresponding to the current luminous value based on a single Micro Controller Unit (MCU); and A step of applying the above AE value to an image sensor connected to any one of the above ISPs; Includes, By repeatedly performing the process of acquiring the current luminance value, calculating the AE value, and applying the AE value to the image sensor for each of the multi-channel image sensors, the exposure of each of the multi-channel image sensors is integratedly adjusted. Integrated processing method for exposure adjustment of a multi-channel image sensor.

2. In Paragraph 1, The step of obtaining an ISP interrupt signal from any one of the Image Signal Processors (ISPs) corresponding to each of the multi-channel image sensors is: A step of recognizing an ISP identifier based on a VSYNC interrupt signal; and A step of recognizing a specific ISP that provided the ISP interrupt signal based on the above identifier; including, Integrated processing method for exposure adjustment of a multi-channel image sensor.

3. In Paragraph 1, The current luminous value calculated by any one of the above ISPs is, Any one of the above ISPs divides a Bayer image acquired from the image sensor into a plurality of blocks, and includes a current luminance value for each block calculated for each of the plurality of blocks, Any one of the above ISPs, When the calculation for the current luminous value for each of the above blocks is completed, the above ISP interrupt signal is generated, and The above single MCU is, Acquiring the luminous value for each block in conjunction with the generation of the above interrupt signal, Integrated processing method for exposure adjustment of a multi-channel image sensor.

4. In Paragraph 1, Based on the single MCU (Micro Controller Unit) above, the step of calculating an AE (Auto Exposure) value corresponding to the luminous value is: A step of calculating the difference between the target luminance value for each image sensor and the current luminance value; A step of determining whether to change the exposure parameter based on the above difference; and If it is decided to change the above exposure parameters, a step of calculating new exposure parameters and updating the above AE values; including, Integrated processing method for exposure adjustment of a multi-channel image sensor.

5. In Paragraph 4, The step of applying the above AE value to an image sensor connected to any one of the above ISPs is, A step of updating the exposure setting register of the image sensor according to the above AE value; including, Integrated processing method for exposure adjustment of a multi-channel image sensor.

6. In Paragraph 1, The above single MCU is, Whenever a new frame starts, the ISP among the multi-channel image sensors to receive data is determined through the VSYNC interrupt service routine (VSYNC ISR), and The block-by-block luminance value calculated by the ISP is received through the ISP interrupt service routine (ISP ISR), and Calculating AE values ​​based on the block-specific luminance values ​​through the main loop and applying them to the image sensor registers, Integrated processing method for exposure adjustment of a multi-channel image sensor.

7. In Paragraph 1, The above single MCU is, Monitor the VSYNC interrupt signals and ISP interrupt signals provided by multiple ISPs, respectively, and Based on the ISP identifier recognized through the above VSYNC interrupt signal, the ISP interrupt signal is processed separately to perform inter-frame synchronization of the multi-channel image sensor, Integrated processing method for exposure adjustment of a multi-channel image sensor.

8. In Paragraph 1, The above single MCU is, Sequentially referencing luminous data by ISP, It uses a shared buffer that temporarily stores block-by-block luminance values ​​calculated per ISP, and Sequentially processing the update of the exposure setting register for each multi-channel image sensor, Integrated processing method for exposure adjustment of a multi-channel image sensor.

9. Memory for storing one or more instructions; and A processor that executes one or more instructions stored in the memory. Including, The above processor executes the above one or more instructions, A device that performs the method of claim 1.

10. A computer program stored on a computer-readable recording medium that is combined with a computer, which is hardware, to perform the method of claim 1.

Images