Image sensor, electronic device comprising image sensor, and operation method thereof
The image sensor and electronic device address the challenge of capturing HDR images with minimal ghost effects by adaptively controlling exposure to match subject conditions, enhancing dynamic range representation.
Patent Information
- Application Number
- PCT/KR2026/095811
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-07-15
- Filing Date
- 2026-04-10
- Publication Date
- 2026-11-19
AI Technical Summary
Existing image sensors struggle to accurately capture high dynamic range (HDR) images due to artifacts caused by subject movement during multiple exposure operations, resulting in ghost effects and inadequate representation of dynamic range in captured images.
An image sensor and electronic device that selectively extends the dynamic range for dark or saturated areas based on the subject scene, using multiple exposure operations with adaptive auto-exposure controls to minimize ghost effects.
Enables the capture of HDR images without ghost effects by dynamically adjusting exposure settings, effectively representing a wider range of brightness levels in captured scenes.
Smart Images

Figure KR2026095811_19112026_PF_FP_ABST
Abstract
Description
Image sensor, electronic device including image sensor, and method of operation thereof
[0001] The present disclosure relates to an image sensor, an electronic device including an image sensor, and a method of operating the same.
[0002] The dynamic range (DR) of a digital image can be defined as the ratio of the brightest pixel value to the darkest pixel value within an image. The human visual perception system can perceive a higher DR range than typical digital cameras or monitors. It may be difficult for images acquired through image sensors to accurately represent images that humans can perceive. Images with a DR greater than that typically displayed by electronic devices can be referred to as high dynamic range (HDR) images. Images covering a DR that is relatively narrow compared to HDR images can be referred to as standard dynamic range (SDR) or low dynamic range images.
[0003] An electronic device can be configured to generate HDR images through multi-frame high dynamic range (MF HDR) technology. MF HDR technology refers to a technology that expands DR by acquiring and synthesizing multiple images corresponding to different exposure times. The electronic device can acquire information about bright areas from short exposure images and information about dark areas from long exposure images. For example, even when there are large differences in brightness between areas within a single image, an image with high DR can be obtained by blending images with different exposure times.
[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.
[0005] A camera module according to one embodiment may include one or more lenses, an image sensor, and a computational processing circuit. The image sensor may include a light receiving element, a floating node, and a detection circuit. The light receiving element may output electrons as at least a portion of the light passing through the one or more lenses is received. The electrons output from the light receiving element may be configured to be transmitted to the floating node. The detection circuit may be configured to read out a signal from the floating node. The computational processing circuit may be configured to process the signal read out from the image sensor. The floating node may include at least two floating node regions such that the charge of the floating node is read out based on a first conversion gain or a second conversion gain higher than the first conversion gain. Based on the fact that information about a subject scene captured by the camera module satisfies a first condition, the image sensor may be configured to perform a first auto exposure (AE) operation based on a first auto exposure operation criterion. Based on the fact that information regarding the subject scene captured by the camera module satisfies a first condition, the image sensor may be configured to acquire a first image frame from a signal read out based on the first conversion gain for a first exposure interval corresponding to the first automatic exposure operation. Based on the fact that information regarding the subject scene captured by the camera module satisfies a first condition, the image sensor may be configured to acquire a second image frame from a signal read out based on the second conversion gain for the first exposure interval. The computation processing circuit may be configured to acquire a third image frame, which is a composite of the first image frame and the second image frame, as a captured image.Based on the fact that the information regarding the subject scene captured by the camera module satisfies a second condition different from the first condition, the image sensor may be configured to perform a second automatic exposure operation based on a second automatic exposure operation different from the first automatic exposure operation. Based on the fact that the information regarding the subject scene captured by the camera module satisfies a second condition different from the first condition, the image sensor may be configured to acquire a fourth image frame from a signal read out based on the first conversion gain for a second exposure interval corresponding to the second automatic exposure operation. Based on the fact that the information regarding the subject scene captured by the camera module satisfies a second condition different from the first condition, the image sensor may be configured to acquire a fifth image frame from a signal read out based on the second conversion gain for the second exposure interval. The computational processing circuit may be configured to acquire a sixth image frame, which is a composite of the fourth image frame and the fifth image frame, as a captured image.
[0006] In one embodiment, the electronic device may include a touchscreen, a camera module, at least one processor, and memory. The touchscreen may be configured to display a screen and receive touch input. The camera module may include an image sensor. The memory may store computer-executable instructions. The instructions may be executed individually or collectively by the at least one processor to enable the electronic device to obtain information about a subject scene being captured through the camera module based on an image capture request received through the touchscreen. The instructions may be executed individually or collectively by the at least one processor to enable the electronic device to determine an automatic exposure operation criterion based on the information about the subject scene. The above instructions may be executed individually or collectively by the at least one processor to control the camera module so that the electronic device acquires a first image frame from a signal read based on a first conversion gain for a first exposure interval corresponding to the first automatic exposure operation standard, based on the fact that the determined automatic exposure operation standard is the first automatic exposure operation standard. The above instructions may be executed individually or collectively by the at least one processor to control the camera module so that the electronic device acquires a second image frame from a signal read based on a second conversion gain higher than the first conversion gain for the first exposure interval, based on the fact that the determined automatic exposure operation standard is the first automatic exposure operation standard.The above instructions may be executed individually or collectively by the at least one processor so that the electronic device acquires a third image frame, which is a composite of the first image frame and the second image frame, as a captured image.
[0007] In one embodiment, the camera module may include one or more lenses, an image sensor, and a computational processing circuit. A light receiving element may output electrons as at least a portion of the light passing through the one or more lenses is received. A floating node may be configured to transmit electrons output from the light receiving element. A detection circuit may be configured to read out a signal from the floating node. A computational processing circuit may be configured to process a signal read out from the image sensor. The floating node may include at least two floating node regions that cause the charge of the floating node to be read out based on a first conversion gain or a second conversion gain higher than the first conversion gain. The image sensor may be configured to determine an auto exposure (AE) operation standard including a first combination of an exposure range, a conversion gain, and an output gain, and a second combination of a conversion gain and an output gain, based on information about the subject scene captured by the camera module. The image sensor may be configured to perform an exposure operation for the light-receiving element during the exposure interval based on the automatic exposure operation criteria. The image sensor may be configured to acquire a first image frame by reading out the charge for the exposure interval based on the first combination. The image sensor may be configured to acquire a second image frame by reading out the charge for the exposure interval based on the second combination. The camera module may be configured to output a third image frame, which is a composite of the first image frame and the second image frame, as a captured image. The first combination and the second combination may be configured such that the brightness of the first image frame is lower than the brightness of the second image frame.
[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.
[0009] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.
[0010] FIG. 3 is a block diagram illustrating the hardware configuration of an electronic device according to one embodiment.
[0011] FIG. 4 is a diagram illustrating the configuration of an image sensor according to one embodiment.
[0012] FIG. 5 is a circuit diagram illustrating a pixel circuit that constitutes a unit pixel of an image sensor included in an image sensor according to one embodiment.
[0013] FIG. 6 illustrates an example of modeling the operation performed by an image sensor according to one embodiment when there is a large amount of charge generated by a light receiving element.
[0014] FIG. 7 illustrates an example of modeling an operation performed by an image sensor according to one embodiment when the charge generated by a light receiving element is low.
[0015] FIG. 8 is a flowchart illustrating the process of an electronic device capturing an image according to one embodiment.
[0016] FIG. 9 is a block diagram illustrating the structure of an electronic device acquiring and processing image data according to one embodiment.
[0017] FIG. 10 is a flowchart illustrating the process of an electronic device capturing an image according to one embodiment.
[0018] FIG. 11 illustrates a structure in which noise is amplified during the process of an image sensor acquiring image data according to one embodiment.
[0019] FIG. 12 illustrates an example in which an electronic device according to one embodiment synthesizes image frames based on a first synthesis method.
[0020] FIG. 13 illustrates an example in which an electronic device according to one embodiment synthesizes image frames based on a second synthesis method different from a first synthesis method.
[0021] FIG. 14 is a circuit diagram illustrating a pixel circuit in which a floating node of an image sensor according to one embodiment includes three floating node regions.
[0022] FIG. 15 is a diagram illustrating examples of patterns of a light-receiving element, a micro lens, and a color filter arranged in an image sensor according to one embodiment.
[0023] FIG. 16 is a diagram illustrating an example of a color pattern of pixels included in an image obtained through an image sensor according to one embodiment.
[0024] FIG. 17 is a diagram illustrating an example of a color pattern of pixels included in an image obtained through an image sensor according to one embodiment.
[0025] FIG. 18 illustrates an automatic exposure program diagram set in an image sensor to perform an automatic exposure operation of an image sensor according to one embodiment.
[0026] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the contents of this disclosure. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.
[0027] When generating an HDR image by synthesizing multiple images obtained by performing multiple exposure operations, artifacts may occur in the synthesized HDR image if the subject moves between the times when the exposure operations are performed, as the exposure operations to obtain the multiple images are performed at different times. For example, a ghost effect may occur in which the image of a moving subject overlaps and appears blurred. An image sensor, an electronic device including an image sensor, and a method of operation according to one embodiment of the present disclosure may be intended to obtain an HDR image without a ghost effect caused by the movement of the subject.
[0028] An image sensor according to one embodiment of the present disclosure, an electronic device including an image sensor, and a method of operation thereof may be for selectively applying whether to extend the dynamic range for dark areas or to extend the dynamic range for saturated areas depending on the subject scene.
[0029] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.
[0030] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).
[0031] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.
[0032] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.
[0033] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).
[0034] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).
[0035] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).
[0036] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.
[0037] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.
[0038] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).
[0039] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
[0040] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
[0041] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0042] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.
[0043] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.
[0044] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).
[0045] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
[0046] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).
[0047] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.
[0048] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).
[0049] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.
[0050] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.
[0051] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.
[0052] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.
[0053] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.
[0054] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0055] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.
[0056] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.
[0057] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.
[0058] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, a wide-angle lens or a telephoto lens.
[0059] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include one image sensor selected from image sensors with different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.
[0060] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) containing it. This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) may be implemented as, for example, an optical image stabilizer. The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.
[0061] The image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) can perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be configured as at least part of the processor (120) or as a separate processor operating independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after further image processing by the processor (120).
[0062] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.
[0063] FIG. 3 is a block diagram illustrating the hardware configuration of an electronic device (101) according to one embodiment.
[0064] In one embodiment, the electronic device (101) may include an image sensor (230) and a processor (120) included in a camera module (e.g., the camera module (180) of FIG. 1 and FIG. 2). The processor (120) may include an image signal processor (e.g., the image signal processor (ISP) (260) of FIG. 2).
[0065] In one embodiment, the image sensor (230) may be a CMOS (complementary metal oxide semiconductor) sensor. A plurality of individual pixels may be integrated in the image sensor (230). An individual pixel (232) is the smallest unit for acquiring image data within the image sensor (230) and may be referred to as a sensor pixel. An individual pixel (232) may include at least some of a microlens, at least some of a color filter, and a light receiving element (e.g., a photodiode (PD)). An individual pixel (232) may convert input light into an electrical signal as a type of light detector. For example, the image sensor (230) may amplify the current generated by the photoelectric effect of the light receiving element through the light received through the lens assembly (e.g., the lens assembly (210) of FIG. 2). For example, individual pixels (232) may include a photoelectric transformation element (or position sensitive detector; PSD) and a plurality of transistors.
[0066] In one embodiment, light information incident through a lens assembly (e.g., lens assembly (210) of FIG. 2) can be converted into an electrical signal by an image sensor (230) and input to a processor (120). If the image signal processor (e.g., image signal processor (260) of FIG. 2) and the image sensor (230) are physically separated, a sensor interface conforming to a specified standard can electrically connect the image sensor (230) and the image signal processor (260).
[0067] In one embodiment, the processor (120) may perform image processing on electrically converted image data. The process performed by the processor (120) may be divided into pre-ISP (hereinafter, pre-processing) and ISP chain (hereinafter, post-processing). Image processing up to the demosaicing process may be classified as pre-processing, and image processing after the demosaicing process may be classified as post-processing. The pre-processing process may include 3A processing, lens shading correction, edge enhancement, dead pixel correction, and knee correction. The 3A may include at least one of AWB (auto white balance), AE (auto exposure), and AF (auto focusing). The post-processing process may include at least one of changing the sensor index value, changing tuning parameters, and adjusting the aspect ratio. The post-processing process may include processing image data output from the image sensor (230) or an image output from the scaler. The processor (120) can adjust at least one of the image's contrast, sharpness, saturation, and dithering through a post-processing process. Here, the contrast, sharpness, and saturation adjustment procedures may be executed in a YUV color space, and the dithering procedure may be executed in an RGB (Red, Green, Blue) color space. Some of the pre-processing steps may be performed in the post-processing step, or some of the post-processing steps may be performed in the pre-processing step. Additionally, some of the pre-processing steps may overlap with some of the steps in the post-processing step.
[0068] In one embodiment, the image sensor (230) may include a plurality of unit pixels (234). A unit pixel (234) may include one or more individual pixels (232). For example, one unit pixel (234) may include four or nine individual pixels (232). In one embodiment, the image sensor (230) may further include an analog digital converter (ADC) (236).
[0069] In one embodiment, a plurality of unit pixels (234) may be integrated in an image sensor (230). Each unit pixel (234) may include a color filter, one or more microlenses, and two or more light-receiving elements. One light-receiving element included in the unit pixel (234) may correspond to one individual pixel (232).
[0070] In one embodiment, the image sensor can convert analog data into digital data through an ADC (236).
[0071] FIG. 4 is a diagram illustrating the configuration of an image sensor according to one embodiment (e.g., the image sensor (230) of FIG. 2 and FIG. 3).
[0072] In one embodiment, the image sensor may include a micro lens array (MLA) (411), a color filter array (413), a light receiving unit (415), and a processing unit (417).
[0073] In one embodiment, the micro-lens array (411) may be configured such that a beam of light (421) that passes through a lens unit (e.g., lens assembly (210) of FIG. 2) and forms an image on an image sensor is collected by a light receiving element of the light receiving unit (415). The beams of light (423) that pass through the micro-lens array (411) may pass through a color filter array (413), at least partially blocking wavelengths other than the band corresponding to a specific color. The beams of light (425) that pass through the color filter array (413) may be detected by a light receiving element (e.g., a photodiode) of the light receiving unit (415). The light receiving unit (415) may include a light receiving element that generates a charge and converts it into an electrical signal upon receiving light, and a circuit that selectively reads out the charge of the light receiving element. Between the light receiving unit (415) and the operation unit (417), a circuit for digitizing the signal read from the light receiving unit (415) or reducing noise may be arranged.
[0074] In one embodiment, the microlens array (411) may be arranged to correspond to at least one light-receiving element. For example, when viewed from the direction in which the light beam (421) is incident, the area in which a single microlens included in the microlens array (411) is arranged may overlap at least partially with the area in which a plurality of light-receiving elements are arranged. The microlenses included in the microlens array (411) may be arranged in a different color channel from adjacent microlenses, but a plurality of microlenses corresponding to the same color channel may be arranged adjacent to each other. The arrangement between the microlens array (411), the color filter array (413), and the light-receiving unit (415) may be configured differently depending on the type of image sensor. For example, in FIG. 4, the color filter array (413) is shown in the form of a Bayer pattern, but the color channels within the color filter array (413) may be arranged in a different form. The areas within the color filter (413) corresponding to a plurality of adjacent microlenses may be configured to include the same color channel. For example, an image sensor including a color filter (413) may include a structure in which the pattern shown in FIG. 15 is repeated.
[0075] In one embodiment, the operation unit (417) may include an operation processing circuit that performs an operation to process electrical data (427) output from the light receiving unit (415). The operation unit (417) may output acquired data based on the result of the operation. The output of the operation unit (417) may be the output of an image sensor (e.g., the image sensor (230) of FIGS. 2 and FIGS. 3). In one embodiment, the operation unit (417) may perform an operation to generate a composite image frame by synthesizing a plurality of image frames acquired by the image sensor. In one embodiment, the operation unit (417) may perform an operation to calibrate the read-out data as an operation to process the electrical data (427). For example, the operations performed by the operation unit (417) may include at least one of operations to reduce deviation between pixels due to optical asymmetry or the relative position of the sensor, operations to reduce noise generated in the analog signal, operations to remove defects, operations to perform remosaic, or operations according to a specific application field (e.g., proximity sensor function, timing control function, HDR (high dynamic range) tone mapping function). The sensor output (429) output from the operation unit (417) may be input to at least one processor (e.g., application processor) through an interface. An electronic device (101) according to one embodiment may be implemented so that the operations of the operation unit (417) are performed by a separate processor physically separated from the image sensor (e.g., ISP (260) of FIG. 2, processor (120) of FIG. 3).
[0076] FIG. 5 is a circuit diagram illustrating a pixel circuit that forms a unit pixel (234) of an image sensor included in an image sensor according to one embodiment (e.g., the image sensor (230) of FIG. 2 and 3).
[0077] According to one embodiment, the unit pixel (234) may include a plurality of light receiving elements (510), transfer gates (TG) (520) corresponding to each light receiving element (510), floating diffusion nodes (FD) (530), source follower (SF) (550), row selector (SEL) (560), reset gate (RST) (570), analog amplifier (581), ADC (583), and digital amplifier (585). The unit pixel (234) may further include a double conversion gain (DCG) gate (540) and a first floating node region (FD1) (535).
[0078] Referring to FIG. 5, a unit pixel (234) may include four light-receiving elements (510). For example, each light-receiving element may include a photodiode. For example, the unit pixel (234) may include four light-receiving elements having a 2x2 array (e.g., a first light-receiving element (511), a second light-receiving element (512), a third light-receiving element (513), and a fourth light-receiving element (514)). According to one embodiment, the unit pixel (234) may include an FD (530) connected to the light-receiving elements (510). The number of light-receiving elements sharing the FD (530) may vary. In one embodiment, the unit pixel (234) may include five or more light-receiving elements connected to the FD (530). For example, a unit pixel (234) may include a plurality of light-extinguishing elements arranged in an NxN array (N is a natural number) (e.g., nine light-receiving elements arranged in a 3x3 array) and FDs (530) connected to the light-receiving elements. For example, a unit pixel (234) may include sixteen light-receiving elements arranged in a 4x4 array and FDs (530) connected to the light-receiving elements. For example, a unit pixel (234) may include eight light-receiving elements having a 2x4 array.
[0079] In one embodiment, the individual pixel (232) of FIG. 3 may correspond to any one of the first light receiving element (511), the second light receiving element (512), the third light receiving element (513), or the fourth light receiving element (514). For example, the individual pixel (232) may include the first light receiving element (511). The individual pixel (232) may further include the first TG (521).
[0080] In one embodiment, the charge accumulated in the light receiving element (510) during the exposure time can be transferred to the FD (530) when the TG (520) is on. For example, the charge accumulated in the first light receiving element (511) can be transferred to the FD (530) when the first TG (521) is on. In one embodiment, the image sensor (230) can acquire an analog signal corresponding to the charge transferred to the FD (530). For example, the analog signal may include an electrical signal (or voltage signal). The analog signal may include information regarding the amount of charge accumulated in the light receiving element (510) during the exposure time. The information regarding the amount of accumulated charge may correspond to light quantity information regarding the amount of light received by the light receiving element (510).
[0081] In one embodiment, the image sensor (230) can acquire an analog signal corresponding to light quantity information acquired through at least one of the light receiving elements (510). For example, the image sensor (230) can acquire an analog signal for each individual pixel (232). For example, the image sensor (230) may acquire an analog signal from two or more of the light receiving elements (510). For example, the image sensor (230) may acquire an analog signal for each unit pixel (234).
[0082] In one embodiment, the charge transferred to the FD (530) can be read out through the SF (550) and output as an electrical analog signal. The analog signal can be amplified through an analog amplifier (581). The image sensor (230) can control the degree to which the analog signal is amplified by adjusting the gain value of the analog amplifier (581). The image sensor (230) can convert the analog signal into a digital signal through an ADC (583). The digital signal converted through the ADC (583) can be amplified through a digital amplifier (585). The digital data output through the digital amplifier (585) can be included in the image frame acquired by the image sensor (230).
[0083] In one embodiment, the image sensor (230) can change the SEL (560) from off to on to output image data of a specific row.
[0084] In one embodiment, the image sensor (230) may perform a correlated double sampling operation to reduce noise from the output signal. For example, the image sensor (230) may switch the RST (570) to on to reset the charge accumulated in the FD (530) and read out the remaining reset data after the reset. The reset data may indicate the measured value of the charge remaining after the reset. The image sensor (230) may transfer the charge accumulated in the light receiving element (510) to the FD (530) after switching the RST (570) to off. The image sensor (230) may read out the charge transferred to the FD (530) to obtain readout data. By excluding the reset data from the readout data, the noise value caused by the charge in the reset state may be removed. The image sensor (230) may obtain digital data based on the difference between the readout data and the reset data. In one embodiment, the image sensor (230) can reduce readout noise by performing the operation of acquiring readout data multiple times and calculating the average of the readout data acquired multiple times. The operation mode of performing the operation of acquiring readout data multiple times may be referred to as an LN (low noise) mode.
[0085] In one embodiment, a floating node included in the pixel circuit of an image sensor (230) may include at least two floating node regions (e.g., FD (530) and FD1 (535)). The conversion gain of the image sensor (230) may vary depending on the floating node storage capacity due to the junction capacitance of the floating node regions. The smaller the floating node storage capacity, the higher the conversion gain may be. When the image sensor (230) reads out charges stored in FD (530) and FD1 (535), a signal may be read out based on a first conversion gain (low conversion gain, LCG). When the image sensor reads out charges stored in FD (530), a signal may be read out based on a second conversion gain (high conversion gain, HCG) which is higher than the first conversion gain.
[0086] In one embodiment, the image sensor (230) can adjust the conversion gain according to the on / off state of the DCG gate (540). For example, when the DCG gate (540) is in the on state, the image sensor (230) can read out a signal based on the first conversion gain. When the DCG gate (540) is in the off state, the image sensor (230) can read out a signal based on the second conversion gain.
[0087] In one embodiment, the image sensor (230) can read out light quantity data input by an exposure operation performed during one time interval as a second conversion gain, and then read out as a first conversion gain. For example, the image sensor (230) can read out a signal based on the second conversion gain after transferring the charge accumulated by the first light receiving element (511) during the exposure interval to the FD (530) while the DCG gate (540) is in an off state. Subsequently, the image sensor (230) can read out a signal based on the first conversion gain by converting the DCG gate (540) to an on state.
[0088] FIG. 6 illustrates an example of modeling the operation performed by an image sensor (230) according to one embodiment when there is a large amount of charge generated by a light receiving element (510).
[0089] Referring to reference number 610 in FIG. 6, when an exposure operation is performed while the TG (520) is in the off state, charge can be accumulated by the light receiving element (510). If the amount of light received by the light receiving element (510) is large, the amount of charge accumulated by the light receiving element (510) may be large.
[0090] Referring to reference number 620 in FIG. 6, when the TG (520) is switched to the on state, the charge accumulated by the light receiving element (510) can be transferred to the FD (530). When the DCG gate (540) is in the off state, the charge accumulated by the light receiving element (510) can be preferentially accumulated in the FD (530). If the amount of charge accumulated by the light receiving element (510) exceeds the capacitance of the FD (530), the capacitance of the FD (530) may be saturated by the charge accumulated in the FD (530). The image sensor (230) can acquire the signal of the saturated FD (530) when reading out a signal based on a high conversion gain when the amount of light received by the light receiving element (510) is large. The image sensor (230) can acquire an image frame from the signal read out based on the high conversion gain.
[0091] Referring to reference numeral 630 in FIG. 6, as the DCG gate (540) is switched to the ON state, some of the charge accumulated in the FD (530) can be transferred to FD1 (535). The image sensor (230) can acquire a signal regarding the charge accumulated in the FD (530) and FD1 (535). The image sensor (230) can acquire another image frame from the acquired signal based on a conversion gain (low conversion gain) corresponding to the FD (530) and FD1 (535). Thus, when the amount of charge generated by the light receiving element (510) exceeds the capacitance of the FD (530) (when photographing a subject with high illumination), the saturation of the readout signal can be reduced by reading out the signal based on a low conversion gain.
[0092] In the present disclosure, an image frame obtained based on a first conversion gain may be referred to as a first image frame. In the present disclosure, an image frame obtained based on a second conversion gain may be referred to as a second image frame. In one embodiment, an electronic device (101) may synthesize a first image frame and a second image frame to obtain a high dynamic range (HDR) image in which the dynamic range is expanded compared to the first image frame and the second image frame. The operation of synthesizing the first image frame and the second image frame may be performed within an image sensor (230). For example, the operation of synthesizing the first image frame and the second image frame may be performed by a computation processing circuit included in the computation unit (417) of FIG. 4. However, it is not limited thereto. For example, the operation of synthesizing the first image frame and the second image frame may be performed by a separate processor connected to the image sensor (230) (e.g., the image signal processor (260) of FIG. 2, the processor (120) of FIG. 1 and FIG. 3).
[0093] FIG. 7 illustrates an example of modeling an operation performed by an image sensor (230) according to one embodiment when the charge generated by a light receiving element (510) is low.
[0094] Referring to reference number 710 in FIG. 7, when an exposure operation is performed while the TG (520) is in the off state, charge can be accumulated by the light receiving element (510). If the amount of light received by the light receiving element (510) is small, the amount of charge accumulated by the light receiving element (510) may be small.
[0095] Referring to reference number 720 in FIG. 7, when the TG (520) is switched to the on state, the charge accumulated by the light receiving element (510) can be transferred to the FD (530). When the DCG gate (540) is in the off state, the charge accumulated by the light receiving element (510) can be accumulated in the FD (530). When the image sensor (230) reads out the charge accumulated in the FD (530), it can obtain a signal with a high gain value applied. The signal read out based on the high conversion gain can obtain a sufficiently large signal even when the amount of charge accumulated by the light receiving element (510) is low. The image sensor (230) can obtain an image frame from the signal read out based on the high conversion gain.
[0096] Referring to reference numeral 730 in FIG. 7, as the DCG gate (540) is switched to the ON state, some of the charge accumulated in the FD (530) can be transferred to the FD1 (535). The image sensor (230) can acquire a signal regarding the charge accumulated in the FD (530) and the FD1 (535). The image sensor (230) can acquire another image frame from the acquired signal based on a conversion gain (low conversion gain) corresponding to the FD (530) and the FD1 (535). When the amount of charge accumulated by the light receiving element (510) is low, the signal read out based on the low conversion gain may have a small signal magnitude. Therefore, when the illumination of the subject scene captured through the image sensor (230) is low, it may be advantageous to read out the signal based on a high conversion gain.
[0097] In the present disclosure, an image frame obtained based on a first conversion gain may be referred to as a first image frame. In the present disclosure, an image frame obtained based on a second conversion gain may be referred to as a second image frame. In one embodiment, an electronic device (101) may synthesize a first image frame and a second image frame to obtain a high dynamic range (HDR) image in which the dynamic range is expanded compared to the first image frame and the second image frame. The operation of synthesizing the first image frame and the second image frame may be performed within an image sensor (230). For example, the operation of synthesizing the first image frame and the second image frame may be performed by a computation processing circuit included in the computation unit (417) of FIG. 4. However, it is not limited thereto. For example, the operation of synthesizing the first image frame and the second image frame may be performed by a separate processor connected to the image sensor (230) (e.g., the image signal processor (260) of FIG. 2, the processor (120) of FIG. 1 and FIG. 3).
[0098] FIG. 8 is a flowchart (800) illustrating a process in which an electronic device (101) captures an image according to one embodiment. The operation of the electronic device (101) mentioned in the present disclosure may be understood as a computational processing or a control operation over a component of the electronic device (101) performed by at least one processor included in the electronic device (101) executing computer-executable instructions stored in memory individually or collectively. The operation of the electronic device (101) mentioned in the present disclosure may also be performed by a camera module.
[0099] According to one embodiment, in operation 810, the electronic device (101) may determine an AE operation standard for the camera module (180) to perform an auto exposure (AE) operation. The AE operation standard may include setting parameters for performing the AE operation of the camera module (180) to achieve a target gain value. For example, the AE operation standard may be composed of a look-up table stored in memory (130, 250) that defines a mapping relationship of parameters (e.g., at least one of shutter speed, sensitivity, or aperture value) to the target gain value. The target gain value may refer to a value determined by the electronic device (101) to perform the AE operation. For example, the lower the brightness of the subject scene (or the illumination of the surrounding environment), the higher the target gain value may be determined to be in order to obtain an image of appropriate brightness.
[0100] In one embodiment, the electronic device (101) may select one of a plurality of AE operation criteria stored in memory (130, 250) based on information about the subject scene. The AE operation criteria may be determined by the user setting whether to extend the DR for the dark area or the DR for the bright area. Information about the subject scene may include information about which scene is being captured through the camera module (180). For example, information about the subject scene may include at least one of a target gain value, information about the dark area, information about the saturated area, or a shooting mode (e.g., still image mode / video mode). For example, information about the dark area may include at least one of the ratio of the dark area to the subject scene, information about the location of the dark area within the image in which the subject scene is captured, or information about the number of dark areas. A dark area may refer to an area where the illumination is below a threshold. For example, information regarding the saturated area may include at least one of the number of subjects in which the pixel values of image pixels included within the subject scene are saturated, information regarding the location of the saturated area within the image in which the subject scene is captured, or information regarding the ratio of the area occupied by the saturated area. The saturated area may refer to an area in which the pixel values of image pixels have saturated values within a dynamic range. When the electronic device (101) performs video recording, it may sequentially obtain information regarding the subject scene from previously captured images.
[0101] In one embodiment, the AE operation standard may include a combination of setting parameters for reading multiple image frames having different brightness for a single exposure interval. For example, the AE operation standard may include an AE setting parameter for reading a first image frame and an AE setting parameter for reading a second image frame that has a brighter brightness (luminance) than the first image frame, based on the charge generated in a single exposure interval through the image sensor (230). For example, the difference between the brightness of the second image frame and the brightness of the first image frame may correspond to the difference in gain values between HCG and LCG.
[0102] In one embodiment, a plurality of AE operation criteria may include a first AE operation criterion and a second AE operation criterion. The first AE operation criterion may include setting parameters that set the exposure operation so that a first image frame with a darker brightness among the image frames to be acquired becomes a reference exposure image. The second AE operation criterion may include setting parameters that set the exposure operation so that a second image frame with a darker brightness among the image frames to be acquired becomes a reference exposure image. In the present disclosure, a reference exposure image may refer to an image in which proper exposure is achieved according to the luminance of the subject scene through AE operation. An image acquired to have a brightness darker than the reference exposure image may be referred to as an underexposed image. An image acquired to have a brightness brighter than the reference exposure image may be referred to as an overexposed image.
[0103] According to one embodiment, in operation 820, the image sensor (230) may perform an exposure operation such that charge generated in the light receiving element is accumulated during an exposure period determined based on a determined AE operation criterion. In operation 830, the image sensor (230) may acquire a first image frame by reading out a signal for the exposure period of operation 820 based on the AE operation criterion. In operation 835, the image sensor (230) may acquire a second image frame read out based on a gain value higher than the first image frame by reading out a signal for the exposure period of operation 820 based on the AE operation criterion. Operation 835 may be performed prior to operation 830, but is not limited thereto.
[0104] In one embodiment, the first image frame may be obtained from a signal read out based on a first conversion gain. The second image frame may be obtained from a signal read out based on a second conversion gain that is higher than the first conversion gain. For example, an example of a first AE operation criterion that makes the image frame with lower brightness among the image frames to be obtained the reference exposure image may be configured as shown in Table 1 below.
[0105] case 1 case 2 case 3 First Image Second Image First Image Second Image First Image Second Image Still RN Glass Target Gain 1 4 16 Sensor Output 12 bit 12 bit 12 bit Shutter Speed 1 / 60 s 1 / 60 s 1 / 60 s 1 / 60 s 1 / 60 s 1 / 60 s Conversion Gain 1 (LCG) 4 (HCG) 4 (HCG) 4 (HCG) 4 (HCG) Analog Gain 1 1 4 4 16 FWC 100000 25000 25000 6250 6250 15 6 2.5 Input Bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit Synthesis First Synthesis Method First Synthesis Method First Synthesis Method
[0106] Table 1 assumes the operation of an image sensor with a maximum full well capacitance (FWC) of 100,000e, a conversion gain of 4 times that of the HCG relative to the LCG, a frame rate of 60 fps, a maximum analog gain of 16, and a 10-bit ADC. In the present disclosure, the maximum full well capacitance may refer to the maximum capacitance of a photodetector (e.g., photodetector (510) of FIG. 5). When a gain value (e.g., conversion gain, analog gain, digital gain) increases, the effective full well capacitance may decrease to the value obtained by dividing the maximum full well capacitance by the gain value. The FWC mentioned in the tables of the present disclosure may indicate the effective full well capacitance. In the present disclosure, the analog gain may be referred to as the sensor gain value. Referring to Table 1, image frames are read according to the first AE operation criteria, and if the target gain value determined to perform the AE operation is less than 4, the image sensor (230) can operate based on case 1. When operating based on case 1, the image sensor (230) can read the first image frame based on the first conversion gain (e.g., LCG) and read the second image frame based on the second conversion gain (e.g., HCG). To maximize the exposure interval while operating based on the specified frame rate, the shutter speed can be set to 1 / 60 s. When operating based on HCG, the FWC can be 25000e, which is 1 / 4 of the maximum FWC. The camera module (180) including the image sensor (230) can obtain a 12-bit third image frame (HDR image) by synthesizing a 10-bit first image frame and a 10-bit second image frame. While operating based on case 1, as the target gain value increases from 1, the analog gain can also increase from 1.
[0107] In one embodiment, among the image frames acquired based on the first AE operation criterion, the first image frame may be a reference exposure image derived based on a gain value corresponding to a target gain value. In operation 840, by synthesizing the first image frame corresponding to the reference exposure image and the second image frame corresponding to the overexposure image, the electronic device (101) may acquire a third image frame in which the dynamic range is expanded in a direction to represent a saturated region. In one embodiment, when the image sensor (230) acquires image frames based on the first AE operation criterion, the electronic device (101) may synthesize the image frames based on a first synthesis method to expand the dynamic range in a direction to represent a saturated region. For example, the first synthesis method may be based on an alpha blending technique. For example, the alpha blending technique may include a synthesis method that acquires a composite image based on the following mathematical formula 1.
[0108]
[0109] In mathematical formula 1, C may mean a composite image, A may mean a first image frame (LCG image), and B may mean a second image frame (HCG image).
[0110] Referring to Table 1, in one embodiment, the first AE operation criterion may be configured such that when the target gain value is greater than or equal to the first threshold (e.g., the difference between the first conversion gain (LCG) and the second conversion gain (HCG) (4 times)), the image sensor (230) reads out both the first image frame and the second image frame based on the second conversion gain (HCG). Since amplifying the signal being read out based on the conversion gain is more advantageous for reducing noise amplification than amplifying it based on the analog gain, when the target gain value is greater than or equal to the difference between the first conversion gain and the second conversion gain (4 times) and the first image frame is a reference exposure image, the image sensor (230) may read out the first image frame based on the second conversion gain instead of lowering the analog gain relative to the target gain value to 1 / 4 times compared to case 1 according to case 2 or case 3 according to the first AE operation criterion. The image sensor (230) can read out a second image frame by applying a second analog gain that is four times higher than the first analog gain for reading out a first image frame. The FWC of case 2 may be 1 / 4, which is the inverse of the difference between the HCG and LCG of the FWC of case 1.
[0111] In one embodiment, the first AE operation reference may be configured such that the first image frame of case 2 is based on the analog gain of LCG and 4, and the first image frame of case 3 is acquired based on the analog gain of LCG and 16.
[0112] For example, an example of a second AE operation criterion that makes the brighter image frame among the acquired image frames the reference exposure image can be configured as shown in Table 2 below.
[0113] case 4 case 5 case 6 First Image Second Image First Image Second Image First Image Second Image Night Video Saturated Glass Target Gain 1 4 1 6 Sensor Output 1 2 bit 1 2 bit 1 2 bit Shutter Speed 1 / 240 s 1 / 240 s 1 / 60 s 1 / 60 s 1 / 60 s 1 / 60 s Conversion Gain 1 (LCG) 4 (HCG) 1 (LCG) 4 (HCG) 4 (HCG) 4 (HCG) Analog Gain 1 1 1 1 4 FWC 100000 25000 100000 25000 25000 6 250 Input Bit 1 0 bit 10 bit 10 bit 10 bit 10 bit 10 bit Synthesis Second Synthesis Method Second Synthesis Method Second Synthesis Method
[0114] Table 2 assumes the operation of an image sensor with a maximum full well capacitance (FWC) of 100,000e, an HCG having a conversion gain of 4 times that of LCG, a frame rate of 60 fps, a maximum analog gain of 16, and an ADC configured as 10 bits. Referring to Table 2, image frames are read according to the second AE operation criteria, and if the target gain value determined to perform the AE operation is less than 4, the image sensor (230) can operate based on case 4. When operating based on case 4, the image sensor (230) can read a first image frame based on a first conversion gain (e.g., LCG) and read a second image frame based on a second conversion gain (e.g., HCG). In order to ensure that the second image frame, which is read out based on the second conversion gain, is read out based on a gain corresponding to the target gain value, the analog gain cannot be lowered to less than 1, so the shutter speed can be set to 1 / 240 s so that the exposure interval becomes 1 / 4. The first image frame can be acquired as an image that is four times darker than the second image frame (the difference between HCG and LCG). The camera module (180) including the image sensor (230) can acquire a 12-bit third image frame (HDR image) by synthesizing the 10-bit first image frame and the 10-bit second image frame. While operating based on case 4, the shutter speed can be slowed down so that the exposure interval becomes longer as the target gain value increases from 1.
[0115] In one embodiment, the second image frame of case 4 may be a reference exposure image derived based on a gain value corresponding to a target gain value. By synthesizing the second image frame corresponding to the reference exposure image and the first image frame corresponding to the underexposure image, the electronic device (101) can obtain a third image frame with a dynamic range expanded in a direction for expressing a dark area. In one embodiment, when image frames are obtained based on a second AE operation standard, the electronic device (101) can synthesize image frames based on a second synthesis method to expand the dynamic range in a direction for expressing a dark area. For example, the second synthesis method may be configured based on a technique for generating an HDR image by synthesizing a short exposure image and a long exposure image.
[0116] Referring to Table 2, in one embodiment, the second AE operation standard may be configured such that when the target gain value is greater than or equal to the first threshold (e.g., the difference between the first conversion gain (LCG) and the second conversion gain (HCG) (4 times)) and less than the second threshold (e.g., 16 times), the image sensor operates at a shutter speed of 1 / 60 s, which is the same as the shutter speed of the first AE operation standard, to ensure sufficient exposure intervals as in case 5. The second AE operation standard may be configured such that in case 5, the image sensor (230) reads out the first image frame based on the first conversion gain (LCG) and reads out the second image frame based on the second conversion gain (HCG). The second AE operation standard may increase the analog gain value while maintaining the shutter speed as the target gain value increases from 4 in case 5.
[0117] Referring to Table 2, in one embodiment, the second AE operation criterion may be configured such that when the target gain value is greater than or equal to the second threshold (e.g., 16 times), the image sensor (230) reads both the first image frame and the second image frame based on the second conversion gain (HCG). The second AE operation criterion may, in case 6, read the first image frame based on the second conversion gain instead of lowering the analog gain relative to the target gain value to 1 / 4 times in case 5. The second AE operation criterion may read the second image frame by applying a second analog gain that is 4 times higher than the first analog gain for reading the first image frame.
[0118] In one embodiment, among the image frames acquired based on the second AE operation criteria, the second image frame may be a reference exposure image derived based on a gain value corresponding to a target gain value. In operation 840, by synthesizing the second image frame corresponding to the reference exposure image and the first image frame corresponding to the underexposure image, the electronic device (101) may acquire a third image frame with a dynamic range expanded in a direction for expressing a dark area. In one embodiment, when the image sensor (230) acquires image frames based on the second AE operation criteria, the electronic device (101) may synthesize the image frames based on a second synthesis method to expand the dynamic range in a direction for expressing a dark area. The second synthesis method may have a different ratio for synthesizing image frames compared to the first synthesis method, thereby configuring the blending area of the image frames differently.
[0119] According to one embodiment, in operation 810, the electronic device (101) may determine an AE operation criterion by considering whether a still image is being captured or a video is being captured. For example, when a still image is being captured, the electronic device (101) may control the image sensor (230) to read out image frames based on the first AE operation criterion. According to one embodiment, in operation 810, the electronic device (101) may determine an AE operation criterion based on a target gain value determined to correspond to brightness information of the subject scene. For example, when the target gain value is less than the first threshold (e.g., when the target gain value is less than 4), the electronic device (101) may control the image sensor (230) to operate according to case 1 by performing an exposure operation and an image frame reading operation based on the first AE operation criterion. For example, when the target gain value is greater than or equal to the first threshold (e.g., when the target gain value is 4 or greater), the electronic device may control the image sensor (230) to operate according to case 5 or case 6 by performing exposure operations and reading operations of image frames based on the second AE operation criteria. According to one embodiment, in operation 810, the electronic device (101) may determine the AE operation criteria based on information regarding the saturated area. The information regarding the saturated area may include, for example, at least one of the number of subjects with saturated pixel values, the area of the saturated pixel value, or the ratio of the saturated pixel value to the total area of the image frame. For example, when the number of subjects with saturated pixel values is less than the threshold, the need for dynamic range expansion in the direction of expressing the saturated area is low, so the electronic device (101) may control the image sensor (230) to perform operations 820, 830, and 835 based on the first AE operation criteria.For example, when photographing a night scene containing multiple streetlights, a significant difference in luminance levels due to multiple light sources may occur, so the electronic device (101) may operate based on the second AE operation standard. According to one embodiment, in operation 810, the electronic device (101) may determine the AE operation standard based on information regarding the dark area. For example, if the dark area is widely formed with low-frequency components, the electronic device (101) may control the image sensor (230) to perform operations 820, 830, and 835 based on the second AE operation standard. For example, in the case of a scene where there are areas saturated by artificial lighting and low-frequency components are widely formed in the dark area, noise removal in the dark area is easy, so the electronic device (101) may control the image sensor (230) according to the second AE operation standard so that the image sensor (230) operates based on case 5.
[0120] FIG. 9 is a block diagram illustrating the structure of an electronic device (101) according to one embodiment acquiring and processing image data.
[0121] In one embodiment, the electronic device (101) may perform an HDR control operation (910) that controls the process of generating an HDR image based on information (900) about the subject scene. The subject scene may refer to a scene included in a previously acquired image. To acquire information (900) about the subject scene, the electronic device (101) may include at least one of a block that determines brightness through image analysis, a block that determines the presence or absence of saturation, or a block that determines the area of a dark part. An AE operation standard may be determined through the HDR control operation (910).
[0122] In one embodiment, the electronic device (101) may set an AE operation (920) based on a determined AE operation standard. The electronic device (101) may acquire a first image (LCG 10-bit image) and a second image (HCG 10-bit image) by performing a photodetector array (e.g., an APS array (active pixel sensor array)) and an ADC operation (930) based on the set AE operation (920). In the present disclosure, the exposure integration time (EIT) is a technique for controlling the time during which an image sensor collects light, and may indicate a technique that allows processing a wide contrast ratio by combining multiple exposure times when capturing HDR images.
[0123] In one embodiment, the electronic device (101) may include an image synthesis block (940) that synthesizes a first image frame and a second image frame according to a determined AE operation standard. The image synthesis block (940) may be configured to synthesize image frames based on a first synthesis method (941) or a second synthesis method (943). The first synthesis method (941) may be performed when synthesizing an HDR image by performing an AE operation based on a first transformation gain (e.g., LCG). For example, the first synthesis method (941) may synthesize image frames based on an alpha blending technique. The second synthesis method (943) may be performed when synthesizing an HDR image by performing an AE operation based on a second transformation gain (e.g., HCG). For example, the second synthesis method (943) may be for generating an HDR image by synthesizing a short exposure image and a long exposure image. The image synthesis block (940) may be configured as a computational processing circuit within the image sensor (230) or implemented through a processor distinct from the image sensor (230) (e.g., the processor (120) of FIG. 1 and 3 or the image signal processor (260) of FIG. 2). If the image synthesis block (940) is implemented through a processor distinct from the image sensor (230), two 10-bit images must be transmitted from the image sensor (230) through an interface; however, if the image synthesis block (940) is included within the image sensor (230), one 12-bit image is transmitted from the image sensor (230) through an interface, thereby reducing the amount of data that must be transmitted through the interface.
[0124] In one embodiment, the electronic device (101) can provide an image captured through the camera module (180) by performing image processing (950) on the synthesized image. The image processing (950) can be performed through a computational processing circuit included in the image sensor (230) or an image signal processor (260).
[0125] FIG. 10 is a flowchart (1000) illustrating the process of an electronic device (101) capturing an image according to one embodiment.
[0126] According to one embodiment, in operation 1010, the electronic device (101) can obtain information about the subject scene. For example, the electronic device (101) can obtain at least one of illuminance information, information about saturated areas, or information about dark areas by analyzing a previously obtained image. For example, the electronic device (101) may obtain information about the subject scene through a separate sensor (e.g., an illuminance sensor).
[0127] According to one embodiment, in operation 1020, the electronic device (101) may determine whether information regarding the subject scene satisfies the first condition. For example, the electronic device (101) may determine that the first condition is satisfied if the target gain value determined based on the illuminance is less than the first threshold, or if the number of subjects including the saturated area is less than the third threshold. For example, if the illuminance for the subject scene is brighter than a specified level, the target gain value may be determined to be less than the first threshold. Operation 1020 may be implemented as an operation that determines whether to switch the operation mode. The first threshold may be a single value, but may consist of two or more values. For example, in an operation mode that performs an exposure operation based on a second automatic exposure operation criterion (e.g., operation 1041), the electronic device may switch to an operation mode that performs an exposure operation based on a first automatic exposure operation criterion (e.g., operation 1031) when the target gain value is less than the first value. In an operation mode (e.g., operation 1031) that performs an exposure operation based on a first automatic exposure operation criterion, the electronic device may switch to an operation mode (e.g., operation 1041) that performs an exposure operation based on a second automatic exposure operation criterion when the target gain value is greater than or equal to a second value which is greater than the first value. When the target gain value is greater than or equal to the first value and less than the second value, the electronic device may maintain the previously performed operation mode. Based on the fact that information about the subject scene satisfies the first condition, in operation 1031, the electronic device (101) may perform an exposure operation based on the first automatic exposure operation criterion. For example, the electronic device (101) may control the image sensor (230) to perform an exposure operation during a first exposure interval corresponding to a shutter speed of 1 / 60 s based on case 1 of Table 1.In operation 1033, the electronic device (101) can acquire a first image frame from a signal read out based on a first conversion gain for a first exposure interval. For example, the electronic device (101) can acquire a first image frame based on an analog gain of LCG and 1 as the target gain value is set to 1 based on case 1 of Table 1. In operation 1035, the electronic device (101) can acquire a second image frame from a signal read out based on a second conversion gain for a first exposure interval. For example, the electronic device (101) can acquire a second image frame based on an analog gain of HCG and 1 based on case 1 of Table 1.
[0128] According to one embodiment, in operation 1037, the electronic device (101) can obtain a third image frame by synthesizing the first image frame and the second image frame. In one embodiment, operation 1037 can be performed within an image sensor (230) included in a camera module (180). Operation 1037 may also be performed by an image signal processor (260). In operation 1037, the electronic device (101) can synthesize the first image frame and the second image frame based on a first synthesis method.
[0129] According to one embodiment, in operation 1020, if the information about the subject scene does not satisfy the first condition (or satisfies the second condition), in operation 1041, the electronic device (101) may perform an exposure operation based on the second automatic exposure operation criteria. For example, if the target gain value is greater than or equal to the first threshold, the number of subjects including the saturated area is greater than or equal to the third threshold, and the low-frequency component of the dark area satisfies the specified conditions, the electronic device (101) may determine that the information about the subject scene satisfies the second condition. For example, in operation 1041, the electronic device (101) may control the image sensor (230) to perform an exposure operation during a second exposure interval corresponding to a shutter speed of 1 / 60 s based on case 5 of Table 2. In operation 1043, the electronic device (101) can control the image sensor (230) to acquire a fourth image frame from a signal read out based on a first conversion gain for a second exposure interval. For example, the electronic device (101) can acquire a fourth image frame based on the analog gains of LCG and 1 as the target gain value is set to 4 based on case 5 of Table 2. In operation 1045, the electronic device (101) can control the image sensor (230) to acquire a fifth image frame from a signal read out based on a second conversion gain for a second exposure interval. For example, the electronic device (101) can acquire a fifth image frame based on the analog gains of HCG and 1 as the target gain value is set to 4 based on case 5 of Table 2.
[0130] According to one embodiment, in operation 1047, the electronic device (101) can obtain a sixth image frame by synthesizing the fourth image frame and the fifth image frame. In one embodiment, operation 1047 can be performed within an image sensor (230) included in a camera module (180). Operation 1047 may also be performed by an image signal processor (260). In operation 1047, the electronic device (101) can synthesize the fourth image frame and the fifth image frame based on a second synthesis method different from the first synthesis method.
[0131] According to one embodiment, in operation 1050, the electronic device (101) may provide the image synthesized in operation 1037 or operation 1047 as the captured image.
[0132] The examples of the first and second conditions described above are for the purpose of explaining the embodiments and are not limited thereto. Additionally, FIG. 10 illustrates a process for determining an image capture operation based on whether information about a subject scene satisfies the first condition, but is not limited thereto. For example, an electronic device (101) according to one embodiment may be configured to determine an AE operation criterion based on whether two or more conditions are satisfied.
[0133] FIG. 11 illustrates a structure in which noise is amplified during the process of acquiring image data by an image sensor (230) according to one embodiment.
[0134] Shot noise (1110) may occur in the charge generated from the light receiving element (e.g., the light receiving element (510) of FIG. 5) of the image sensor (230) according to one embodiment. Additionally, a first dark noise (N1) (1120), a second dark noise (N2) (1130), and a third dark noise (N3) (1140) may occur even when there are no signal electrons, and may occur at each step of reading out the signal. Assuming the structure of the image sensor (230) is as in FIG. 11, the shot noise (1110) and the first dark noise (1120) may be amplified by the conversion gain (530, 535). Assuming the structure of the image sensor (230) is as in FIG. 11, the shot noise (1110), the first dark noise (1120), and the second dark noise (1130) may be amplified by the analog gain (581). The digital gain (585) performed after the ADC (583) can amplify the shooting noise (1110), the first arm noise (1120), the second arm noise (1130), and the third arm noise (1140). The total noise included in the digitally converted image can be expressed as shown in Equation 2 below.
[0135]
[0136] In the above mathematical formula 2, CG indicates a conversion gain (530, 535), AG indicates an analog gain (581), and DG indicates a digital gain (585). According to one embodiment of the present disclosure, since an HDR image is generated based on the analog gain (581) or the conversion gain (530, 535), the analog gain (581) prevents amplification of the third arm noise (1140), and the conversion gain (530, 535) prevents amplification of the second arm noise (1130) and the third arm noise (1140).
[0137] FIG. 12 illustrates an example in which an electronic device (101) according to one embodiment synthesizes image frames based on a first synthesis method.
[0138] In one embodiment, the electronic device (101) can acquire a first image frame (1201) and a second image frame (1202) by controlling the image sensor (230) to operate based on a first AE operation reference. The first image frame (1201) may be a reference exposure image. The first image frame (1201) may include first digital data (1210) expressed in N bits (e.g., 10 bits), and the second image frame (1202) may include second digital data (1220) expressed in N bits (e.g., 10 bits). Since the second digital data (1220) is read out with a higher gain applied compared to the first digital data (1210), the second digital data (1220) may have more information about dark areas than the first digital data (1210).
[0139] In one embodiment, the electronic device (101) can obtain a third digital data (1240) and a fourth digital data (1230) represented in N + n bits (e.g., 12 bits) from the first digital data (1210) and the second digital data (1220). For example, the electronic device (101) can obtain the third digital data (1240) and the fourth digital data (1230) through zero padding on the first digital data (1210) and the second digital data (1220). The electronic device (101) can obtain the third digital data (1240) by adding n bits (e.g., 2 bits) of dummy data to the first digital data (1210). The electronic device (101) can obtain a fourth digital data (1230) by adding n bits (e.g., 2 bits) of dummy data to the second digital data (1220).
[0140] In one embodiment, the electronic device (101) may acquire a third image frame (1203) comprising a fifth digital data (1250) represented by N + n bits from the third digital data (1240) and the fourth digital data (1230). The electronic device (101) may construct the fifth digital data (1250) based on a value obtained by blending the third digital data (1240) and the fourth digital data (1230) according to a specified synthesis ratio for a blending region (1255). Bits in a region higher than the blending region (1255) of the fifth digital data (1250) may correspond to bits of the third digital data (1240). Bits in a region lower than the blending region (1255) of the fifth digital data (1250) may correspond to bits of the fourth digital data (1230). According to the first synthesis method, the blending region (1255) may be formed in a region biased toward the dark side or in a low bit (e.g., 64 bits or less). However, the blending region (1255) does not have to be formed in bits 2 and 3 as shown in FIG. 12. For example, the blending region (1255) may be formed in bits 2, 3, and 4.
[0141] Referring to FIG. 12, bits (1251) lower than the blending region (1255) of the fifth digital data (1250) can be configured based on the corresponding bits (1231) of the fourth digital data (1230). Bits (1252) higher than the blending region (1255) of the fifth digital data (1250) can be configured based on the corresponding bits (1242) of the third digital data (1240). The blending region (1255) of the fifth digital data (1250) can be configured by blending the corresponding bits (1234) of the third digital data (1240) and the fourth digital data (1230).
[0142] FIG. 13 illustrates an example in which an electronic device (101) according to one embodiment synthesizes image frames based on a second synthesis method different from a first synthesis method.
[0143] In one embodiment, the electronic device (101) can acquire a fourth image frame (1304) and a fifth image frame (1305) by controlling the image sensor (230) to operate based on a second AE operation reference. The fifth image frame (1305) may be a reference exposure image. The fourth image frame (1304) may include sixth digital data (1310) expressed in N bits (e.g., 10 bits), and the fifth image frame (1305) may include seventh digital data (1320) expressed in N bits (e.g., 10 bits). Since the fourth image frame (1304) is read out with a lower gain applied compared to the fifth image frame (1305), the fourth image frame (1304) may have more information about the saturated area in the fifth image frame (1305).
[0144] In one embodiment, the electronic device (101) can obtain the eighth digital data (1330) and the ninth digital data (1340) represented in N + n bits (e.g., 12 bits) from the sixth digital data (1310) and the seventh digital data (1320). For example, the electronic device (101) can obtain the eighth digital data (1330) and the ninth digital data (1340) through zero padding for the sixth digital data (1310) and the seventh digital data (1320).
[0145] In one embodiment, the electronic device (101) may acquire a sixth image frame (1306) comprising a tenth digital data (1350) represented by N + n bits from eighth digital data (1330) and ninth digital data (1340). The electronic device (101) may construct the tenth digital data (1350) based on a value obtained by blending the eighth digital data (1330) and the ninth digital data (1340) according to a specified synthesis ratio for a blending region (1355). Bits in a region higher than the blending region (1355) of the tenth digital data (1350) may correspond to bits of the eighth digital data (1330). Bits in a region lower than the blending region (1355) of the tenth digital data (1350) may correspond to bits of the ninth digital data (1340). According to the second synthesis method, the blending region (1355) may be formed in a region biased toward the list side or in a high bit (e.g., 512 bits or higher). However, the blending region (1355) does not have to be formed in bits 8 and 9 as shown in FIG. 13. For example, the blending region (1355) may be formed in bits 7, 8, and 9.
[0146] Referring to FIG. 13, bits (1351) lower than the blending region (1355) of the 10th digital data (1350) can be configured based on the corresponding bits (1341) of the 9th digital data (1340). Bits (1352) higher than the blending region (1355) of the 10th digital data (1350) can be configured based on the corresponding bits (1332) of the 8th digital data (1330). The blending region (1355) of the 10th digital data (1350) can be configured by blending the 8th digital data (1330) and the corresponding bits (1334) of the 9th digital data (1340).
[0147] FIG. 14 is a circuit diagram illustrating a pixel circuit in which a floating node of an image sensor (230) according to one embodiment includes three floating node regions (1431, 1432, 1433).
[0148] In one embodiment, the pixel circuit of the image sensor (230) may include three or more floating node regions. Referring to FIG. 14, the pixel circuit of the image sensor (230) may include a first floating node region (1431), a second floating node region (1432), and a third floating node region (1433). The third floating node region (1433) may be connected to the first floating node region (1431) through a first DCG gate (1441). The second floating node region (1432) may be connected to the first floating node region (1431) through a second DCG gate (1442). The image sensor (230) can read an image frame based on three or more conversion gains, including a first conversion gain, a second conversion gain, and a third conversion gain, depending on whether the second floating node region (1432) and the third floating node region (1433) are connected to the first floating node region (1431) according to the on / off state of the first DCG gate (1441) and the second DCG gate (1442), respectively. The ratio between the first conversion gain, the second conversion gain, and the third conversion gain can be configured in various ways. For example, the ratio between the first conversion gain, the second conversion gain, and the third conversion gain may be configured as 1:4:16 or as 1:2:8. The first conversion gain may be referred to as a low conversion gain (LCG), the second conversion gain which is higher than the first conversion gain may be referred to as a medium conversion gain (MCG), and the third conversion gain which is higher than the second conversion gain may be referred to as a high conversion gain.
[0149] In one embodiment, the electronic device (101) may acquire a first image frame, a second image frame, and a third image frame based on any one of a plurality of operation criteria including a first AE operation criterion, a second AE operation criterion, and a third AE operation criterion. For example, the electronic device may read out image frames based on the first AE operation criterion when information about the subject scene satisfies the first criterion. For example, the electronic device may read out image frames based on the second AE operation criterion when information about the subject scene satisfies the second criterion. For example, the electronic device may read out image frames based on the third AE operation criterion when information about the subject scene satisfies the third criterion. Table 3 below is an example of the operation of the image sensor (230) according to the first AE operation criterion. Tables 3 to 5 below assume a case where the ratio between the first conversion gain, the second conversion gain, and the third conversion gain is configured as 1:4:16.
[0150] case 1 case 2 case 3 First Image Second Image Third Image First Image Second Image Third Image First Image Second Image Third Image Target Gain 14 16 Sensor Output 12 bit (or 14 bit) 12 bit (or 14 bit) 12 bit (or 14 bit) Shutter Speed 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s Conversion Gain 14 16 4 16 16 16 16 Analog Gain 1 1 1 1 4 1 4 16 FWC 10 0 0 0 25 0 0 62 5 0 0 62 5 0 15 6 2.5 6 2 5 0 15 6 2.5 3 9 1 Input Bits 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit Synthesis First Synthesis Method First Synthesis Method First Synthesis Method
[0151] In one embodiment, when image frames are acquired based on a first AE operation standard, the first image frame may be a reference exposure image. The electronic device (101) may acquire a 14-bit HDR image by synthesizing a 10-bit first image frame, a second image frame, and a third image frame based on a first synthesis method. The electronic device (101) may also output a 12-bit HDR image by compressing the tone according to the sensor mode. The first synthesis method may be configured to synthesize image frames by forming a blending region at a low bit.
[0152] Table 4 below is an example of the operation of the image sensor (230) according to the second AE operation standard.
[0153] case 4 case 5 case 6 1st Image 2nd Image 3rd Image 1st Image 2nd Image 3rd Image 1st Image 2nd Image 3rd Image Target Gain 14 16 Sensor Output 12 bit (or 14 bit) 12 bit (or 14 bit) 12 bit (or 14 bit) Shutter Speed 1 / 240s 1 / 240s 1 / 240s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s 1 / 60s Conversion Gain 14 16 14 16 4 16 16 Analog Gain 11 11 14 11 4FWC 100000 25000 6250 100000 25000 6250 25000 6250 15 6 2.5 Input Bits 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit Synthesis 2nd Synthesis Method 2nd Synthesis Method 2nd Synthesis Method
[0154] In one embodiment, when image frames are acquired based on a second AE operation standard, the second image frame may become a reference exposure image. The electronic device (101) can acquire a 14-bit HDR image by synthesizing a 10-bit first image frame, a second image frame, and a third image frame based on a second synthesis method. The electronic device (101) may also output a 12-bit HDR image by compressing the tone according to the sensor mode. The second synthesis method may be configured to synthesize image frames by forming a blending area at a higher bit compared to the first synthesis method.
[0155] case 4 case 5 case 6 1st Image 2nd Image 3rd Image 1st Image 2nd Image 3rd Image 1st Image 2nd Image 3rd Image Target Gain 14 16 Sensor Output 12 bit (or 14 bit) 12 bit (or 14 bit) 12 bit (or 14 bit) Shutter Speed 1 / 9 60s 1 / 9 60s 1 / 9 60s 1 / 240s 1 / 240s 1 / 60s 1 / 60s 1 / 60s Conversion Gain 14 16 14 16 14 16 Analog Gain 11 11 11 11 FWC 100000 25000 6250 100000 25000 6250 100000 25000 6250 Input Bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit 10 bit Synthesis 3rd Synthesis Method 3rd Synthesis Method 3rd Synthesis Method
[0156] In one embodiment, when image frames are acquired based on a third AE operation standard, the third image frame may become a reference exposure image. The electronic device (101) may acquire a 14-bit HDR image by synthesizing a 10-bit first image frame, a second image frame, and a third image frame based on a third synthesis method. The electronic device (101) may also output a 12-bit HDR image by compressing the tone according to the sensor mode. The third synthesis method may be configured to synthesize image frames by forming a blending area at a higher bit compared to the second synthesis method.
[0157] Referring to FIG. 14, eight light-receiving elements (1411, 1412, 1413, 1414, 1415, 1416, 1717, 1418) can share a single floating node by being connected to a first floating node region (1431) through transmission gates (1421, 1422, 1423, 1424, 1425, 1426, 1427, 1428). However, this is not limited thereto. For example, as in the pixel circuit shown in FIG. 5, four light-receiving elements (1411, 1412, 1413, 1414) share a first floating node region (1431), and other light-receiving elements (1415, 1416, 1717, 1418) may be connected to other floating nodes.
[0158] FIG. 15 is a drawing illustrating examples of patterns of a light receiving element, a micro lens, and a color filter placed in an image sensor (230) according to one embodiment.
[0159] In one embodiment, the pattern in which the microlens, light-receiving element, and color filter are arranged in the image sensor (230) of the electronic device (101) can be configured in various ways. For example, the image sensor may include a pattern in which either a first sensor pattern (1501) or a second sensor pattern (1503) is repeated. However, the patterns illustrated in FIG. 15 are merely examples for explaining one embodiment and are not limited thereto. In one embodiment, the image sensor may output an image of a resolution according to an operating mode based on the pattern. At least one light-receiving element included in the image sensor may be arranged in an area corresponding to one microlens. In FIG. 15, the area corresponding to each light-receiving element may be referred to as a sensor pixel.
[0160] According to one embodiment, in the first sensor pattern (1501), four light-receiving elements (1511, 1512, 1513, 1514) arranged in a 2 x 2 array may be arranged to correspond to one microlens (1520). The four light-receiving elements (1511, 1512, 1513, 1514) may correspond, for example, to the light-receiving elements (511, 512, 513, 514) of FIG. 5. The microlens (1520) within the first sensor pattern (1501) may be placed in an area corresponding to one color channel (e.g., green channel) of a color filter. Within the first sensor pattern (1501), another microlens placed adjacent to the microlens (1520) may be placed in an area corresponding to the color of the color filter corresponding to the area where the microlens (1520) is placed and the other color channel of the color filter (e.g., red channel, blue channel).
[0161] According to one embodiment, in the second sensor pattern (1503), four microlenses (1591, 1592, 1593, 1594) arranged in a 2 x 2 array may be placed in the same color channel (e.g., green channel) of the color filter. The light receiving elements may include four light receiving elements arranged in a 2 x 2 array and positioned to correspond to one microlens. For example, in the second sensor pattern (1503), four light receiving elements (1571, 1572, 1573, 1574) may be positioned to correspond to the microlens (1591). Four light receiving elements (1575, 1576, 1577, 1578) may be positioned to correspond to the microlens (1592). Four light-receiving elements (1579, 1580, 1581, 1582) can be arranged to correspond to a microlens (1593). Four light-receiving elements (1583, 1584, 1585, 1586) can be arranged to correspond to a microlens (1594). Light-receiving elements arranged in a 2 x 4 arrangement (1571, 1572, 1573, 1574, 1575, 1576, 1577, 1578) can correspond to the light-receiving elements (1411, 1412, 1413, 1414, 1415, 1416, 1417, 1418) shown in FIG. 14.
[0162] In one embodiment, the pattern of the image sensor is not limited to the patterns shown in FIG. 15. For example, the pattern of the image sensor may be replaced with other patterns that can operate by changing the resolution.
[0163] FIG. 16 is a diagram illustrating an example of a color pattern of pixels included in an image obtained through an image sensor (230) according to one embodiment.
[0164] In one embodiment, the image sensor (230) of the electronic device (101) may output an image having a first raw image pattern (1610) corresponding to a Bayer pattern by performing a binning operation that merges the outputs of sensor pixels in a 2 x 2 array (e.g., light receiving elements (1511, 1512, 1513, 1514) of FIG. 15) containing information for the same color channel within a first sensor pattern (1501). In this case, the image having the first raw image pattern (1610) may have a resolution of 1 / 4 relative to the number of sensor pixels of the image sensor (230). For example, from light receiving elements in an 8 x 8 array included in the first sensor pattern (1501) shown in FIG. 15, an image having a first raw image pattern (1610) having a resolution of 4 x 4 may be output.
[0165] In one embodiment, an image sensor (230) of an electronic device (101) may acquire an image having a second raw image pattern (1620) in which each sensor pixel corresponds to the pixels of the image. The electronic device may acquire an image having a Bayer pattern (1625) by performing a remosaic operation to convert the color order on the image having the second raw image pattern (1620). In the present disclosure, the remosaic operation may be performed on the image acquired from the image sensor (230) by at least one processor (e.g., the processor (120) of FIG. 1 and 3, the image signal processor (260) of FIG. 2), but it may be understood that the image sensor (230) may output the result of performing the remosaic operation. However, the remosaic operation to acquire an image having a Bayer pattern (1625) may be omitted. In one embodiment, the electronic device may generate a color image (e.g., an RGB image) from the second raw image pattern (1620). An image obtained based on the second raw image pattern (1620) (e.g., an image having a Bayer pattern (1625)) may have a resolution four times higher than an image obtained based on the first raw image pattern (1610).
[0166] In one embodiment, in an operation mode that outputs a raw image of a first raw image pattern (1610), the image sensor operates based on HCG and LCG, so the electronic device (101) can generate an HDR image based on any one of the AE operation criteria exemplified in Table 1 and Table 2. In an operation mode that outputs a raw image of a second raw image pattern (1620), the image sensor operates based only on HCG, so the electronic device (101) can generate an SDR image (e.g., an image represented in 10 bits) rather than an HDR image.
[0167] FIG. 17 is a diagram illustrating an example of a color pattern of pixels included in an image obtained through an image sensor (230) according to one embodiment.
[0168] In one embodiment, the image sensor (230) of the electronic device (101) may output an image having a first raw image pattern (1710) corresponding to a Bayer pattern by performing a binning operation that merges sensor pixels in a 4 x 4 array containing information for the same color channel within a second sensor pattern (1503). The image having the first raw image pattern (1710) may have a resolution of 1 / 16 relative to the number of sensor pixels of the image sensor.
[0169] In one embodiment, the electronic device (101) can obtain an image having a second raw image pattern (1720) by performing a binning operation that merges the outputs of sensor pixels in a 2 x 2 array. The electronic device (101) can obtain an image having a Bayer pattern (1725) by performing a remosaic operation on the image having the second raw image pattern (1720). The image obtained based on the second raw image pattern (1720) (e.g., an image having a Bayer pattern (1725)) may have a resolution four times higher than the image obtained based on the first raw image pattern (1710).
[0170] In one embodiment, the electronic device (101) can acquire an image having a third raw image pattern (1730) in which each sensor pixel corresponds to the pixels of the image. The electronic device (101) can acquire an image having a Bayer pattern (1735) by performing a remosaic operation on the image having the third raw image pattern (1730). The image acquired based on the third raw image pattern (1730) (e.g., an image having the Bayer pattern (1735)) may have a resolution 16 times higher than the image acquired based on the first raw image pattern (1710).
[0171] In one embodiment, in an operation mode that outputs a raw image of a first raw image pattern (1710), the image sensor operates based on LCG, MCG, and HCG, so the electronic device (101) can generate an HDR image based on any one of the AE operation criteria exemplified in Tables 3 to 5. In an operation mode that outputs a raw image of a second raw image pattern (1720), the image sensor operates based on LCG and MCG, so the electronic device (101) can generate an HDR image based on any one of the AE operation criteria exemplified in Tables 1 and 2 by replacing the HCG of the AE operation criteria exemplified in Tables 1 and 2 with the operation of MCG. In an operation mode that outputs a raw image of a third raw image pattern (1730), the image sensor operates only based on HCG, so the electronic device (101) can generate an SDR image (e.g., an image represented in 10 bits) rather than an HDR image.
[0172] FIG. 18 illustrates an automatic exposure program diagram set in an image sensor (230) to perform an automatic exposure operation of an image sensor (230) according to one embodiment.
[0173] In one embodiment, the electronic device (101) may determine at least one of a sensitivity value (SV) (1801), a time value (TV) (1802), or an aperture value (AV) (1803) according to an exposure value (EV) to perform an automatic exposure operation. However, the program curve shown in FIG. 18 is merely an example, and at least one of the sensitivity value, time value, or aperture value may be set differently from the form shown in FIG. 18. For example, in the case of a camera module with a fixed aperture, the aperture value may be excluded from the program curve or set to a fixed value. The electronic device (101) may determine at least one of the sensitivity value, time value, or aperture value based on a predetermined program curve. The AE program curve set to perform an HDR control operation (e.g., HDR control operation (910) of FIG. 9) of the electronic device (101) according to one embodiment of the present disclosure may be changed. For example, the image sensor (230) may operate based on different AE program lines when operating based on the first AE operation standard and when operating based on the second AE operation standard.
[0174] According to one embodiment of the present disclosure, in the case of an electronic device with dual conversion gains applied, at the part where the photon transfer curve (PTC) is synthesized, the dark noise appearing in the first conversion gain (LCG) and the second conversion gain (HCG) is different, so a change in imaging noise may occur. In a synthesis operation performed based on the first conversion gain, since the dark noise in the second conversion gain is a major factor, the synthesis of image frames may proceed based on low pixel values. In a synthesis operation performed based on the second conversion gain, since digital gain is applied when using the first conversion gain, a transition may occur around a specified value (e.g., 1023).
[0175] A camera module according to one embodiment (e.g., camera module (180) of FIG. 1 and FIG. 2) may include one or more lenses (e.g., lens assembly (210) of FIG. 2), an image sensor (e.g., image sensor (230) of FIG. 2 and FIG. 3), and a computation processing circuit (e.g., computation unit (417) of FIG. 4). The image sensor (e.g., image sensor (230) of FIG. 2 and FIG. 3)) may include a light receiving element (e.g., light receiving unit (415) of FIG. 4), a floating node (e.g., floating node (530) of FIG. 5, floating node (1431) of FIG. 14), and a detection circuit. The light receiving element (e.g., light receiving unit (415) of FIG. 4) may output electrons as at least a portion of the light passing through the one or more lenses (e.g., lens assembly (210) of FIG. 2) is received. The above floating node (e.g., floating node (530) in FIG. 5, floating node (1431) in FIG. 14) may be configured to transmit electrons output from the light receiving element (e.g., light receiving unit (415) in FIG. 4). The detection circuit may be configured to read out a signal from the above floating node (e.g., floating node (530) in FIG. 5, floating node (1431) in FIG. 14). The above computation processing circuit (e.g., computation unit (417) in FIG. 4) may be configured to process a signal read out from the above image sensor (e.g., image sensor (230) in FIG. 2 and FIG. 3). The above floating node (e.g., floating node (530) in FIG. 5, floating node (1431) in FIG. 14)) may include at least two floating node regions such that the charge of the above floating node (e.g., floating node (530) in FIG. 5, floating node (1431) in FIG. 14) is read out based on a first conversion gain or a second conversion gain higher than the first conversion gain.Based on the fact that information about a subject scene captured by the camera module (e.g., camera module (180) of FIGS. 1 and 2) satisfies a first condition, the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) may be configured to perform a first auto exposure (AE) operation based on a first auto exposure operation criterion. Based on the fact that information about a subject scene captured by the camera module (e.g., camera module (180) of FIGS. 1 and 2) satisfies a first condition, the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) may be configured to acquire a first image frame from a signal read out based on the first conversion gain for a first exposure interval corresponding to the first auto exposure operation. Based on the fact that information about the subject scene captured by the camera module (e.g., camera module (180) of FIGS. 1 and 2) satisfies a first condition, the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) may be configured to acquire a second image frame from a signal read out based on the second conversion gain for the first exposure interval. The computation processing circuit (e.g., computation unit (417) of FIG. 4) may be configured to acquire a third image frame, which is a composite of the first image frame and the second image frame, as a captured image. Based on the fact that information about the subject scene captured by the camera module (e.g., camera module (180) of FIGS. 1 and 2) satisfies a second condition different from the first condition, the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) may be configured to perform a second automatic exposure operation based on a second automatic exposure operation different from the first automatic exposure operation.Based on the fact that information about the subject scene captured by the camera module (e.g., camera module (180) of FIGS. 1 and 2) satisfies a second condition different from the first condition, the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) may be configured to acquire a fourth image frame from a signal read out based on the first conversion gain for a second exposure interval corresponding to the second automatic exposure operation. Based on the fact that information about the subject scene captured by the camera module (e.g., camera module (180) of FIGS. 1 and 2) satisfies a second condition different from the first condition, the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) may be configured to acquire a fifth image frame from a signal read out based on the second conversion gain for the second exposure interval. The above-mentioned operation processing circuit (e.g., the operation unit (417) of FIG. 4) may be configured to obtain a sixth image frame, which is a composite of the fourth image frame and the fifth image frame, as a captured image.
[0176] In one embodiment, the second automatic exposure operation criterion may be a setting of the automatic exposure operation such that image frames with relatively darker brightness are obtained for the same target gain value compared to the first automatic exposure operation criterion.
[0177] In one embodiment, the second automatic exposure operation criterion may be set so that an image frame with a brightness darker by an amount corresponding to the difference between the second conversion gain and the first conversion gain for the same target gain value compared to the first automatic exposure operation criterion is obtained.
[0178] In one embodiment, the first exposure interval and the second exposure interval may correspond to the same shutter speed.
[0179] In one embodiment, the camera module (e.g., the camera module (180) of FIGS. 1 and 2) may further include memory. The memory may store a first look-up table corresponding to the first automatic exposure operation criterion and a second look-up table corresponding to the second automatic exposure operation criterion. The first look-up table may include at least one mapping relationship of a shutter speed, sensitivity, or aperture value to a target gain value. The second look-up table may include at least one mapping relationship of a different shutter speed, different sensitivity, or different aperture value to the target gain value. The camera module (e.g., the camera module (180) of FIGS. 1 and 2) may be configured to determine a target gain value based on the brightness of the subject scene. The camera module (e.g., camera module (180) of FIGS. 1 and 2) may be configured to determine at least one of a shutter speed, sensitivity, or aperture value through the first look-up table or the second look-up table based on the determined target gain value. The camera module (e.g., camera module (180) of FIGS. 1 and 2) may be configured to control the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) to perform the first automatic exposure operation or the second automatic exposure operation based on at least one of the determined shutter speed, sensitivity, or aperture value.
[0180] In one embodiment, the information about the subject scene may include at least one of information about a dark area or information about a saturated area. The camera module (e.g., camera module (180) of FIGS. 1 and 2) may be configured to control the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) to perform an automatic exposure operation based on an automatic exposure operation criterion determined based on a category corresponding to at least one of the information about the dark area or the information about the saturated area.
[0181] In one embodiment, the first condition may be that the illuminance of the subject scene falls within a first illuminance range. The second condition may be that the illuminance of the subject scene falls within a second illuminance range indicating an illuminance darker than the first illuminance range. The first condition and the second condition may be determined based on a target gain value to be applied during the process in which the camera module acquires an image frame.
[0182] In one embodiment, the camera module (e.g., the camera module (180) of FIGS. 1 and 2) may be configured to obtain the third image frame by synthesizing the first image frame and the second image frame based on a first synthesis ratio. The camera module (e.g., the camera module (180) of FIGS. 1 and 2) may be configured to obtain the sixth image frame by synthesizing the fourth image frame and the fifth image frame based on a second synthesis ratio different from the first synthesis ratio.
[0183] In one embodiment, the first automatic exposure operation criterion may include a first sensor gain value for the first image frame and a second sensor gain value for the second image frame. The second automatic exposure operation criterion may include a third sensor gain value for the fourth image frame and a fourth sensor gain value for the fifth image frame.
[0184] In one embodiment, the first automatic exposure operation criterion may be setting the exposure operation so that the first image frame becomes the reference exposure image. The second automatic exposure operation criterion may be setting the exposure operation so that the fourth image frame becomes the reference exposure image.
[0185] In one embodiment, the electronic device may include a touchscreen, a camera module (e.g., the camera module (180) of FIGS. 1 and 2), at least one processor, and memory. The touchscreen may be configured to display a screen and receive touch input. The camera module (e.g., the camera module (180) of FIGS. 1 and 2) may include an image sensor (e.g., the image sensor (230) of FIGS. 2 and 3). The memory may store computer-executable instructions. The instructions may be executed individually or collectively by the at least one processor so that the electronic device obtains information about a subject scene being captured through the camera module (e.g., the camera module (180) of FIGS. 1 and 2) based on an image capture request received through the touchscreen. The above instructions may be executed individually or collectively by the at least one processor to cause the electronic device to determine an automatic exposure operation standard based on information about the subject scene. The above instructions may be executed individually or collectively by the at least one processor to cause the electronic device to control the camera module (e.g., camera module (180) of FIGS. 1 and 2) to acquire a first image frame from a signal read based on a first conversion gain for a first exposure interval corresponding to the first automatic exposure operation standard, based on the fact that the determined automatic exposure operation standard is a first automatic exposure operation standard.The above instructions may be executed individually or collectively by the at least one processor to control the camera module (e.g., camera module (180) of FIGS. 1 and 2) to acquire a second image frame from a signal read out based on a second conversion gain higher than the first conversion gain for the first exposure interval, based on the fact that the determined automatic exposure operation standard is the first automatic exposure operation standard. The above instructions may be executed individually or collectively by the at least one processor to enable the electronic device to acquire a third image frame, which is a composite of the first image frame and the second image frame, as a captured image.
[0186] In one embodiment, the memory may store a first look-up table and a second look-up table. The first look-up table may include a mapping relationship of at least one of a shutter speed, sensitivity, or aperture value for a target gain value. The second look-up table may include a mapping relationship of at least one of a different shutter speed, different sensitivity, or different aperture value for the target gain value. The instructions may be executed individually or collectively by the at least one processor to enable the electronic device to determine at least one of information regarding a dark area or information regarding a saturated area for the subject scene. The instructions may be executed individually or collectively by the at least one processor to enable the electronic device to determine the category to which the subject scene belongs based on at least one of information regarding the dark area or information regarding the saturated area. The instructions may be executed individually or collectively by the at least one processor to enable the electronic device to determine one of the first look-up table or the second look-up table based on the category. The above instructions may be executed individually or collectively by the at least one processor to cause the electronic device to determine a target gain value based on the brightness of the subject scene. The above instructions may be executed individually or collectively by the at least one processor to cause the electronic device to control the camera module (e.g., camera module (180) of FIGS. 1 and 2) to acquire the first image frame and the second image frame based on a shutter speed, sensitivity, or aperture value corresponding to the brightness of the subject scene based on the determined look-up table.
[0187] In one embodiment, the instructions may be executed individually or collectively by the at least one processor to cause the electronic device to determine the first automatic exposure operation standard as the automatic exposure operation standard based on the fact that the illuminance of the subject scene falls within a first illuminance range. The instructions may be executed individually or collectively by the at least one processor to cause the electronic device to determine a second automatic exposure operation standard different from the first automatic exposure operation standard as the automatic exposure operation standard based on the fact that the illuminance of the subject scene falls within a second illuminance range. The above instructions may be executed individually or collectively by the at least one processor to control the camera module (e.g., camera module (180) of FIGS. 1 and 2) so that the electronic device acquires a fourth image frame from a signal read based on a first conversion gain for a second exposure interval corresponding to the second automatic exposure operation standard, based on the fact that the determined automatic exposure operation standard is the second automatic exposure operation standard, and acquires a fifth image frame from a signal read based on the second conversion gain for the first exposure interval. The above instructions may be executed individually or collectively by the at least one processor to cause the electronic device to acquire a sixth image frame, which is a composite of the fourth image frame and the fifth image frame, as a captured image.
[0188] In one embodiment, the first automatic exposure operation criterion may be setting the exposure operation so that the first image frame becomes the reference exposure image. The second automatic exposure operation criterion may be setting the exposure operation so that the fourth image frame becomes the reference exposure image.
[0189] In one embodiment, the instructions may be executed individually or collectively by the at least one processor to enable the electronic device to obtain the third image frame by synthesizing the first image frame and the second image frame based on a first synthesis ratio. The instructions may be executed individually or collectively by the at least one processor to enable the electronic device to obtain the sixth image frame by synthesizing the fourth image frame and the fifth image frame based on a second synthesis ratio different from the first synthesis ratio.
[0190] In one embodiment, a camera module (e.g., camera module (180) of FIG. 1 and 2) may include one or more lenses (e.g., lens assembly (210) of FIG. 2), an image sensor (e.g., image sensor (230) of FIG. 2 and 3), and a computation processing circuit (e.g., computation unit (417) of FIG. 4). A light receiving element (e.g., light receiving unit (415) of FIG. 4) may output electrons as at least a portion of the light passing through the one or more lenses (e.g., lens assembly (210) of FIG. 2) is received. A floating node (e.g., floating node (530) of FIG. 5, floating node (1431) of FIG. 14)) may be configured to transmit electrons output from the light receiving element (e.g., light receiving unit (415) of FIG. 4). The detection circuit may be configured to read out a signal from the floating node (e.g., floating node (530) in FIG. 5, floating node (1431) in FIG. 14). The computation processing circuit (e.g., computation unit (417) in FIG. 4) may be configured to process the signal read out from the image sensor (e.g., image sensor (230) in FIG. 2 and FIG. 3). The floating node (e.g., floating node (530) in FIG. 5, floating node (1431) in FIG. 14)) may include at least two floating node regions such that the charge of the floating node (e.g., floating node (530) in FIG. 5, floating node (1431) in FIG. 14) is read out based on a first conversion gain or a second conversion gain higher than the first conversion gain. The image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) may be configured to determine an auto exposure (AE) operation standard including a first combination of an exposure range, a conversion gain, and an output gain, and a second combination of a conversion gain and an output gain, based on information about a subject scene captured by the camera module (e.g., the camera module (180) of FIGS. 1 and 2).The image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) may be configured to perform an exposure operation for the light receiving element (e.g., the light receiving part (415) of FIG. 4) during the exposure period based on the automatic exposure operation criteria. The image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) may be configured to acquire a first image frame by reading out the charge for the exposure period based on the first combination. The image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) may be configured to acquire a second image frame by reading out the charge for the exposure period based on the second combination. The camera module (e.g., the camera module (180) of FIGS. 1 and 2) may be configured to output a third image frame, which is a composite of the first image frame and the second image frame, as a captured image. The first combination and the second combination may be configured such that the brightness of the first image frame is lower than the brightness of the second image frame.
[0191] In one embodiment, the information about the subject scene may include at least one of information about a dark area or information about a saturated area. The camera module (e.g., camera module (180) of FIGS. 1 and 2) may be configured to control the image sensor (e.g., image sensor (230) of FIGS. 2 and 3) to perform an automatic exposure operation based on an exposure operation criterion determined based on a category corresponding to at least one of the information about the dark area or the information about the saturated area.
[0192] In one embodiment, the information regarding the subject scene may include a target gain value corresponding to the brightness of the subject scene.
[0193] In one embodiment, the image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) may be configured to acquire the first image frame and the second image frame based on an automatic exposure criterion that acquires image frames that are relatively darker for the same target gain value based on the target gain value being greater than or equal to a threshold value.
[0194] In one embodiment, the camera module (e.g., the camera module (180) of FIGS. 1 and 2) may further include a memory. The memory may store a plurality of look-up tables that include at least one mapping relationship of a shutter speed, sensitivity, or aperture value to a target gain value. The first combination and the second combination may be determined based on a look-up table among the plurality of look-up tables that corresponds to the automatic exposure operation criteria.
[0195] According to one embodiment of the present disclosure, an image sensor capable of acquiring an HDR image with an expanded dynamic range without ghosting, an electronic device including the image sensor, and a method of operating the same may be provided.
[0196] According to one embodiment of the present disclosure, an image sensor, an electronic device including the image sensor, and a method of operation thereof may be provided, which can obtain an image with an extended desired dynamic range by setting different exposure operations according to the characteristics of the scene when generating an HDR image from an image obtained based on different conversion gains.
[0197] According to one embodiment of the present disclosure, an image sensor capable of selectively applying whether to extend the dynamic range for a dark area or to extend the dynamic range for a saturated area depending on the subject scene, an electronic device including the image sensor, and a method of operating the same may be provided.
[0198] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description of the present disclosure.
[0199] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.
[0200] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims or embodiments described in the specification of this disclosure.
[0201] In the present disclosure, the function or operation performed by an electronic device may be performed by one or more processors executing one or more instructions stored in memory. The function or operation of the electronic device mentioned in the present disclosure may be performed by a single processor executing one or more instructions, or by a combination of multiple processors executing one or more instructions. A processor mentioned in the present disclosure is understood to include a circuit for performing operations or controlling other components of the electronic device. For example, the one or more processors may include at least one of a central processing unit (CPU), a micro-processor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on chip (SoC), an application-specific integrated circuit (ASIC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operation of the electronic device described above.
[0202] In the present disclosure, a program (software module, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other forms of optical storage devices, or a magnetic cassette. Alternatively, it may be stored in a memory composed of some or all of these. The memory may be composed of a single storage medium or a combination of multiple storage media. The one or more instructions may be stored in a single storage medium or distributed across multiple storage media.
[0203] Additionally, the above program may be stored on an attachable storage device that can be accessed via a communication network such as the Internet, Intranet, LAN (local area network), WLAN (wide LAN), or SAN (storage area network), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.
[0204] In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression is selected to suit the situation presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural form, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.
[0205] Additionally, in the present disclosure, terms such as “part,” “module,” etc. may be a hardware component, such as a processor or circuit, and / or a software component executed by a hardware component, such as a processor.
[0206] "Parts" and "modules" may be implemented by a program that is stored on an addressable storage medium and can be executed by a processor. For example, "parts" and "modules" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as by processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.
[0207] The specific embodiments described in this disclosure are merely examples and do not limit the scope of this disclosure in any way. For the sake of brevity, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted.
[0208] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, comprising only b, comprising only c, or comprising a combination of two or more (comprising a and b, comprising b and c, comprising a and c, or comprising all of a, b, and c).”
[0209] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.
[0210] In the present disclosure, the term “if” will be understood, depending on the context, to mean “when, upon,” “in response to a decision,” or “in response to a detection.” Similarly, “when decided to,” or “when [mentioned condition or event] is detected” will be understood, optionally, to mean “when decided,” or “in response to a decision,” “when [mentioned condition or event] is detected,” or “in response to a detection.”
[0211] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit (or processing circuit) may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.
[0212] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.
[0213] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several hardware combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by an app store that distributes applications or a site or server that supplies or distributes various other software.
[0214] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.
Claims
In camera modules, One or more lenses; An image sensor comprising: a light receiving element that outputs electrons as at least a portion of light passing through one or more lenses is received; a floating node to which electrons output from the light receiving element are transmitted; and a detection circuit that reads out a signal from the floating node; and It includes a computational processing circuit that processes a signal read from the image sensor, and The above floating node includes at least two floating node regions such that the charge of the above floating node is read out based on a first conversion gain or a second conversion gain higher than the first conversion gain, and Based on the fact that information about the subject scene captured by the above camera module satisfies the first condition: The image sensor performs a first auto exposure (AE) operation based on a first auto exposure operation standard, acquires a first image frame from a signal read out based on the first conversion gain for a first exposure interval corresponding to the first auto exposure operation, and acquires a second image frame from a signal read out based on the second conversion gain for the first exposure interval. The above-mentioned computational processing circuit is configured to acquire a third image frame, which is a composite of the first image frame and the second image frame, as a captured image. Based on the fact that the information regarding the subject scene captured by the above camera module satisfies the second condition different from the first condition: The image sensor performs a second automatic exposure operation based on a second automatic exposure operation different from the first automatic exposure operation, acquires a fourth image frame from a signal read out based on the first conversion gain for a second exposure interval corresponding to the second automatic exposure operation, and acquires a fifth image frame from a signal read out based on the second conversion gain for the second exposure interval. A camera module configured such that the above-described operation processing circuit acquires a sixth image frame, which is a composite of the fourth image frame and the fifth image frame, as a captured image. In claim 1, A camera module in which the second automatic exposure operation standard sets the automatic exposure operation so that image frames with relatively darker brightness are obtained for the same target gain value compared to the first automatic exposure operation standard. In claim 2, A camera module in which the above second automatic exposure operation standard sets the automatic exposure operation so that an image frame with a brightness darker by an amount corresponding to the difference between the second conversion gain and the first conversion gain for the same target gain value compared to the above first automatic exposure operation standard. In claim 2, A camera module in which the first exposure interval and the second exposure interval correspond to the same shutter speed. In claim 1, The above camera module further includes memory, and The memory stores a first look-up table corresponding to the first automatic exposure operation criterion and a second look-up table corresponding to the second automatic exposure operation criterion, and The first look-up table above includes a mapping relationship of at least one of a shutter speed, sensitivity, or aperture value for a target gain value, and The second look-up table includes at least one mapping relationship among different shutter speeds, different sensitivities, or different aperture values for the target gain value, and The above camera module is: A target gain value is determined based on the brightness of the above subject scene, and Based on the above-determined target gain value, at least one of the shutter speed, sensitivity, or aperture value is determined through the first look-up table or the second look-up table, and A camera module configured to control the image sensor to perform the first automatic exposure operation or the second automatic exposure operation based on at least one of the determined shutter speed, sensitivity, or aperture value. In claim 5, The information regarding the subject scene above includes at least one of information regarding dark areas or information regarding saturated areas, and The above camera module is, A camera module that controls the image sensor to perform an automatic exposure operation based on an automatic exposure operation criterion determined based on a category corresponding to at least one of the information about the dark area or the information about the saturated area. In claim 1, The first condition above is that the illumination of the subject scene is included in the first illumination range, and The second condition above is that the illuminance of the subject scene is included in a second illuminance range indicating an illuminance darker than the first illuminance range, and A camera module in which the first condition and the second condition are determined based on a target gain value to be applied during the process of the camera module acquiring an image frame. In claim 1, The above camera module is: The third image frame is obtained by synthesizing the first image frame and the second image frame based on the first synthesis ratio, and A camera module that obtains the sixth image frame by synthesizing the fourth image frame and the fifth image frame based on a second synthesis ratio different from the first synthesis ratio. In claim 1, The above first automatic exposure operation standard includes a first sensor gain value for the first image frame and a second sensor gain value for the second image frame, and A camera module in which the second automatic exposure operation standard includes a third sensor gain value for the fourth image frame and a fourth sensor gain value for the fifth image frame. In claim 1, The above first automatic exposure operation standard is to set the exposure operation so that the first image frame becomes a reference exposure image, and The above second automatic exposure operation standard is a camera module that sets the exposure operation so that the above fourth image frame becomes a reference exposure image. In electronic devices, A touchscreen that displays a screen and receives touch input; A camera module including an image sensor; At least one processor; and It includes memory that stores instructions executable by a computer, and The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Based on an image capture request received through the touchscreen, information about the subject scene captured through the camera module is obtained, and Based on information regarding the above subject scene, determine the automatic exposure operation criteria, and Based on the fact that the determined automatic exposure operation standard is the first automatic exposure operation standard, the camera module is controlled to acquire a first image frame from a signal read out based on a first conversion gain for a first exposure interval corresponding to the first automatic exposure operation standard, and to acquire a second image frame from a signal read out based on a second conversion gain higher than the first conversion gain for the first exposure interval. An electronic device that obtains a third image frame, formed by synthesizing the first image frame and the second image frame, as a captured image. In claim 11, The above memory stores a first look-up table and a second look-up table, and The first look-up table above includes a mapping relationship of at least one of a shutter speed, sensitivity, or aperture value for a target gain value, and The second look-up table includes at least one mapping relationship among different shutter speeds, different sensitivities, or different aperture values for the target gain value, and The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Determining at least one of information regarding dark areas or saturated areas for the above subject scene, and Determining the category to which the subject scene belongs based on at least one of the information regarding the dark area or the information regarding the saturated area, and Based on the above category, one of the first look-up table or the second look-up table is determined, and A target gain value is determined based on the brightness of the above subject scene, and An electronic device for controlling the camera module to acquire the first image frame and the second image frame based on a shutter speed, sensitivity, or aperture value corresponding to the brightness of the subject scene, based on the above-determined look-up table. In claim 11, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Based on the fact that the illuminance of the above subject scene is included in the first illuminance range, the first automatic exposure operation standard is determined as the automatic exposure operation standard, and Based on the fact that the illumination of the above-mentioned subject scene is included in the second illumination range, a second automatic exposure operation standard different from the first automatic exposure operation standard is determined as the automatic exposure operation standard, and Based on the fact that the determined automatic exposure operation standard is the second automatic exposure operation standard, the camera module is controlled to acquire a fourth image frame from a signal read based on the first conversion gain for a second exposure interval corresponding to the second automatic exposure operation standard, and to acquire a fifth image frame from a signal read based on the second conversion gain for the first exposure interval. An electronic device that obtains a sixth image frame, which is a composite of the fourth image frame and the fifth image frame, as an image captured. In claim 13, The above first automatic exposure operation standard sets the exposure operation so that the first image frame becomes a reference exposure image, and The above second automatic exposure operation standard is an electronic device that sets the exposure operation so that the above fourth image frame becomes a reference exposure image. In claim 13, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: The third image frame is obtained by synthesizing the first image frame and the second image frame based on the first synthesis ratio, and An electronic device that obtains the sixth image frame by synthesizing the fourth image frame and the fifth image frame based on a second synthesis ratio different from the first synthesis ratio.
