Image sensor including sub-color filter, electronic device including same, and operating method thereof
The integration of sub-color filters in image sensors addresses the trade-off between sensitivity and color precision, enhancing image sensor performance by allowing wider wavelength bands without compromising color separation.
Patent Information
- Application Number
- PCT/KR2026/096588
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-08-11
- Filing Date
- 2026-05-06
- Publication Date
- 2026-11-19
AI Technical Summary
Existing image sensors face a trade-off between increasing sensitivity by widening the wavelength band of the color filter and maintaining color precision, leading to decreased color separation characteristics.
Incorporating a sub-color filter in the image sensor to allow for wider color-specific bandpass width while enhancing color precision by using sub-color filters with different spectral responses.
The solution improves color precision while maintaining sensitivity by allowing for wider wavelength bands, optimizing the image sensor's performance.
Smart Images

Figure KR2026096588_19112026_PF_FP_ABST
Abstract
Description
Image sensor including a sub-color filter, electronic device including the same, and method of operation thereof
[0001] The present disclosure relates to an image sensor comprising a sub-color filter, an electronic device comprising the same, and a method of operating the same.
[0002] Various types of electronic devices utilizing electrical and electronic technology or information technology are being developed and distributed. For example, there are portable electronic devices with various functions, such as smartphones, tablet PCs, or wearable devices (e.g., smartwatches, smart rings, smart glasses). Electronic devices may be equipped with cameras that include image sensors.
[0003] An image sensor may include a color filter to distinguish and detect color information. By transmitting the target wavelength band for each pixel through the color filter, an electronic device can acquire an image from different color information through image signal processor (ISP) processing.
[0004] The color of light that can be detected by the pixels of an image sensor is determined by the width of the wavelength band of light transmitted through the color filter and the center wavelength, and the signal that can be detected by the pixels corresponds to only a portion of the input signal. The wider the wavelength band of light transmitted through the color filter, the wider the wavelength band of light can be delivered to the light receiving element, so the sensitivity of the image sensor can be increased.
[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.
[0006] In one embodiment, the electronic device may include an image sensor. The image sensor may include a microlens array, a color filter array, and a sensor pixel array. The color filter array may include a first color filter group corresponding to a first color having a first wavelength band. The sensor pixel array may output a signal detecting light transmitted through the microlens array and the color filter array. The first color filter group may include a first sub-color filter and a second sub-color filter. The first sub-color filter may allow light corresponding to a first sub-spectral response to be transmitted. The second sub-color filter may allow light corresponding to a second sub-spectral response to be transmitted. A first wavelength value corresponding to a first peak value of the first sub-spectral response may be longer than a second wavelength value corresponding to a second peak value of the second sub-spectral response. The first wavelength value and the second wavelength value may be included within the first wavelength band corresponding to the first color.
[0007] In one embodiment, a method of operating an electronic device including an image sensor may include a color filter array. The color filter array may include a first color filter group corresponding to a first color. The method may include an operation of obtaining a first pixel value that detects light transmitted through a first sub-color filter included in the first color filter group. The method may include an operation of obtaining a second pixel value that detects light transmitted through a second sub-color filter included in the first color filter group. The first sub-color filter may allow light corresponding to a first sub-spectral response to be transmitted. The second sub-color filter may allow light corresponding to a second sub-spectral response different from the first sub-spectral response to be transmitted. A first wavelength value corresponding to a first peak value of the first sub-spectral response may be longer than a second wavelength value corresponding to a second peak value of the second sub-spectral response. The first wavelength value and the second wavelength value may be included within the first wavelength band corresponding to the first color.
[0008] In one embodiment, a computer-readable non-transient recording medium may have a computer program recorded thereon that causes an electronic device including an image sensor to perform the method when executed.
[0009] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment of the present disclosure.
[0010] FIG. 2 is a block diagram illustrating a camera module according to one embodiment of the present disclosure.
[0011] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to one embodiment.
[0012] FIG. 4 is a diagram illustrating the configuration of an image sensor according to one embodiment.
[0013] FIG. 5 is a diagram illustrating an example of wavelength bands of a sensor pixel, a microlens, a color filter, and a sub-color filter arranged in an image sensor according to one embodiment.
[0014] FIG. 6 is a drawing illustrating examples of patterns of a color filter array included in an image sensor according to one embodiment.
[0015] FIG. 7 is a graph illustrating an example of a subspectral response for a wavelength band of a subcolor filter according to one embodiment.
[0016] FIG. 8 is a circuit diagram illustrating an example of a circuit for reading the values of pixels included in an image sensor according to one embodiment.
[0017] FIG. 9 is a drawing illustrating examples of patterns of various color filter arrays according to one embodiment.
[0018] FIG. 10 is a flowchart illustrating a process in which an electronic device according to one embodiment acquires an image through pixel values obtained by an image sensor.
[0019] FIG. 11 is a flowchart illustrating a process in which an electronic device according to one embodiment acquires a color image from raw data.
[0020] FIG. 12 illustrates an example of a process in which an electronic device according to one embodiment calculates pixel values obtained from an image sensor.
[0021] FIG. 13 illustrates examples of a process in which an electronic device according to one embodiment calculates pixel values obtained by an image sensor.
[0022] FIG. 14 illustrates an example of a process in which an electronic device according to one embodiment calculates pixel values obtained from an image sensor.
[0023] FIG. 15 illustrates a process in which an electronic device according to one embodiment processes data according to a shooting mode.
[0024] FIG. 16 illustrates a process in which an electronic device according to one embodiment acquires an image through a plurality of cameras.
[0025] FIG. 17 is a diagram illustrating examples of pixels for obtaining phase difference information from an image sensor according to one embodiment.
[0026] A color filter may be placed between the microlens and the sensor pixel to acquire an image by distinguishing color information through an image sensor. In order to increase the sensitivity of the image sensor by adjusting the color filter, if the wavelength band of light passing through the color filter is wide, the color separation characteristics decrease, which may lower the color precision of the image sensor.
[0027] An image sensor including a sub-color filter according to the present disclosure, an electronic device including the same, and a method of operation thereof may be intended to improve the color precision of an image acquired through the image sensor while widening the color-specific bandpass width of the color filter included in the image sensor.
[0028] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.
[0029] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the contents of this disclosure. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.
[0030] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).
[0031] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.
[0032] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.
[0033] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).
[0034] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).
[0035] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).
[0036] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.
[0037] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.
[0038] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).
[0039] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
[0040] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
[0041] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0042] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.
[0043] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.
[0044] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).
[0045] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
[0046] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).
[0047] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.
[0048] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).
[0049] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.
[0050] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.
[0051] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.
[0052] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.
[0053] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.
[0054] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0055] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.
[0056] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.
[0057] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.
[0058] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, a wide-angle lens or a telephoto lens.
[0059] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include one image sensor selected from image sensors with different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.
[0060] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) containing it. This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) may be implemented as, for example, an optical image stabilizer. The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.
[0061] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). The above one or more image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) is at least part of the processor (120). It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after additional image processing by the processor (120).
[0062] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.
[0063] FIG. 3 is a block diagram illustrating the configuration of an electronic device (101) according to one embodiment.
[0064] In one embodiment, the electronic device (101) may include at least one processor (320) (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2), memory (330) (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2), and at least one camera (381, 383) (e.g., camera module (180) of FIG. 1 and 2). The at least one camera (381, 383) may include a first camera (381) including a first image sensor (382). The at least one camera may further include a second camera (383) including a second image sensor (384), but the at least one camera may be composed of a single camera. The at least one camera may include three or more cameras. Memory (330) may store a computer program containing computer-executable instructions that at least one processor executes individually or collectively to cause the electronic device to operate. In the present disclosure, the operation performed by the electronic device (101) may be understood as being performed by at least one processor executing instructions stored in memory to perform operations or by controlling components of the electronic device (101). In one embodiment, at least one processor (320) may include at least one of a central processing unit (CPU), an image signal processor (ISP) (e.g., the image signal processor (260) of FIG. 2), a graphical processing unit (GPU), or a neural processing unit (NPU). For example, at least one processor (320) may include an application processor configured to perform a plurality of functions.At least one processor (320) can acquire image data including information read from at least one of the first image sensor (382) or the second image sensor (384).
[0065] In one embodiment, at least one processor (320) can control the operation of at least one of the first image sensor (382) or the second image sensor (384). The electronic device (101) can acquire image data based on pixel values obtained from pixels placed in a portion of a plurality of sensor pixels included in the first image sensor (382). The portion of the area where the sensor pixels for acquiring image data are placed may be referred to as a region of interest (RoI). At least one processor (320) can control the image sensor to read pixel values from a region including the region of interest for acquiring image data. For example, at least one processor (320) can transmit a control signal related to the area where the image sensor reads pixel values to the first image sensor (382) based on control communication (e.g., I2C (inter-integrated circuit) communication, I3C (improved inter-integrated circuit) communication). The first image sensor (382) can output pixel values read from sensor pixels placed in an area including a region of interest within a pixel array included in the first image sensor (382). For example, the first image sensor (382) can transmit image data through an interface (e.g., MIPI (mobile industry processor interface)) connected to at least one processor (320). For example, the image data may correspond to pixel values output from the image sensor. At least one processor (320) can obtain an enlarged image corresponding to the region of interest based on the pixel values output from the first image sensor (382).
[0066] In one embodiment, the electronic device (101) may further include a display (360) (e.g., the display module (160) of FIG. 1). At least one processor (320) may control the display (360) to display the acquired image. If the display (360) includes a touchscreen, the electronic device (101) may receive touch inputs corresponding to the location of the displayed screen. The at least one camera may further include a second camera (383) including a second image sensor (384). At least one processor (320) may acquire image data through at least one of the first camera (381) or the second camera (383). At least one processor (320) may control the display (360) to display a preview screen based on the image acquired through at least one of the first camera (381) or the second camera (383). In one embodiment, the first camera (381) supports a first field of view (FoV), and the second camera (383) may support a second FoV different from the first FoV. For example, the first camera may include a wide-angle camera, and the second camera may include an ultra-wide-angle camera that supports a FoV including a scene of an area formed at a wider angle than the first camera. However, it is not limited thereto.
[0067] FIG. 4 is a diagram illustrating the configuration of an image sensor according to one embodiment (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384)).
[0068] In one embodiment, the image sensor may include a micro lens array (MLA) (411), a color filter array (CFA) (413), a light receiving unit (415), and a processing unit (417).
[0069] In one embodiment, the micro-lens array (411) may be configured such that a beam of light (421) that passes through a lens unit (e.g., lens assembly (210) of FIG. 2) and forms an image on an image sensor is collected by a light receiving element of the light receiving unit (415). The beams of light (423) that pass through the micro-lens array (411) may pass through a color filter array (413), at least some wavelengths other than the band corresponding to a specific color may be blocked. The beams of light (425) that pass through the color filter array (413) may be detected by a light receiving element (e.g., a photodiode) of the light receiving unit (415). The light receiving unit (415) may include a light receiving element that generates a charge and converts it into an electrical signal upon receiving light, and a circuit that selectively reads out the charge of the light receiving element. Between the light receiving unit (415) and the operation unit (417), a circuit for digitizing the signal read from the light receiving unit (415) or reducing noise may be arranged.
[0070] In one embodiment, the microlens array (411) may be arranged to correspond to at least one light-receiving element. For example, when viewed from the direction in which the light beam (421) is incident, the area in which a single microlens included in the microlens array (411) is arranged may overlap at least partially with the area in which a plurality of light-receiving elements are arranged. The microlenses included in the microlens array (411) may be arranged on a color filter of a different color channel from an adjacent microlens, but a plurality of microlenses corresponding to a color filter of the same color channel may also be arranged adjacent to each other. The arrangement between the microlens array (411), the color filter array (413), and the light-receiving unit (415) may be configured differently depending on the type of image sensor.
[0071] In one embodiment, the operation unit (417) can perform an operation to process electrical data (427) output from the light receiving unit (415). The operation unit (417) can output the acquired data based on the result of the operation. The output of the operation unit (417) can be the output of an image sensor (e.g., the image sensor (230) of FIG. 2), the first image sensor (382) of FIG. 3, and the second image sensor (384). In one embodiment, the operation unit (417) can perform an operation to calibrate the data read from the operation to process the electrical data (427). For example, the operations performed by the operation unit (417) may include at least one of the following: operations to reduce deviation between pixels due to optical asymmetry or the relative position of the sensor, operations to crop some of the read pixel values, operations to reduce noise generated in the analog signal, operations to remove defects, operations to perform remosaic, or operations according to a specific application (e.g., proximity sensor function, timing control function, HDR (high dynamic range) tone mapping function). The sensor output (429) output from the operation unit (417) may be input to at least one processor (e.g., application processor) through an interface.
[0072] In FIG. 4, the color pattern of the color filter array (413) is illustrated based on a Bayer pattern, but the color pattern of the color filter array (413) according to one embodiment is not limited to that illustrated in FIG. 4. Regions within the color filter array (413) that are adjacent to each other may be configured to include a plurality of sub-channels for the same color. For example, an image sensor including the color filter array (413) may include a structure in which the pattern illustrated in FIG. 6 is repeated.
[0073] FIG. 5 is a drawing illustrating examples of wavelength bands of a light receiving element (e.g., a light receiving element included in the light receiving unit (415) of FIG. 4)), a micro lens (e.g., a micro lens included in the micro lens array (411) of FIG. 4), a color filter (e.g., a color filter array (413) of FIG. 4) and a sub-color filter placed in an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384)) according to one embodiment.
[0074] In one embodiment, the color filter array (413) of FIG. 4 may include a first color filter group (501) in a 2x2 array. The first color filter group (501) may allow light corresponding to a first color to be transmitted. The first color may be any one of red (R), green (G), or blue (B). However, the first color may be any other color. For example, the first color may be any one of cyan (C), magenta (M), or yellow (Y). The first color filter group (501) may include a first sub-color filter (511) that transmits light corresponding to a first sub-spectral response (531). For example, the first bandwidth of the light corresponding to the first sub-spectral response (531) may be included in the color bandwidth of the light corresponding to the first color. The first bandwidth may be formed narrower than the first color bandwidth. The first color filter group (501) may include a second sub-color filter (512) that transmits light corresponding to a second sub-spectral response (532) that is different from the first sub-spectral response (531). For example, the second bandwidth of light corresponding to the second sub-spectral response (532) may be included in the color bandwidth of light corresponding to the first color. The second bandwidth may be formed narrower than the first color bandwidth. In addition to a 2x2 array having four sub-color filters including the first sub-color filter (511) and the second sub-color filter (512), the first color filter group (501) may have an array including more than one sub-color filter. In one embodiment, a single microlens may be positioned to correspond to the first color filter group. In one embodiment, the light receiving unit (415) of FIG. 4 may include a first sensor pixel (521) and a second sensor pixel (522). The sensor pixel may include a light receiving element of an image sensor. The first sensor pixel (521) can be positioned to correspond to the first sub-color filter (511).The second sensor pixel (522) may be arranged to correspond to the second sub-color filter (512). However, the arrangement shown in FIG. 5 is merely an example for explaining one embodiment and is not limited thereto. In the present disclosure, the path through which pixels are obtained based on the sub-spectral response of the sub-color filter may be referred to as a sub-color channel.
[0075] In one embodiment, a graph showing the first subspectral response (531) and the second subspectral response (532) may be represented as a graph (561) of the first color filter group. For example, the first subspectral response (531) may have a first peak value (541) at a first wavelength value (551). The first peak value (541) may be the maximum value of the first subspectral response (531), but is not limited thereto. For example, the second subspectral response (532) may have a second peak value (542) at a second wavelength value (552). The second peak value (542) may be the maximum value of the second subspectral response (532), but is not limited thereto. For example, the first wavelength value (551) may be higher than the second wavelength value (552). The first subspectral response (531) and the second subspectral response (532) may each have a bandwidth similar to the spectrum response (575) of a color filter with low sensitivity.
[0076] In one embodiment, the first wavelength value (551) and the second wavelength value (552) may be included within a first wavelength band corresponding to a first color. For example, the first wavelength value (551) and the second wavelength value (552) may be included within a wavelength band of green light. For example, the first wavelength value (551) and the second wavelength value (552) may be included within a wavelength band of red light. For example, the first wavelength value (551) and the second wavelength value (552) may be included within a wavelength band of blue light. In one embodiment, information corresponding to the first spectrum response (570) of a high-sensitivity color filter can be obtained by synthesizing the signal of light detected through the first sub-color filter (511) and the signal of light detected through the second sub-color filter (512) through the first color filter group (501). In order for the signal obtained through the first color filter group (501) to have the characteristics of the first spectrum response (570), the bandwidth of the first subspectral response (531) and the bandwidth of the second subspectral response (532) may be configured to be included within the bandwidth of the first spectrum response (570). For example, the first wavelength value (551) and the second wavelength value (552) may be included within a bandwidth (580) in which the first spectrum response (570) is at least half of its maximum value (max). For example, the value of the first spectrum response (570) at a specific wavelength may correspond to the larger value between the value of the first subspectral response (531) and the value of the second subspectral response (532) at a specific wavelength.
[0077] FIG. 6 is a drawing illustrating examples of patterns of a color filter array (e.g., a color filter array (413) of FIG. 4) included in an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384))) according to one embodiment.
[0078] In one embodiment, the first pixel group (651) is arranged such that four sensor pixels correspond to a single microlens, and the color filters placed on adjacent microlenses may have a pixel structure (hereinafter referred to as a 4PD pixel structure) that includes color filter groups of different colors. In one embodiment, the first pixel group (651) may have a first color filter group (601), a second color filter group, and a third color filter group. For example, the first color filter group (601) may be configured to transmit light corresponding to a first color. For example, the second color filter group may be configured to transmit light corresponding to a second color. For example, the third color filter group may be configured to transmit light corresponding to a third color. For example, the first color may be green (G), the second color may be red (R), and the third color may be blue (B). However, each color is not limited thereto. In an image sensor according to one embodiment, the first pixel group (651) may be formed in a repeated pattern.
[0079] In one embodiment, the first color filter group (601) may include a first sub-color filter (611), a second sub-color filter (612), a third sub-color filter (613), and a fourth sub-color filter (614). For example, the first sub-color filter (611) and the third sub-color filter (613) may be configured to transmit light corresponding to a first sub-spectral response (e.g., the first sub-spectral response (531) of FIG. 5). For example, the second sub-color filter (612) and the fourth sub-color filter (614) may be configured to transmit light corresponding to a second sub-spectral response (e.g., the second sub-spectral response (532) of FIG. 5) that is different from the first sub-spectral response.
[0080] In one embodiment, the first color filter group (601) may include a first sensor pixel (621), a second sensor pixel (622), a third sensor pixel (623), and a fourth sensor pixel (624). For example, the first sensor pixel (621) may be positioned to correspond to a first sub-color filter (611). For example, the second sensor pixel (622) may be positioned to correspond to a second sub-color filter (612). For example, the third sensor pixel (623) may be positioned to correspond to a third sub-color filter (613). For example, the fourth sensor pixel (624) may be positioned to correspond to a fourth sub-color filter (614). In the present disclosure, the phrase "a sensor pixel is positioned to correspond to a sub-color filter" may mean that a light receiving element is positioned below the sub-color filter so that light passing through the sub-color filter is incident on the sensor pixel.
[0081] In one embodiment, the first pixel group (651) may include a first microlens (641). For example, the first microlens (641) may be included in a microlens array (e.g., the microlens array (411) of FIG. 4). The first color filter group (601) corresponds to the first microlens (641) and may be located below the first microlens (641).
[0082] In one embodiment, the second pixel group (652) is arranged so that four sensor pixels correspond to a single microlens, and the color filters placed on adjacent microlenses may have a pixel structure (4PD pixel structure) that includes color filter groups of different colors. In one embodiment, the second pixel group (652) may have a first color filter group, a second color filter group (602), and a third color filter group (603). For example, the first color filter group may allow light corresponding to a first color to be transmitted. For example, the second color filter group (602) may allow light corresponding to a second color to be transmitted. For example, the third color filter group (603) may allow light corresponding to a third color to be transmitted. For example, the first color may be green (G), the second color may be red (R), and the third color may be blue (B). However, each color is not limited thereto. In one embodiment, the image sensor may have a second pixel group (652) formed in a repeated pattern.
[0083] In one embodiment, the second color filter group (602) may include a fifth sub-color filter (615) and a sixth sub-color filter (616). For example, the fifth sub-color filter (615) may allow light corresponding to a third sub-spectral response to be transmitted. For example, the sixth sub-color filter (616) may allow light corresponding to a fourth sub-spectral response different from the third sub-spectral response to be transmitted. In one embodiment, the third color filter group (603) may include a seventh sub-color filter (617) and an eighth sub-color filter (618). For example, the seventh sub-color filter (617) may allow light corresponding to a fifth sub-spectral response to be transmitted. For example, the eighth sub-color filter (618) may allow light corresponding to a sixth sub-spectral response to be transmitted.
[0084] In one embodiment, the second color filter group (602) may include a fifth sensor pixel (625) and a sixth sensor pixel (626). For example, the fifth sensor pixel (625) may be positioned to correspond to a fifth sub-color filter (615). For example, the sixth sensor pixel (626) may be positioned to correspond to a sixth sub-color filter (616). In one embodiment, the third color filter group (603) may include a seventh sensor pixel (627) and an eighth sensor pixel (628). For example, the seventh sensor pixel (627) may be positioned to correspond to a seventh sub-color filter (617). For example, the eighth sensor pixel (628) may be positioned to correspond to an eighth sub-color filter (618).
[0085] In one embodiment, the second pixel group (652) may include a first micro lens (641), a second micro lens (642), and a third micro lens (643). For example, the first micro lens (641), the second micro lens (642), and the third micro lens (643) may be included in a micro lens array (e.g., the micro lens array (411) of FIG. 4). The first color filter group corresponds to the first micro lens (641) and may be located below the first micro lens (641). The second color filter group (602) corresponds to the second micro lens (642) and may be located below the second micro lens (642). The third color filter group (603) corresponds to the third micro lens (643) and may be located below the third micro lens (643).
[0086] In one embodiment, sub-color filters belonging to a color filter group of different colors may be arranged such that sub-color filters with a high wavelength of subspectral response (e.g., the first wavelength value (551) in FIG. 5) within the color filter group are adjacent to each other, and sub-color filters with a low wavelength of subspectral response (e.g., the second wavelength value (552) in FIG. 5) are adjacent to each other.
[0087] In one embodiment, the second color filter group (602) may be positioned adjacent to the first color filter group (601), and the third color filter group (603) may be positioned adjacent to the first color filter group (601). For example, when the second color filter group (602) is positioned adjacent to the first color filter group (601), the fifth sub-color filter (615) and the first sub-color filter (611) may be positioned adjacent to each other, and the sixth sub-color filter (616) and the second sub-color filter (612) may be positioned adjacent to each other. For example, when the third color filter group (603) is positioned adjacent to the first color filter group (601), the seventh sub-color filter (617) and the first sub-color filter (611) may be positioned adjacent to each other, and the eighth sub-color filter (618) and the fourth sub-color filter (614) may be positioned adjacent to each other. However, the structure in which sub-color filters of different color filter groups are arranged adjacently is not limited to this. For example, sub-color filters of the first color filter group (601) can be arranged as in the first pixel group (651).
[0088] FIG. 7 is a graph illustrating an example of a subspectral response for a wavelength band having a subcolor filter according to one embodiment (e.g., a subcolor filter including the first pixel group (651) of FIG. 6, a subcolor filter including the second pixel group (652)).
[0089] In one embodiment, the first subspectral response (711) may have a first peak value (721) at a first wavelength value (731), and the second subspectral response (712) may have a second peak value (722) at a second wavelength value (732). For example, the first wavelength value (731) may be higher than the second wavelength value (732). In one embodiment, the third subspectral response (713) may have a third peak value (723) at a third wavelength value (733), and the fourth subspectral response (714) may have a fourth peak value (724) at a fourth wavelength value (734). For example, the third wavelength value (733) may be higher than the fourth wavelength value (734). In one embodiment, the fifth subspectral response (715) may have a fifth peak value (725) at a fifth wavelength value (735), and the sixth subspectral response (716) may have a sixth peak value (726) at a sixth wavelength value (736). For example, the fifth wavelength value (735) may be higher than the sixth wavelength value (736).
[0090] In one embodiment, the width between the wavelengths where the subspectral responses of subcolor filters belonging to the same color filter group have peak values may be smaller than the width between the wavelengths where the subspectral responses of subcolor filters belonging to a different color filter group have peak values. The difference (703) between the third wavelength value (733) and the fourth wavelength value (734) may be smaller than the difference (705) between the first wavelength value (731) and the fourth wavelength value (734). For example, the greater the difference (705) between the first wavelength value (731) and the fourth wavelength value (734) than the difference (703) between the third wavelength value (733) and the fourth wavelength value (734), the clearer the distinction between the first color and the second color can be, thereby increasing color reproducibility. In one embodiment, the difference (701) between the fifth wavelength value (735) and the sixth wavelength value (736) may be smaller than the difference (704) between the second wavelength value (732) and the fifth wavelength value (735). In one embodiment, the difference (702) between the first wavelength value (731) and the second wavelength value (732) may be the smallest among the differences between the first wavelength value (731) and the second to sixth wavelength values (732, 733, 734, 735, 736). For example, by configuring the sub-color channels corresponding to the sub-color filters so that the difference (702) between the first wavelength value (731) and the second wavelength value (732) is the smallest, the distinction between the first color and other colors can be clearly made.
[0091] In one embodiment, the difference in peak values between different sub-color filters included within the same color filter group may be smaller than the difference in peak values between different color filter groups. For example, the difference between the third peak value (723) and the fourth peak value (724) may be smaller than the difference between the first peak value (721) and the fourth peak value (724). For example, the difference between the first peak value (721) and the second peak value (722) may be smaller than the difference between the first peak value (721) and the fourth peak value (724), and the difference between the second peak value (722) and the fifth peak value (725). For example, the difference between the fifth peak value (725) and the sixth peak value (726) may be smaller than the difference between the second peak value (722) and the fifth peak value (725). The greater the difference in peak values of sub-color filters between different color filter groups, the clearer the distinction between each color can be. For example, if the area of the subspectral response distribution is the same, the greater the difference in peak values, the clearer the distinction between colors can be.
[0092] In one embodiment, sub-color filters belonging to color filter groups of different colors may be arranged such that sub-color filters with a high wavelength of subspectral response (e.g., the first wavelength value (551) in FIG. 5) within the color filter group are adjacent to each other, and sub-color filters with a low wavelength of subspectral response (e.g., the second wavelength value (552) in FIG. 5) are adjacent to each other. Referring to FIG. 7, when a sub-color filter having a second subspectral response (712) among the sub-color filters of the green color filter group (e.g., the second sub-color filter (612) in FIG. 6) and a sub-color filter having a fourth subspectral response (714) among the sub-color filters of the red color filter group (e.g., the sixth sub-color filter (616) in FIG. 6) are arranged adjacently, the overlapping area of the two subspectral responses (712, 714) is wide, so there is a high possibility of interference occurring. Accordingly, a color filter array according to one embodiment may be arranged such that, in an area where groups of color filters of different colors are adjacent, sub-color filters having a first sub-spectral response (711), a third sub-spectral response (713), or a fifth sub-spectral response (715) are adjacent to each other, and sub-color filters having a second sub-spectral response (712), a fourth sub-spectral response (714), or a sixth sub-spectral response (716) are adjacent to each other.
[0093] FIG. 8 is a circuit diagram illustrating an example of a circuit for reading the values of pixels included in an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384)) according to one embodiment.
[0094] In one embodiment, the pixel circuit constituting the sensor pixels included in the image sensor may include a plurality of at least one photodiode (811, 812, 813, 814). For example, four photodiodes (811, 812, 813, 814) having a 2 x 2 array arranged to correspond to one microlens may correspond to sensor pixels (621, 622, 623, 624) corresponding to the first color filter group (601) of FIG. 6. For example, four photodiodes (811, 812, 813, 814) having a 2 x 2 array arranged to correspond to one microlens may correspond to sensor pixels (including 625, 626) corresponding to the second color filter group (602) of FIG. 6. For example, four photodiodes (811, 812, 813, 814) having a 2 x 2 array arranged to correspond to a single microlens may correspond to sensor pixels (including 627, 628) included in the third color filter group (603) of FIG. 6. However, FIG. 8 is merely an example of a pixel circuit for explaining one embodiment, and the configuration of the pixel circuit is not limited to that shown in FIG. 8. In one embodiment, five or more photodiodes may be arranged to correspond to a single microlens. For example, nine photodiodes having a 3 x 3 array may be arranged to correspond to a single microlens.
[0095] Referring to FIG. 8, four photodiodes (e.g., a first photodiode (811), a second photodiode (812), a third photodiode (813), and a fourth photodiode (814)) arranged to correspond to a single microlens may be connected to a floating diffusion node (830). Switches (821, 822, 823, 824) may be connected between each of the photodiodes (811, 812, 813, 814) and the floating node (830). For example, the switches (821, 822, 823, 824) may include transistors (e.g., transfer gates).
[0096] According to one embodiment, charge can be accumulated in the photodiodes (811, 812, 813, 814) during the exposure time. While charge is being accumulated, the switch can be kept off so that the photodiode and the floating node (830) are not connected. When the switch is turned on, the photodiode is connected to the floating node (830), and the accumulated charge can move to the floating node (830). For example, when the first switch (821) is turned on, the charge accumulated in the first photodiode (811) can move to the floating node (830). The charge stored in the floating node (830) can be read out through the source follower (SF) (850) and output as an electrical signal. An image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (382), second image sensor (384) of FIG. 3)) can acquire analog data corresponding to the charge transferred to the floating node (830). For example, the analog data may include information about the amount of charge accumulated in at least one photodiode during the exposure time.
[0097] In one embodiment, the line selector (SEL) (860) may be controlled to an on or off state to output analog data for the selected line. The line selector (860) may be controlled to an on state to read out the value of a sensor pixel connected to a floating node (830) according to the read order.
[0098] In one embodiment, the image sensor can acquire analog data corresponding to the pixel value of the raw image data. For example, the image sensor can acquire analog data corresponding to light intensity data acquired through at least one of the photodiodes (811, 812, 813, 814) by controlling switches (821, 822, 823, 824). When the image sensor turns on the first switch (821), the image sensor can acquire analog data based on the light intensity data acquired through the first photodiode (811). The image sensor can acquire pixel values by converting the acquired analog data into digital data through an analog-digital converter (ADC).
[0099] In one embodiment, after acquiring analog data, the image sensor may perform a reset operation to remove the charge accumulated in the floating node (830) by turning on a reset switch (e.g., a reset transistor (RST) (870). The image sensor may read out the output of each of the photodiodes (811, 812, 813, 814) as a single pixel value by sequentially turning on the switches (821, 822, 823, 824) one by one and performing the reset operation with a time delay. After performing the reset operation by turning on the reset transistor (870), the image sensor may turn on the first switch (821) to transfer the charge accumulated by the first photodiode (811) to the floating node (830). When the charge output by the first photodiode (811) is accumulated in the floating node (830), and the line selector (860) is in the ON state, the image The sensor can read out a voltage value corresponding to the charge accumulated by the first photodiode (811) through the source follower (850). The image sensor can perform a reset operation to remove the charge accumulated in the floating node (830) by turning on the reset transistor (870). Afterwards, the image sensor can turn on the second switch (822) to transfer the charge accumulated by the second photodiode (812) to the floating node (830). When the charge output by the second photodiode (812) is accumulated in the floating node (830) and the line selector (860) is in the ON state, the image sensor can read out a voltage value corresponding to the charge accumulated by the second photodiode (812) through the source follower (850). Afterwards, the image sensor can perform a reset operation to remove the charge accumulated in the floating node by turning on the reset transistor (870). Afterwards, the image sensor the third By turning on the switch (823), the charge accumulated by the third photodiode (813) can be transferred to the floating node (830).When the charge output by the third photodiode (813) is accumulated in the floating node (830) and the line selector (860) is turned on, the image sensor can read out a voltage value corresponding to the charge accumulated by the third photodiode (813) through the source follower (850). Afterwards, the image sensor can turn on the reset transistor (870) to perform a reset operation to remove the charge accumulated in the floating node (830). Afterwards, the image sensor can turn on the fourth switch (824) to transfer the charge accumulated by the fourth photodiode (814) to the floating node (830). When the charge output by the fourth photodiode (814) is accumulated in the floating node (830) and the line selector (860) is turned on, the image sensor can read out a voltage value corresponding to the charge accumulated by the fourth photodiode (814) through the source follower (850). Afterward, the image sensor can perform a reset operation by turning on the reset transistor (870) to remove the charge accumulated in the floating node (830). For example, when operating based on a high-pixel mode that generates a high-pixel image, the image sensor can read out one pixel value from each of the photodiodes (811, 812, 813, 814).
[0100] In one embodiment, when the switches (821, 822, 823, 824) are turned on together, the charge accumulated in the photodiodes (811, 812, 813, 814) can be moved to the floating node (830). In this case, by analogously summing the light amounts corresponding to the four photodiodes (811, 812, 813, 814), the image sensor can acquire light amount data corresponding to one microlens. For example, when operating based on a low-pixel mode that generates an image of lower pixel count than a high-pixel mode, the image sensor can read out the sum of the charges accumulated in the four photodiodes as a single pixel value.
[0101] In one embodiment, when the first switch (821) and the third switch (823) are turned on while the second switch (822) and the fourth switch (824) are off, the image sensor can obtain pixel values for pixels placed on the first side among the pixels corresponding to the micro-lens (e.g., the first photodiode (811), the third photodiode (813) of FIG. 8) (hereinafter referred to as 'left pixels'). When the second switch (822) and the fourth switch (824) are turned on while the first switch (821) and the third switch (823) are off, the image sensor can obtain pixel values for pixels placed on the second side, which is in a direction different from the first side among the pixels corresponding to the micro-lens (e.g., the second photodiode (812), the fourth photodiode (814) of FIG. 8) (hereinafter referred to as 'right pixels'). An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can obtain phase difference information based on the correlation between the left pixel value and the right pixel value. However, this is for the purpose of explaining an example, and the method of obtaining the left pixel value and the right pixel value is not limited thereto.
[0102] In one embodiment, if the output image data is acquired based on a pattern different from the Bayer pattern, the electronic device can perform a remosaic operation to convert the output image data into a Bayer pattern that can be processed by an image signal processor.
[0103] The pixel circuit illustrated in FIG. 8 is an example illustrating one embodiment, and the pixel circuit included in the image sensor according to one embodiment may include a circuit different from the circuit illustrated in FIG. 8. For example, the pixel circuit illustrated in FIG. 8 may be configured such that four sensor pixels corresponding to four photodiodes (811, 812, 813, 814) share one floating node (830), or eight sensor pixels share one floating node.
[0104] FIG. 9 is a drawing illustrating examples of patterns of various color filter arrays (e.g., the color filter array (413) of FIG. 4) according to one embodiment.
[0105] In one embodiment, the first structure (910) may have eight sub-color filters per color filter group. One color filter group may include four sub-color filters having a first spectral response and four sub-color filters having a second spectral response. Four microlenses arranged in a 2x2 array may be placed on one color filter group. Two sub-color filters of different sub-color channels may be placed to correspond to a single microlens. Sensor pixels may be placed to correspond to each sub-color filter.
[0106] In one embodiment, the second structure (920) may have 16 sub-color filters per color filter group. One color filter group may include eight sub-color filters having a first spectral response and eight sub-color filters having a second spectral response. Within a color filter group, different types of sub-color filters may be arranged adjacent to each other. A single microlens may be arranged to correspond to four sub-color filters. Sensor pixels may be arranged to correspond to each sub-color filter.
[0107] In one embodiment, the third structure (930) may have 16 sub-color filters per color filter group. A color filter group may include four sub-color filters (++) having a first spectral response, four sub-color filters (+-) having a second spectral response, four sub-color filters (-+) having a third spectral response, and four sub-color filters having a fourth spectral response (--). The first to fourth spectral responses may be different from each other. Within a color filter group, different types of sub-color filters may be arranged adjacent to each other. For example, if there are four lines (931, 932, 933, 934) within the same color filter group, the first line (931) and the third line (933) may be arranged in order from a sub-color filter with a relatively large wavelength of sub-spectral response to a sub-color filter with a relatively small wavelength of sub-spectral response, and the second line (932) and the fourth line (934) may be arranged in order from a sub-color filter with a relatively small wavelength of sub-spectral response to a sub-color filter with a relatively large wavelength of sub-spectral response. However, this is merely an example of how sub-color filters can be arranged in the third structure (930) and is not limited thereto. A single microlens may be arranged to correspond to four sub-color filters. Sensor pixels may be arranged to correspond to each sub-color filter.
[0108] FIG. 10 is a flowchart (1000) illustrating a process of acquiring an image through pixel values obtained by an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, the second image sensor (384)) according to one embodiment of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3).
[0109] In the present disclosure, the operation of an electronic device (e.g., the electronic device (101) of FIG. 1) may be understood as being performed by a processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2)) performing operations by executing instructions stored in memory (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2) or by controlling components of the electronic device.
[0110] According to one embodiment, in operation 1010, the electronic device may obtain a first pixel value that detects light transmitted through a first sub-color filter (e.g., the first sub-color filter (511) of FIG. 5). For example, the electronic device may control an image sensor to read out the first pixel value by outputting a signal detected through a first sensor pixel (e.g., the first sensor pixel (521) of FIG. 5) placed under the first sub-color filter.
[0111] According to one embodiment, in operation 1020, the electronic device can obtain a second pixel value that detects light transmitted through a second sub-color filter (the second sub-color filter (512) of FIG. 5). For example, the electronic device can control an image sensor to read out the second pixel value by outputting a signal detected through a second sensor pixel (e.g., the second sensor pixel (522) of FIG. 5) placed under the second sub-color filter.
[0112] According to one embodiment, in operation 1030, the electronic device may acquire an image based on a first pixel value acquired in operation 1010 and a second pixel value acquired in operation 1020. For example, the electronic device may acquire an image including a pixel having a value for a first color acquired based on a first pixel value for light transmitted through a first sub-color filter (e.g., the first sub-color filter (511) of FIG. 5) of a color filter group for a first color (e.g., the first color filter group (501) of FIG. 5) and a second pixel value for light transmitted through a second sub-color filter (e.g., the second sub-color filter (512) of FIG. 5) of a color filter group for a first color (e.g., the first color filter group (501) of FIG. 5). In operation 1030, the image sensor acquires image data including the first pixel value and the second pixel value, and the electronic device may acquire a color image by performing image signal processing on the acquired image data. The process of performing image signal processing can be configured in various ways as illustrated in FIGS. 11 to 16.
[0113] FIG. 11 is a flowchart (1100) illustrating an image signal processing process in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment acquires a color image from raw data.
[0114] According to one embodiment, in operation 1110, the electronic device may acquire raw data output from an image sensor. For example, the electronic device may acquire pixel values, which are data obtained based on light received by a micro-lens array (e.g., micro-lens array (411) of FIG. 4), a color filter array (e.g., color filter array (413) of FIG. 4), and a light receiving unit (e.g., light receiving unit (415) of FIG. 4) of an image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (382) of FIG. 3, second image sensor (384)) and data before image processing. For example, the pixel values may include a first pixel value and a second pixel value of FIG. 10.
[0115] According to one embodiment, in operation 1120, the electronic device may obtain data in which a black level adjust is applied to the data obtained in operation 1110. For example, the electronic device may modify a specific pixel value using an electrical signal generated in the absence of light. For example, operation 1120 may be applied first in an image signal processing pipeline. The expressiveness of dark areas within an image may be improved through black level adjustment.
[0116] According to one embodiment, in operation 1130, the electronic device can obtain data in which white balance is applied to the data obtained in operation 1120. For example, the electronic device can correct colors by adjusting the gain of each color channel for colors that are distorted according to the color temperature of the light. Through white balance, the color accuracy of the image can be improved by removing the influence of the lighting color.
[0117] According to one embodiment, in operation 1140, the electronic device may obtain data in which a color correction matrix (CCM) is applied to the data obtained in operation 1130. For example, the electronic device may calculate different weights for each color pixel value to correct the previously obtained data to match the actual visible color data. For example, the matrix used for the CCM may be 3x3 or 3x6. However, the matrix used is not limited thereto.
[0118] According to one embodiment, in operation 1150, the electronic device can acquire a color image through the data obtained in operation 1140. For example, the electronic device may additionally perform other correction operations before acquiring the color image.
[0119] According to one embodiment, the electronic device can acquire a color image by executing operations 1110, 1120, 1130, 1140, and 1150 in sequence. However, the order of operations included in the flowchart (1100) is not limited thereto, and at least some of the operations may be processed in parallel, and image signal processing may be performed by changing the order between the operations.
[0120] FIG. 12 illustrates an example of a process in which an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) calculates pixel values obtained by an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384)).
[0121] In one embodiment, when the electronic device executes a first mode, which is a high-sensitivity mode, it may obtain low-resolution image data (1230) having a Bayer pattern by summing the data (1210) obtained through sub-color channels included in a pixel group through image signal processing. The electronic device may obtain a low-resolution color image (1240) from the low-resolution image data (1230). Referring to FIG. 12, the number of image pixels of the color image (1240) may be 1 / 4 of the number of sensor pixels constituting the image sensor. For example, the first mode may be executed when operating in a low-resolution mode that outputs raw image data of a relatively low resolution. In one embodiment, the image sensor may sum the data obtained from sensor pixels belonging to a single color filter group and output the low-resolution image data (1230) as raw data to a processor (e.g., the image signal processor (260) of FIG. 2). For example, the image sensor may include a processor configured to generate low-resolution image data (1230) by summing the data (1210). In one embodiment, the image sensor may output the data (1210) as raw data to a processor (e.g., the image signal processor (260) of FIG. 2). The processor receiving the raw data may obtain low-resolution image data (1230) by binning the data corresponding to a single color filter group within the data (1210).
[0122] According to one embodiment, a pixel group may have a first color filter group (1211), a second color filter group (1212), and a third color filter group (1223). For example, the first color filter group (1211) may allow light corresponding to a first color to be transmitted. For example, the second color filter group (1212) may allow light corresponding to a second color to be transmitted. For example, the third color filter group (1213) may allow light corresponding to a third color to be transmitted. For example, the first color may be green (G), the second color may be red (R), and the third color may be blue (B). However, each color is not limited thereto.
[0123] According to one embodiment, the first color filter group (1211) may include a first sub-color filter (1221) and a second sub-color filter (1222). For example, the first sub-color filter (1221) may allow light corresponding to a first sub-spectral response to be transmitted. For example, the second sub-color filter (1222) may allow light corresponding to a second sub-spectral response different from the first sub-spectral response to be transmitted. In one embodiment, the second color filter group (1212) may include a third sub-color filter (1223) and a fourth sub-color filter (1224). For example, the third sub-color filter (1223) may allow light corresponding to a third sub-spectral response to be transmitted. For example, the fourth sub-color filter (1224) may allow light corresponding to a fourth sub-spectral response different from the third sub-spectral response to be transmitted. In one embodiment, the third color filter group (1213) may include a fifth sub-color filter (1225) and a sixth sub-color filter (1226). For example, the fifth sub-color filter (1225) may allow light corresponding to a fifth sub-spectral response to be transmitted. For example, the sixth sub-color filter (1226) may allow light corresponding to a sixth sub-spectral response different from the fifth sub-spectral response to be transmitted.
[0124] According to one embodiment, summing data (1230) can be obtained by summing the pixel values of light that has passed through different sub-color filters within the same color filter group. For example, a first pixel value (1231) can be obtained by summing the pixel value of light detected after passing through the first sub-color filter (1221) and the pixel value of light detected after passing through the second sub-color filter (1222). For example, a second pixel value (1232) can be obtained by summing the pixel value of light detected after passing through the third sub-color filter (1223) and the pixel value of light detected after passing through the fourth sub-color filter (1224). For example, a third pixel value (1233) can be obtained by summing the pixel value of light detected after passing through the fifth sub-color filter (1225) and the pixel value of light detected after passing through the sixth sub-color filter (1226). However, the pixel values subject to the summing operation are not limited thereto. For example, the summing operation can be processed based on the following mathematical formula 1.
[0125]
[0126]
[0127]
[0128] FIG. 13 illustrates examples of a process in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment calculates pixel values obtained by an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384)).
[0129] According to one embodiment, an electronic device can obtain a pixel value that detects light passing through a pixel group (1310) having different types of sub-color filters for each color filter group. For example, the pixel group (1310) may be composed of any one of the patterns of the first pixel group (651) and the second pixel group (652) of FIG. 6, the first structure (910) or the second structure (920) of FIG. 9. However, the pattern of the pixel group (1310) is not limited to the examples described above.
[0130] According to one embodiment, in operation 1320, the electronic device may obtain data in which black level adjustment is applied to pixel values that detect light passing through the pixel group (1310). For example, the electronic device may modify specific pixel values using electrical signals generated in the absence of light. For example, operation 1320 may be applied first in an image signal processing pipeline. The expressiveness of dark areas within the image may be improved through black level adjustment.
[0131] According to one embodiment, the electronic device may perform different image signal processing on the data of a sub-color filter that passes light in a high wavelength band in each color filter group (hereinafter, TF+ data) (1330) and the data of a sub-color filter that passes light in a low wavelength band (hereinafter, TF- data (1335)) among the data obtained in operation 1320. For example, the electronic device may perform a first white balance operation (1340) on the TF+ data (1330) and apply a first color correction matrix (+CCM) (1350). For the TF- data (1335), the electronic device may perform a second white balance operation (1345) that applies a gain value different from the first white balance operation (1340) and apply a second color correction matrix (-CCM) (1355) different from the first color correction matrix (1350). For example, the TF+ data (1330) is the first sub-color It may include values for light that has passed through the filter (1331), the second sub-color filter (1332), and the third sub-color filter (1333). TF-data (1335) may include values for light that has passed through the fourth sub-color filter (1336), the fifth sub-color filter (1337), and the sixth sub-color filter (1338).
[0132] According to one embodiment, the first color correction matrix (1350) can be applied as shown in Equation 2 below.
[0133]
[0134] According to one embodiment, the second color correction matrix (1355) can be applied as shown in Equation 3 below.
[0135]
[0136] The first color correction matrix (1350) and the second color correction matrix (1355) may be 3x3. However, the matrices used are not limited thereto. For example, when an electronic device uses a high-pixel mode, the electronic device may apply a color correction matrix of size 3x6 to the acquired data. In one embodiment, the first color correction matrix (1350) and the second color correction matrix (1355) may have the same size, but color correction matrices of different sizes may also be applied.
[0137] According to one embodiment, an electronic device can obtain first RGB channel data (1360) converted into RGB channels by applying a first color correction matrix (1350) to TF+ data (1330). According to one embodiment, an electronic device can obtain second RGB channel data (1365) converted into RGB channels by applying a second color correction matrix (1355) to TF- data (1335). According to one embodiment, an electronic device can obtain an RGB channel image (1370) containing the first RGB channel data (1360) and the second RGB channel data (1365).
[0138] According to one embodiment, in operation 1380, the electronic device may perform binning on an RGB channel image (1370). For example, the electronic device may modify the data of the sub-color filter into a single channel corresponding to each color by performing CCM correction on each. The electronic device may perform operation 1380 when using a second mode, which is a pseudo-conventional mode. The electronic device may obtain data of the second mode (1390) by performing binning on the data obtained in operation 1370. The second mode may be executed when operating in a low-resolution mode that outputs raw image data of a relatively low resolution. For example, the electronic device may operate in the second mode when ambient light is low or depending on the user's resolution selection. For example, the second mode can be processed based on a method of summing the first RGB channel data (1360) obtained by applying the first color correction matrix (1350) to the TF+ data (1330) and the second RGB channel data (1365) obtained by applying the second color correction matrix (1355) to the TF- data (1335) for each color channel.
[0139] According to one embodiment, in operation 1385, the electronic device may perform remosaic on the RGB channel image (1370). For example, the electronic device may perform CCM correction on the data of each sub-color filter so that each data constitutes a single pixel within the image. The electronic device may perform operation 1385 when using a third mode, which is a high-resolution mode. The electronic device may obtain data (1395) of the third mode by performing remosaic on the RGB channel image (1370). For example, the third mode may be executed when operating in a high-resolution mode that outputs raw image data of a relatively high resolution.
[0140] FIG. 14 is a flowchart illustrating an example of a process in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment calculates pixel values obtained by an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384)).
[0141] In one embodiment, when the electronic device executes a fourth mode, which is a multi-spectral sensor mode, it may use data (1430) obtained from each sub-channel containing a sub-color filter based on data (1410) obtained from a pixel group. In one embodiment, the fourth mode may use a 3x6 color correction matrix.
[0142] According to one embodiment, a pixel group may have a first color filter group (1411), a second color filter group (1412), and a third color filter group (1413). For example, the first color filter group (1411) may allow light corresponding to a first color to be transmitted. For example, the second color filter group (1412) may allow light corresponding to a second color to be transmitted. For example, the third color filter group (1413) may allow light corresponding to a third color to be transmitted. For example, the first color may be green (G), the second color may be red (R), and the third color may be blue (B). However, each color is not limited thereto.
[0143] According to one embodiment, the first color filter group (1411) may include a first sub-color filter (1421) and a second sub-color filter (1422). For example, the first sub-color filter (1421) may allow light corresponding to a first sub-spectral response to be transmitted. For example, the second sub-color filter (1422) may allow light corresponding to a second sub-spectral response different from the first sub-spectral response to be transmitted. In one embodiment, the second color filter group (1412) may include a third sub-color filter (1423) and a fourth sub-color filter (1424). For example, the third sub-color filter (1423) may allow light corresponding to a third sub-spectral response to be transmitted. For example, the fourth sub-color filter (1424) may allow light corresponding to a fourth sub-spectral response different from the third sub-spectral response to be transmitted. In one embodiment, the third color filter group (1413) may include a fifth sub-color filter (1425) and a sixth sub-color filter (1426). For example, the fifth sub-color filter (1425) may allow light corresponding to a fifth sub-spectral response to be transmitted. For example, the sixth sub-color filter (1426) may allow light corresponding to a sixth sub-spectral response different from the fifth sub-spectral response to be transmitted.
[0144] According to one embodiment, the fourth mode may use all sub-channels as individual channels. For example, image signal processing may be performed using all pixel values of light detected after passing through the first sub-color filter (1421), light detected after passing through the second sub-color filter (1422), light detected after passing through the third sub-color filter (1423), light detected after passing through the fourth sub-color filter (1424), light detected after passing through the fifth sub-color filter (1425), and light detected after passing through the sixth sub-color filter (1426). Referring to FIG. 14, the electronic device may perform image processing based on six color channels. However, the sub-channels that can be used as individual channels are not limited thereto.
[0145] FIG. 15 is a flowchart (1500) illustrating a process in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment processes data according to a shooting mode.
[0146] In the present disclosure, the operation of an electronic device (e.g., the electronic device (101) of FIG. 1) may be understood as being performed by a processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2) executing instructions stored in memory (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2) to perform operations or by controlling components of the electronic device.
[0147] According to one embodiment, in operation 1510, the electronic device may determine a shooting mode for obtaining an image. For example, the electronic device may determine a shooting mode based on various elements of the surrounding environment (e.g., brightness, distance, movement) or the user's purpose. For example, the shooting mode may include a first mode, a second mode, a third mode, or a fourth mode. However, the shooting mode is not limited thereto. For example, the electronic device may include all of only some of the four shooting modes. For example, the electronic device may include additional shooting modes of different types. For example, the electronic device may perform different shooting modes together. In one embodiment, the order of operations performed by the electronic device to determine the shooting mode is not limited to the order of the flowchart (1500). For example, the electronic device may determine whether to perform the first mode, the second mode, the third mode, and the fourth mode in parallel. For example, the electronic device may simultaneously determine the mode to be performed among the first mode, the second mode, the third mode, and the fourth mode.
[0148] According to one embodiment, the electronic device may determine in operation 1511 whether the shooting mode is a first mode. For example, the electronic device may capture an image based on the first mode when the illumination of the surrounding environment is lower than a threshold or when the user selects a low resolution. Based on the fact that the shooting mode is a first mode (e.g., the first mode of FIG. 12) (operation 1511-e), the electronic device may perform operation 1520. In operation 1520, the electronic device may acquire low-resolution Bayer pattern image data by merging a first pixel value and a second pixel value.
[0149] According to one embodiment, in operation 1512, the electronic device can determine whether the shooting mode is a second mode. For example, the electronic device can capture an image based on the second mode if the ambient illumination is lower than a threshold or if the user has selected a low resolution, but the scene being captured is classified by image analysis as a scene requiring color precision. Based on the fact that the shooting mode is a second mode (e.g., the second mode of FIG. 13) (operation 1512-e), the electronic device can perform operation 1530. In operation 1530, the electronic device can apply a first CCM (e.g., 1350 of FIG. 13) to a first pixel value and apply a second CCM (e.g., 1355 of FIG. 13) to a second pixel value. In operation 1535, the electronic device can perform a binning operation (e.g., 1380 of FIG. 13) to obtain low-resolution Bayer pattern image data.
[0150] According to one embodiment, in operation 1513, the electronic device may determine whether the shooting mode is a third mode. For example, the electronic device may capture an image based on the third mode when the illumination of the surrounding environment is above a threshold or when the user selects a high resolution. Based on the fact that the shooting mode is a third mode (e.g., the third mode of FIG. 13) (operation 1513-e), the electronic device may perform operation 1540. In operation 1540, the electronic device may apply a first CCM (e.g., 1350 of FIG. 13) to a first pixel value and a second CCM (e.g., 1355 of FIG. 13) to a second pixel value. In operation 1540, the CCMs applied to the first pixel value and the second pixel value may use matrices of the same size as the CCMs applied in operation 1530, but different CCMs may also be applied. In operation 1545, the electronic device can perform a remosaic operation (e.g., 1385 in FIG. 13) to acquire high-resolution Bayer pattern image data.
[0151] According to one embodiment, in operation 1514, the electronic device can determine whether the shooting mode is a fourth mode. For example, the electronic device can capture an image based on the fourth mode if the user selects a multi-spectral shooting mode. Based on the fact that the shooting mode is a fourth mode (e.g., the fourth mode of FIG. 14) (operation 1514-e), the electronic device can perform operation 1550. In operation 1550, the electronic device can perform a multi-spectral sensor mode. For example, the electronic device can use all sub-channels as individual channels. For example, if an electronic device includes a first color filter group corresponding to a first color, a second color filter group corresponding to a second color, and a third color filter group corresponding to a third color, and the first color filter group includes a first sub-color filter and a second sub-color filter, the second color filter group includes a third sub-color filter and a fourth sub-color filter, and the third color filter group includes a fifth sub-color filter and a sixth sub-color filter, the electronic device may perform image signal processing by using all of the pixel values detecting light that passed through the first sub-color filter, the pixel values detecting light that passed through the second sub-color filter, the pixel values detecting light that passed through the third sub-color filter, the light that passed through the fourth sub-color filter, the pixel values detecting light that passed through the fifth sub-color filter, and the pixel values detecting light that passed through the sixth sub-color filter. However, the sub-channels that can be used as individual channels are not limited thereto.
[0152] According to one embodiment, based on the fact that the shooting mode is a first mode (e.g., the first mode of FIG. 12), the electronic device can perform operation 1560 from the data obtained in operation 1520. Based on the fact that the shooting mode is a second mode (e.g., the second mode of FIG. 13), the electronic device can perform operation 1560 from the data obtained in operation 1535. Based on the fact that the shooting mode is a third mode (e.g., the third mode of FIG. 13), the electronic device can perform operation 1560 from the data obtained in operation 1545. In operation 1560, the electronic device can obtain a color image from the Bayer pattern image data.
[0153] FIG. 16 is a flowchart (1600) illustrating a process in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment acquires images through a plurality of cameras (e.g., the first camera (381) and the second camera (383) of FIG. 3).
[0154] According to one embodiment, the first camera (1610) may include a first image sensor comprising a color filter group composed of sub-color filters. For example, the first camera (1610) may be a camera (e.g., a telephoto camera) in which the angle at which the field of view (FoV) is formed is relatively narrow among multiple cameras. The telephoto camera may have a relatively high aperture value, so the amount of light incident on the image sensor may be small. Even when the amount of light is small, an image according to an embodiment of the present disclosure may be included in the telephoto camera to ensure color precision. In one embodiment, when acquiring image data by applying different CCMs for each sub-color filter, the camera may reduce noise included in the acquired image data by reducing the magnitude of the gain value that can amplify noise. The second camera (1615) may include a second image sensor composed of a general color filter. For example, the second camera (1615) may be a camera (e.g., a wide-angle camera) in which the angle at which the FoV is formed is wider than that of the first camera. However, the structure of the first image sensor included in the first camera and the structure of the second image sensor included in the second camera may be interchangeable. For example, the structure of the image sensor applied may change depending on the pixel size of the image sensor.
[0155] According to one embodiment, in operation 1620, the electronic device may acquire a sub-channel image (1620) through the first camera (1610). The sub-channel image (1620) may include raw data including data acquired through sub-channels. In operation 1630, the electronic device may perform image signal processing on the sub-channel image (1620). The electronic device may acquire processed image data (1640).
[0156] According to one embodiment, the electronic device can acquire a raw image (1645) through a second camera (1615). In operation 1650, the electronic device can acquire a color image (1660) by performing image processing on the raw data (1645) by referring to the processed image data (1640). For example, the electronic device can provide a preview or captured image by processing an image captured through the second camera (1615) by referring to the image data (1640) processed through the first camera (1610). For example, the electronic device can complement the data received through the second image sensor by mapping the data received through the first image sensor with the data received through the second image sensor. For example, since the first camera (1610) including the first image sensor is a telephoto camera, the central part of the raw image (1645) acquired through the second image sensor may have data included in the processed image (1640) applied to it, and the peripheral part may consist only of data included in the raw image (1645). For example, when zooming in beyond a certain amount using the second image sensor, the electronic device may provide an image acquired based on the value of the data received through the first image sensor. Through operation 1650, the color reproduction of the second camera (1615) can be improved. In operation 1660, the electronic device may perform operation 1650 to acquire a color image (1660) of the data received through the second image sensor.
[0157] FIG. 17 is a diagram illustrating examples of pixels for obtaining phase difference information from an image sensor according to one embodiment.
[0158] In one embodiment, the first arrangement (1701) may include a first sub-color filter (1721), a second sub-color filter (1722), a third sub-color filter (1723), and a fourth sub-color filter (1724) for a first color. For example, the second sub-color filter (1722) may be arranged adjacent to the first sub-color filter (1721) in a first direction (1711). For example, the third sub-color filter (1723) may be arranged adjacent to the second sub-color filter (1722) in a second direction (1712) perpendicular to the first direction (1711). For example, the fourth sub-color filter (1724) may be arranged adjacent to the first sub-color filter (1721) in a second direction (1712). In one embodiment, the electronic device may acquire first phase difference information for auto focus. For example, the first phase difference information can be obtained as the average of the difference between the pixel value of the light passing through the fourth sub-color filter (1724) and the pixel value of the light passing through the second sub-color filter (1722), and the difference between the pixel value of the light passing through the third sub-color filter (1723) and the pixel value of the light passing through the first sub-color filter (1721). For example, the operation to obtain the first phase difference information can be processed based on the following Equation 4. However, the method of obtaining the phase difference information is not limited thereto. For example, the operation to obtain the first phase difference information can be processed based on Equation 5.
[0159]
[0160]
[0161] In one embodiment, the second arrangement (1702) may include a fifth sub-color filter (1731), a sixth sub-color filter (1732), a seventh sub-color filter (1733), and an eighth sub-color filter (1734) for the first color. For example, the sixth sub-color filter (1732) may be arranged adjacent to the fifth sub-color filter (1731) in a first direction (1711). For example, the seventh sub-color filter (1733) may be arranged adjacent to the sixth sub-color filter (1732) in a second direction (1712) perpendicular to the first direction (1711). For example, the eighth sub-color filter (1734) may be arranged adjacent to the fifth sub-color filter (1731) in a second direction (1712). In one embodiment, the electronic device may acquire second phase difference information for autofocus. For example, the second phase difference information can be obtained as the average of the difference between the pixel value of the light passing through the fifth sub-color filter (1731) and the pixel value of the light passing through the sixth sub-color filter (1732), and the difference between the pixel value of the light passing through the eighth sub-color filter (1734) and the pixel value of the light passing through the seventh sub-color filter (1733). For example, the operation to obtain the second phase difference information can be processed based on the following mathematical formula 6. For example, a correlation operation can be performed on the pixel value obtained using mathematical formula 6, and the second phase difference information can be obtained through the performance of the correlation operation. However, the method of obtaining the phase difference information is not limited thereto.
[0162]
[0163] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may include an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384))). The image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, and the second image sensor (384))) may include a micro-lens array (e.g., the micro-lens array (411) of FIG. 4), a color filter array (e.g., the color filter array (413) of FIG. 4), and a sensor pixel array (e.g., the light receiving unit (415) of FIG. 4). The color filter array (e.g., the color filter array (413) of FIG. 4) may include a first color filter group (e.g., the first color filter group (501) of FIG. 5) corresponding to a first color having a first wavelength band. The light-receiving material array (e.g., the light-receiving part (415) of FIG. 4) can output a signal that detects light transmitted through the micro-lens array (e.g., the micro-lens array (411) of FIG. 4) and the color filter array (e.g., the color filter array (413) of FIG. 4). The first color filter group (e.g., the first color filter group (501) of FIG. 5) may include a first sub-color filter (e.g., the first sub-color filter (511) of FIG. 5) and a second sub-color filter (e.g., the second sub-color filter (512) of FIG. 5). The first sub-color filter (e.g., the first sub-color filter (511) of FIG. 5) may allow light corresponding to a first sub-spectral response (e.g., the first sub-spectral response (531) of FIG. 5) to be transmitted. The second sub-color filter (e.g., the second sub-color filter (512) of FIG. 5) can transmit light corresponding to the second sub-spectral response (e.g., the second sub-spectral response (532) of FIG. 5).The first wavelength value (e.g., the first wavelength value (551) of FIG. 5) corresponding to the first peak value (e.g., the first peak value (541) of FIG. 5) of the first subspectral response (e.g., the first subspectral response (531) of FIG. 5) may be longer than the second wavelength value (e.g., the second wavelength value (552) of FIG. 5) corresponding to the second peak value (e.g., the second peak value (542) of FIG. 5) of the second subspectral response (e.g., the second subspectral response (532) of FIG. 5). The first wavelength value (e.g., the first wavelength value (551) of FIG. 5) and the second wavelength value (e.g., the second wavelength value (552) of FIG. 5) may be included within the first wavelength band corresponding to the first color.
[0164] In one embodiment, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may further include a memory (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2, the memory (330) of FIG. 3)) and at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the processor (320) of FIG. 3). The memory (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2, the memory (330) of FIG. 3)) may store at least one computer program including computer-executable instructions. The above processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3)) can be communicated to the above image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (382) of FIG. 3, second image sensor (384))) and the above memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (330) of FIG. 3). The sensor pixel array (e.g., the light receiving unit (415) of FIG. 4) may include a first sensor pixel (e.g., the first sensor pixel (521) of FIG. 5) that outputs a signal corresponding to light transmitted through the first sub-color filter (e.g., the first sub-color filter (511) of FIG. 5) and a second sensor pixel (e.g., the second sensor pixel (522) of FIG. 5) that outputs a signal corresponding to light transmitted through the second sub-color filter (e.g., the second sub-color filter (512) of FIG. 5).The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to obtain at least one color value based on a first pixel value read out through the first sensor pixel (e.g., first sensor pixel (521) of FIG. 5) and a second pixel value read out through the second sensor pixel (e.g., second sensor pixel (522) of FIG. 5). The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to acquire an image including image pixels having the acquired color values.
[0165] In one embodiment, the instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to apply a first color correction matrix (CCM) (e.g., first CCM (1350) of FIG. 13)) to data obtained from the first sensor pixel (e.g., first sensor pixel (521) of FIG. 5). The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) may apply a second CCM (e.g., second CCM (1355) of FIG. 13) that is different from the first CCM (e.g., first CCM (1350) of FIG. 13) to the data obtained from the second sensor pixel (e.g., second sensor pixel (522) of FIG. 5).
[0166] In one embodiment, the first color filter group (e.g., the first color filter group (601) of FIG. 6) may further include a third sub-color filter (e.g., the third sub-color filter (613) of FIG. 6) and a fourth sub-color filter (e.g., the fourth sub-color filter (614) of FIG. 6). The third sub-color filter (e.g., the third sub-color filter (613) of FIG. 6) may allow light corresponding to the first sub-spectral response (e.g., the first sub-spectral response (711) of FIG. 7) to be transmitted. The fourth sub-color filter (e.g., the fourth sub-color filter (614) of FIG. 6) may allow light corresponding to the second sub-spectral response (e.g., the second sub-spectral response (712) of FIG. 7) to be transmitted.
[0167] In one embodiment, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may further include a memory (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2, the memory (330) of FIG. 3)) and at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the processor (320) of FIG. 3). The memory (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2, the memory (330) of FIG. 3)) may store at least one computer program including computer-executable instructions. The above processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3)) can be communicated to the image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (382) of FIG. 3, second image sensor (384))) and the memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (330) of FIG. 3). The image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (382) of FIG. 3, second image sensor (384))) can support a low-resolution mode that outputs raw image data of relatively low resolution and a high-resolution mode that outputs raw image data of relatively high resolution. The sensor pixel array (e.g., the light receiving unit (415) of FIG. 4) may include a first sensor pixel (e.g., the first sensor pixel (621) of FIG. 6), a second sensor pixel, a third sensor pixel (e.g., the third sensor pixel (623) of FIG. 6), and a fourth sensor pixel (e.g., the fourth sensor pixel (624) of FIG. 6). The first sensor pixel (e.g., the first sensor pixel (621) of FIG. 6) may output a signal corresponding to light transmitted through the first sub-color filter (e.g., the first sub-color filter (611) of FIG. 6).The second sensor pixel (e.g., the second sensor pixel (622) of FIG. 6) can output a signal corresponding to light that has passed through the second sub-color filter (e.g., the second sub-color filter (612) of FIG. 6). The third sensor pixel (e.g., the third sensor pixel (623) of FIG. 6) can output a signal corresponding to light that has passed through the third sub-color filter (e.g., the third sub-color filter (613) of FIG. 6). The fourth sensor pixel (e.g., the fourth sensor pixel (624) of FIG. 6) can output a signal corresponding to light that has passed through the fourth sub-color filter (e.g., the fourth sub-color filter (614) of FIG. 6). The above instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to acquire at least one color value based on data read through the first sensor pixel (e.g., first sensor pixel (621) of FIG. 6), the second sensor pixel (e.g., second sensor pixel (622) of FIG. 6), the third sensor pixel (e.g., third sensor pixel (623) of FIG. 6) and the fourth sensor pixel (e.g., fourth sensor pixel (624) of FIG. 6). The above instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) so that the electronic A device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may be configured to acquire an image including image pixels having the acquired color values.
[0168] In one embodiment, the instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) to obtain a pixel value of the sensor pixel array by summing the data read through the first sensor pixel (e.g., first sensor pixel (621) of FIG. 6), the second sensor pixel (e.g., second sensor pixel (622) of FIG. 6), the third sensor pixel (e.g., third sensor pixel (623) of FIG. 6) and the fourth sensor pixel (e.g., fourth sensor pixel (624) of FIG. 6) based on the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3)) and the fourth sensor pixel (e.g., fourth sensor pixel (624) of FIG. 6) based on the electronic device (e.g., first sensor pixel (621) of FIG. 6) operating in the low-resolution mode.
[0169] In one embodiment, the instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) to obtain pixel values of the sensor pixel array based on each of the data read through the first sensor pixel (e.g., first sensor pixel (621) of FIG. 6), the second sensor pixel (e.g., second sensor pixel (622) of FIG. 6), the third sensor pixel (e.g., third sensor pixel (623) of FIG. 6) and the fourth sensor pixel (e.g., fourth sensor pixel (624) of FIG. 6) based on the electronic device (e.g., first sensor pixel (621) of FIG. 6) operating in the high-resolution mode.
[0170] In one embodiment, the second sub-color filter (e.g., the second sub-color filter (612) of FIG. 6, the second sub-color filter (1722) of FIG. 17) may be positioned adjacent to the first sub-color filter (e.g., the first sub-color filter (611) of FIG. 6, the first sub-color filter (1721) of FIG. 17) in a first direction (e.g., the first direction (1711) of FIG. 17). The third sub-color filter (e.g., the third sub-color filter (613) of FIG. 6, the third sub-color filter (1723) of FIG. 17) may be positioned adjacent to the second sub-color filter (e.g., the second sub-color filter (612) of FIG. 6, the second sub-color filter (1722) of FIG. 17) in a second direction (e.g., the first direction (1712) of FIG. 17) perpendicular to the first direction (e.g., the first direction (1711) of FIG. 17). The fourth sub-color filter (e.g., the fourth sub-color filter (614) of FIG. 6, the fourth sub-color filter (1724) of FIG. 17)) may be positioned adjacent to the first sub-color filter (e.g., the first sub-color filter (611) of FIG. 6, the first sub-color filter (1721) of FIG. 17) in the second direction (e.g., the first direction (1712) of FIG. 17).The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to obtain phase difference information using the average of the first data which is the phase difference between the first pixel value read through the first sensor pixel (e.g., first sensor pixel (621) of FIG. 6) and the third pixel value read through the third sensor pixel (e.g., third sensor pixel (623) of FIG. 6) and the second data which is the phase difference between the second pixel value read through the second sensor pixel (e.g., second sensor pixel (622) of FIG. 6) and the fourth pixel value read through the fourth sensor pixel (e.g., fourth sensor pixel (624) of FIG. 6).
[0171] In one embodiment, the color filter array (e.g., the color filter array (413) of FIG. 4) may further include a second color filter group (e.g., the second color filter group (602) of FIG. 6) that is positioned adjacent to the first color filter group (e.g., the first color filter group (601) of FIG. 6) and corresponds to a second color different from the first color. The second color filter group (e.g., the second color filter group (602) of FIG. 6) may include a fifth sub-color filter (e.g., the fifth sub-color filter (615) of FIG. 6) and a sixth sub-color filter (e.g., the sixth sub-color filter (616) of FIG. 6). The fifth sub-color filter (e.g., the fifth sub-color filter (615) of FIG. 6) may allow light corresponding to a third sub-spectral response (e.g., the third sub-spectral response (713) of FIG. 7) to be transmitted. The sixth sub-color filter (e.g., the sixth sub-color filter (616) of FIG. 6) may allow light corresponding to a fourth sub-spectral response (e.g., the fourth sub-spectral response (714) of FIG. 7) that is different from the third sub-spectral response (e.g., the third sub-spectral response (713) of FIG. 7) to be transmitted. The third wavelength value (e.g., the third wavelength value (733) of FIG. 7) corresponding to the third peak value (e.g., the third peak value (723) of FIG. 7) of the third sub-spectral response (e.g., the third sub-spectral response (713) of FIG. 7) is the fourth of the fourth sub-spectral response (e.g., the fourth sub-spectral response (714) of FIG. 7). It may be longer than the fourth wavelength value (e.g., the fourth wavelength value (734) of FIG. 7) corresponding to the peak value (e.g., the fourth peak value (724) of FIG. 7). The first sub-color filter (e.g., the first sub-color filter (611) of FIG. 6) and the fifth sub-color filter (e.g., the fifth sub-color filter (615) of FIG. 6) may be placed adjacent to each other.The second sub-color filter (e.g., the second sub-color filter (612) of FIG. 6) and the sixth sub-color filter (e.g., the sixth sub-color filter (616) of FIG. 6) may be placed adjacent to each other.
[0172] In one embodiment, the difference between the third wavelength value (e.g., the third wavelength value (733) of FIG. 7) and the fourth wavelength value (e.g., the fourth wavelength value (734) of FIG. 7) may be smaller than the difference between the first wavelength value (e.g., the first wavelength value (731) of FIG. 7) and the fourth wavelength value (e.g., the fourth wavelength value (734) of FIG. 7).
[0173] In one embodiment, the difference between the third peak value (e.g., the third peak value (723) in FIG. 7) and the fourth peak value (e.g., the fourth peak value (724) in FIG. 7) may be smaller than the difference between the first peak value (e.g., the first peak value (721) in FIG. 7) and the fourth peak value (e.g., the fourth peak value (724) in FIG. 7).
[0174] In one embodiment, the microlens array (e.g., the microlens array (411) of FIG. 4) may include a first microlens (e.g., the first microlens (641) of FIG. 6) and a second microlens (e.g., the second microlens (642) of FIG. 6). The first color filter group (e.g., the first color filter group (601) of FIG. 6) may be located below the first microlens (e.g., the first microlens (641) of FIG. 6). The second color filter group (e.g., the second color filter group (602) of FIG. 6) may be located below the second microlens (e.g., the second microlens (642) of FIG. 6).
[0175] In one embodiment, the first wavelength band may include a wavelength band (e.g., bandwidth (580) of FIG. 5) in which the first spectral response (e.g., first spectral response (570) of FIG. 5) for the first color filter group (e.g., first color filter group (501) of FIG. 5) has a value greater than or equal to half of the maximum value of the first spectral response (e.g., first spectral response (570) of FIG. 5).
[0176] In one embodiment, a method of operation for an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) including an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, the second image sensor (384)) may include a color filter array (e.g., the color filter array (413) of FIG. 4). The color filter array (e.g., the color filter array (413) of FIG. 4) may include a first color filter group (e.g., the first color filter group (501) of FIG. 5) corresponding to a first color having a first wavelength band. The above method may include an operation (e.g., operation 1010 of FIG. 10) of detecting light transmitted through a first sub-color filter (e.g., first sub-color filter (511) of FIG. 5) included in the first color filter group (e.g., first color filter group (501) of FIG. 5). The above method may include an operation (e.g., operation 1020 of FIG. 10) of detecting light transmitted through a second sub-color filter (e.g., second sub-color filter (512) of FIG. 5) included in the first color filter group (e.g., first color filter group (501) of FIG. 5). The first sub-color filter (e.g., first sub-color filter (511) of FIG. 5) may allow light corresponding to a first sub-spectral response (e.g., first sub-spectral response (531) of FIG. 5) to be transmitted. The second sub-color filter (e.g., the second sub-color filter (512) of FIG. 5) can transmit light corresponding to a second sub-spectral response (e.g., the second sub-spectral response (532) of FIG. 5) that is different from the first sub-spectral response (e.g., the first sub-spectral response (531) of FIG. 5).The first wavelength value (e.g., the first wavelength value (551) in FIG. 5) corresponding to the first peak value (e.g., the first peak value (541) in FIG. 5) of the first subspectral response (e.g., the first subspectral response (531) in FIG. 5) may be higher than the second wavelength value (e.g., the second wavelength value (552) in FIG. 5) corresponding to the second peak value (e.g., the second peak value (542) in FIG. 5) of the second subspectral response (e.g., the second subspectral response (532) in FIG. 5). The first wavelength value (e.g., the first wavelength value (551) in FIG. 5) and the second wavelength value (e.g., the second wavelength value (552) in FIG. 5)) may be included within a first wavelength band corresponding to the first color.
[0177] In one embodiment, the operation of acquiring the first pixel value (e.g., operation 1010 of FIG. 10) may include the operation of detecting and outputting a signal of light transmitted through the first sub-color filter (e.g., the first sub-color filter (511) of FIG. 5) via a first sensor pixel (e.g., the first sensor pixel (521) of FIG. 5). The operation of acquiring the second pixel value (e.g., operation 1020 of FIG. 10) may include the operation of detecting and outputting a signal of light transmitted through the second sub-color filter (e.g., the second sub-color filter (512) of FIG. 5) via a second sensor pixel (e.g., the second sensor pixel (522) of FIG. 5). The operation of acquiring the image may include the operation of acquiring an image (e.g., operation 1030 of FIG. 10) comprising an image pixel having a color value acquired based on the first pixel value and the second pixel value.
[0178] In one embodiment, the operation of acquiring the image may include the operation of applying a first color correction matrix (CCM) (e.g., the first CCM (1350) of FIG. 13) to the data obtained from the first sensor pixel (e.g., the first sensor pixel (521) of FIG. 5). The operation of acquiring the image may include the operation of applying a second CCM (e.g., the second CCM (1355) of FIG. 13) that is different from the first CCM (e.g., the first CCM (1350) of FIG. 13) to the data obtained from the second sensor pixel (e.g., the second sensor pixel (522) of FIG. 5).
[0179] In one embodiment, the first color filter group (e.g., the first color filter group (601) of FIG. 6) may further include a third sub-color filter (e.g., the third sub-color filter (613) of FIG. 6) and a fourth sub-color filter (e.g., the fourth sub-color filter (614) of FIG. 6). The third sub-color filter (e.g., the third sub-color filter (613) of FIG. 6) may allow light corresponding to the first sub-spectral response (e.g., the first sub-spectral response (711) of FIG. 7) to be transmitted. The fourth sub-color filter (e.g., the fourth sub-color filter (614) of FIG. 6) may allow light corresponding to the second sub-spectral response (e.g., the second sub-spectral response (712) of FIG. 7) to be transmitted.
[0180] In one embodiment, the image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (382) of FIG. 3, second image sensor (384))) may support a low-resolution mode that outputs raw image data of relatively low resolution and a high-resolution mode that outputs raw image data of relatively high resolution. The operation of acquiring the first pixel value (e.g., operation 1010 of FIG. 10) may include the operation of detecting and outputting a signal of light transmitted through the first sub-color filter (e.g., first sub-color filter (611) of FIG. 6) through the first sensor pixel (e.g., first sensor pixel (621) of FIG. 6). The operation of obtaining the second pixel value (e.g., operation 1020 of FIG. 10) may include an operation of detecting and outputting a signal of light that has passed through the second sub-color filter (e.g., the second sub-color filter (612) of FIG. 6) through a second sensor pixel (e.g., the second sensor pixel (622) of FIG. 6). The method may further include an operation of detecting and outputting a signal of light that has passed through the third sub-color filter (e.g., the third sub-color filter (613) of FIG. 6) through a third sensor pixel (e.g., the third sensor pixel (623) of FIG. 6). The method may further include an operation of detecting and outputting a signal of light that has passed through the fourth sub-color filter (e.g., the fourth sub-color filter (614) of FIG. 6) through a fourth sensor pixel (e.g., the fourth sensor pixel (624) of FIG. 6). The operation of acquiring the above image may include the operation of acquiring at least one color value based on data read by the electronic device through the first sensor pixel (e.g., the first sensor pixel (621) of FIG. 6), the second sensor pixel (e.g., the second sensor pixel (622) of FIG. 6), the third sensor pixel (e.g., the third sensor pixel (623) of FIG. 6) and the fourth sensor pixel (e.g., the fourth sensor pixel (624) of FIG. 6).A method comprising the operation of acquiring the above image, the operation of acquiring an image including an image pixel having the acquired color value.
[0181] In one embodiment, the operation of acquiring the image may include an operation (e.g., operation 1380 of FIG. 13) of acquiring the pixel value of the first color for one sensor pixel by summing the data read through the first sensor pixel (e.g., first sensor pixel (621) of FIG. 6), the second sensor pixel (e.g., second sensor pixel (622) of FIG. 6), the third sensor pixel (e.g., third sensor pixel (623) of FIG. 6) and the fourth sensor pixel (e.g., fourth sensor pixel (624) of FIG. 6) based on the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3) of FIG. 3) and the fourth sensor pixel (e.g., fourth sensor pixel (624) of FIG. 6) of FIG. 6) based on the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3) of FIG. 3) of FIG. 6) of FIG. 6) of FIG. 13 operating in the low-resolution mode.
[0182] In one embodiment, the operation of acquiring the image may include an operation of acquiring pixel values of the first color for four sensor pixels (e.g., operation 1385 of FIG. 13) based on data read through the first sensor pixel (e.g., first sensor pixel (621) of FIG. 6), the second sensor pixel (e.g., second sensor pixel (622) of FIG. 6), the third sensor pixel (e.g., third sensor pixel (623) of FIG. 6) and the fourth sensor pixel (e.g., fourth sensor pixel (624) of FIG. 6), based on the image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (382) of FIG. 3 of FIG. 6) of FIG. 6, based on the image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (382) of FIG. 3) of FIG. 3) of FIG. 6) of FIG. 6) of FIG. 6) being operated in the high-resolution mode.
[0183] In one embodiment, a computer-readable non-transient recording medium may have a computer program recorded thereon that causes an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) including an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (382) of FIG. 3, the second image sensor (384)) to perform the method when executed.
[0184] An image sensor according to one embodiment of the present disclosure, an electronic device including the same, or a method of operation thereof can provide an image with high color precision by limiting crosstalk between color filters of different colors.
[0185] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description of the present disclosure.
[0186] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.
[0187] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims or embodiments described in the specification of this disclosure.
[0188] In the present disclosure, the function or operation performed by an electronic device may be performed by one or more processors executing one or more instructions stored in memory. The function or operation of the electronic device mentioned in the present disclosure may be performed by a single processor executing one or more instructions, or by a combination of multiple processors executing one or more instructions. A processor mentioned in the present disclosure is understood to include a circuit for performing operations or controlling other components of the electronic device. For example, the one or more processors may include at least one of a central processing unit (CPU), a micro-processor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on chip (SoC), an application-specific integrated circuit (ASIC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operation of the electronic device described above.
[0189] In the present disclosure, a program (software module, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other forms of optical storage devices, or a magnetic cassette. Alternatively, it may be stored in a memory composed of some or all of these. The memory may be composed of a single storage medium or a combination of multiple storage media. The one or more instructions may be stored in a single storage medium or distributed across multiple storage media.
[0190] Additionally, the above program may be stored on an attachable storage device that can be accessed via a communication network such as the Internet, Intranet, LAN (local area network), WLAN (wide LAN), or SAN (storage area network), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.
[0191] In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression is selected to suit the situation presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural form, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.
[0192] Additionally, in the present disclosure, terms such as “part,” “module,” etc. may be a hardware component, such as a processor or circuit, and / or a software component executed by a hardware component, such as a processor.
[0193] "Parts" and "modules" may be implemented by a program that is stored on an addressable storage medium and can be executed by a processor. For example, "parts" and "modules" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as by processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.
[0194] The specific embodiments described in this disclosure are merely examples and do not limit the scope of this disclosure in any way. For the sake of brevity, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted.
[0195] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, comprising only b, comprising only c, or comprising a combination of two or more (comprising a and b, comprising b and c, comprising a and c, or comprising all of a, b, and c).”
[0196] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.
[0197] In the present disclosure, the term “if” will be understood, depending on the context, to mean “when, upon,” “in response to a decision,” or “in response to a detection.” Similarly, “when decided to,” or “when [mentioned condition or event] is detected” will be understood, optionally, to mean “when decided,” or “in response to a decision,” “when [mentioned condition or event] is detected,” or “in response to a detection.”
[0198] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit (or processing circuit) may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.
[0199] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.
[0200] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several hardware combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by an app store that distributes applications or a site or server that supplies or distributes various other software.
[0201] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.
Claims
1. In an electronic device including an image sensor, The above image sensor is: Microlens array; A color filter array comprising a first color filter group corresponding to a first color having a first wavelength band; and It includes a sensor pixel array that outputs a signal detecting light transmitted through the micro-lens array and the color filter array, and The first color filter group comprises a first sub-color filter that transmits light corresponding to a first sub-spectral response and a second sub-color filter that transmits light corresponding to a second sub-spectral response different from the first sub-spectral response. The first wavelength value corresponding to the first peak value of the first subspectral response is longer than the second wavelength value corresponding to the second peak value of the second subspectral response, and An electronic device in which the first wavelength value and the second wavelength value are included within the first wavelength band corresponding to the first color.
2. In Claim 1, Memory for storing at least one computer program including computer-executable instructions; and It further includes at least one processor communicateably connected to the image sensor and the memory, and The sensor pixel array includes a first sensor pixel that outputs a signal corresponding to light transmitted through the first sub-color filter and a second sensor pixel that outputs a signal corresponding to light transmitted through the second sub-color filter. The above instructions are executed individually or collectively by the at least one processor, and the electronic device: At least one color value is obtained based on a first pixel value read out through the first sensor pixel and a second pixel value read out through the second sensor pixel, and An electronic device for acquiring an image including image pixels having the above-mentioned acquired color values.
3. In Claim 2, The above instructions are executed individually or collectively by the at least one processor, and the electronic device, An electronic device that applies a first color correction matrix (CCM) to data obtained from the first sensor pixel and applies a second CCM different from the first CCM to data obtained from the second sensor pixel.
4. In Claim 1, The electronic device further comprises a first color filter group, a third sub-color filter that transmits light corresponding to the first sub-spectral response, and a fourth sub-color filter that transmits light corresponding to the second sub-spectral response.
5. In Claim 4, Memory for storing at least one computer program including computer-executable instructions; and It includes at least one processor that is communically connected to the image sensor and the memory, and The above image sensor supports a low-resolution mode that outputs raw image data of relatively low resolution and a high-resolution mode that outputs raw image data of relatively high resolution, and The sensor pixel array comprises a first sensor pixel that outputs a signal corresponding to light transmitted through the first sub-color filter, a second sensor pixel that outputs a signal corresponding to light transmitted through the second sub-color filter, a third sensor pixel that outputs a signal corresponding to light transmitted through the third sub-color filter, and a fourth sensor pixel that outputs a signal corresponding to light transmitted through the fourth sub-color filter. The above instructions are executed individually or collectively by the above at least one processor, and the electronic device: At least one color value is obtained based on data read through the first sensor pixel, the second sensor pixel, the third sensor pixel, and the fourth sensor pixel, and An electronic device for acquiring an image including image pixels having the color values acquired above.
6. In Claim 5, The above instructions are executed individually or collectively by the at least one processor, and based on the electronic device operating in the low-resolution mode, the data read through the first sensor pixel, the second sensor pixel, the third sensor pixel, and the fourth sensor pixel is summed to obtain a pixel value of the sensor pixel array.
7. In Claim 5, The above instructions are executed individually or collectively by the at least one processor to obtain pixel values of the sensor pixel array based on each of the data read through the first sensor pixel, the second sensor pixel, the third sensor pixel, and the fourth sensor pixel, based on the electronic device operating in the high-resolution mode.
8. In Claim 5, The second sub-color filter is positioned adjacent to the first sub-color filter in a first direction, the third sub-color filter is positioned adjacent to the second sub-color filter in a second direction perpendicular to the first direction, and the fourth sub-color filter is positioned adjacent to the first sub-color filter in the second direction. The above instructions are executed individually or collectively by the at least one processor to obtain phase difference information using the average of a first data which is a phase difference between a first pixel value read through the first sensor pixel and a third pixel value read through the third sensor pixel, and a second data which is a phase difference between a second pixel value read through the second sensor pixel and a fourth pixel value read through the fourth sensor pixel.
9. In Claim 1, The color filter array further includes a second color filter group disposed adjacent to the first color filter group and corresponding to a second color different from the first color, and The second color filter group includes a fifth sub-color filter that transmits light corresponding to a third sub-spectral response and a sixth sub-color filter that transmits light corresponding to a fourth sub-spectral response different from the third sub-spectral response. The third wavelength value corresponding to the third peak value of the third subspectral response is longer than the fourth wavelength value corresponding to the fourth peak value of the fourth subspectral response, An electronic device in which the first sub-color filter and the fifth sub-color filter are arranged adjacent to each other, and the second sub-color filter and the sixth sub-color filter are arranged adjacent to each other.
10. An electronic device according to claim 9, wherein the difference between the third wavelength value and the fourth wavelength value is smaller than the difference between the first wavelength value and the fourth wavelength value.
11. An electronic device according to claim 9, wherein the difference between the third peak value and the fourth peak value is smaller than the difference between the first peak value and the fourth peak value.
12. In claim 9, the micro-lens array comprises a first micro-lens and a second micro-lens, and An electronic device in which the first color filter group is located below the first micro lens, and the second color filter group is located below the second micro lens.
13. In Claim 1, An electronic device comprising a wavelength band in which the first wavelength band comprises a wavelength band in which the first spectral response for the first color filter group has a value greater than or equal to half of the maximum value of the first spectral response.
14. A method of operating an electronic device including an image sensor, The image sensor includes a color filter array comprising a first color filter group corresponding to a first color having a first wavelength band, and An operation to obtain a first pixel value that detects light transmitted through a first sub-color filter included in the first color filter group; and The operation includes obtaining a second pixel value that detects light transmitted through a second sub-color filter included in the first color filter group, and The first subcolor filter transmits light corresponding to the first subspectral response, and The second sub-color filter transmits light corresponding to a second sub-spectral response different from the first sub-spectral response, and The first wavelength value corresponding to the first peak value of the first subspectral response is longer than the second wavelength value corresponding to the second peak value of the second subspectral response, and A method including an operation in which the first wavelength value and the second wavelength value are included within a first wavelength band corresponding to the first color.
15. In Claim 14, The operation of acquiring the first pixel value includes the operation of detecting and outputting a signal of light transmitted through the first sub-color filter via the first sensor pixel. The operation of acquiring the second pixel value includes the operation of detecting and outputting a signal of light transmitted through the second sub-color filter through the second sensor pixel. A method comprising the operation of acquiring the above image, wherein the operation of acquiring the above image includes the operation of acquiring an image including an image pixel having a color value acquired based on the first pixel value and the second pixel value.
