Image sensor, imaging module, electronic device, and image processing method and apparatus

Abstract

The present application relates to an image sensor, an imaging module, an electronic device, and an image processing method and apparatus. The image sensor comprises: a pixel layer (102), which is used for acquiring an optical signal, and converting the optical signal into an electrical signal; a spectral circuit layer (104), which is used for processing the electrical signal, so as to obtain spectral information; and an event camera circuit layer (106), which is used for processing the electrical signal, so as to obtain event information.

Description

Image sensors, imaging modules, electronic devices, image processing methods and apparatus

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on December 25, 2024, application number 2024119338418, entitled "Image Sensor, Imaging Module, Electronic Device, Image Processing Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and in particular to an image sensor, imaging module, electronic device, image processing method and apparatus. Background Technology

[0004] With the development of image processing technology, the quality of image data has been greatly improved. Generally, the quality of image data can be improved by adjusting the white balance and removing flicker.

[0005] In traditional technology, multispectral sensors can detect color temperature information and perform white balance processing based on the color temperature information, but they cannot detect flicker, which requires a flicker sensor to detect, increasing additional costs. Summary of the Invention

[0006] This application provides an image sensor, an imaging module, an electronic device, an image processing method, and an apparatus.

[0007] In a first aspect, this application provides an image sensor, the image sensor comprising at least:

[0008] A pixel layer is used to acquire light signals and convert the light signals into electrical signals;

[0009] A spectral circuit layer is used to process the electrical signal to obtain spectral information; and

[0010] The event camera circuit layer is used to process the electrical signals to obtain event information.

[0011] Secondly, this application also provides an imaging module, including:

[0012] An imaging lens, and an image sensor as described in the first aspect; the image sensor is configured to receive light signals passing through the imaging lens, convert the light signals into electrical signals, and process the electrical signals to obtain spectral information and event information.

[0013] Thirdly, this application also provides an electronic device, including an imaging module as described in the second aspect; the image sensor is used to receive light signals passing through the imaging lens, convert the light signals into electrical signals, and process the electrical signals to obtain spectral information and event information.

[0014] Fourthly, this application also provides an image processing method, comprising:

[0015] The image sensor acquires spectral information corresponding to multiple bands, as well as information on multiple events.

[0016] Based on the spectral information corresponding to the multiple bands, the white balance parameters are determined;

[0017] Based on the aforementioned event information, the flicker detection result is determined; and

[0018] Based on the white balance parameters and the flicker detection results, the target image data is determined.

[0019] Fifthly, this application also provides an image processing apparatus, the apparatus comprising:

[0020] The acquisition module is used to acquire spectral information corresponding to multiple bands and multiple event information through the image sensor;

[0021] The first determining module is used to determine the white balance parameters based on the spectral information corresponding to the multiple bands;

[0022] The second determining module is used to determine the flicker detection result based on the multiple event information; and

[0023] The processing module is used to determine the target image data based on the white balance parameters and the flicker detection results.

[0024] In a sixth aspect, this application also provides an electronic device, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the process of any one of the methods in the fourth aspect.

[0025] In a seventh aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the process of any one of the methods in the fourth aspect.

[0026] Eighthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the process of any one of the methods in the fourth aspect.

[0027] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.

[0029] Figure 1 is a schematic diagram of an image sensor in one embodiment.

[0030] Figure 2 is a partial schematic diagram of the pixel layer in one embodiment.

[0031] Figure 3 is a schematic diagram of the connection between the pixel unit and the circuit layer in one embodiment.

[0032] Figure 4 is a partial schematic diagram of the pixel layer in another embodiment.

[0033] Figure 5 is a partial schematic diagram of the pixel layer in another embodiment.

[0034] Figure 6 is a schematic diagram of the connection between the pixel unit and the circuit layer in another embodiment.

[0035] Figure 7 is a flowchart illustrating an image processing method in one embodiment.

[0036] Figure 8 is a structural block diagram of an image processing device in one embodiment.

[0037] Figure 9 is an internal structure diagram of an electronic device in one embodiment. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first pixel unit may be referred to as a second pixel unit, and similarly, a second pixel unit may be referred to as a first pixel unit. Both the first pixel unit and the second pixel unit are pixel units, but they are not the same pixel unit.

[0041] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.

[0042] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, process, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, processes, operations, components, parts, or combinations thereof.

[0043] In some exemplary embodiments, as shown in FIG1, an image sensor is provided, which includes at least a pixel layer 102, a spectral circuit layer 104, and an event camera circuit layer 106.

[0044] Pixel layer 102 is used to acquire light signals and convert them into electrical signals.

[0045] In this context, pixel layer 102 refers to the photosensitive and data acquisition layer of the image sensor. Pixel layer 102 includes multiple pixel units, which can acquire light signals and convert them into electrical signals. The light signal refers to the light wave information in the scene being captured by the pixel layer, including but not limited to the intensity, color, and wavelength of the light wave. The electrical signal refers to the signal after the light signal has been converted by the photosensitive element. The electrical signal can be represented as one of current, voltage, or charge change, and it characterizes the properties of the light signal. In the image sensor shown in Figure 1, pixel layer 102 is located in the first layer, spectral circuit layer 104 is located in the second layer, and event camera circuit layer 106 is located in the third layer. It should be understood that the image sensor shown in Figure 1 is only an image sensor in some embodiments and is not intended to limit the spatial relationship between pixel layer 102, spectral circuit layer 104, and event camera circuit 106.

[0046] For example, the pixel units in pixel layer 102 acquire light signals in the shooting scene and then convert the light signals into corresponding electrical signals.

[0047] The spectral circuit layer 104 is used to process electrical signals to obtain spectral information.

[0048] The spectral circuit layer 104 refers to the circuit layer used to determine spectral information. The spectral circuit layer 104 includes, but is not limited to, an electrical signal readout circuit and a spectral logic circuit. The electrical signal readout circuit in the spectral circuit layer 104 reads electrical signals from pixel units connected to the spectral circuit layer 104 in the pixel layer 102. The spectral logic circuit processes the electrical signals to obtain spectral information. Spectral information refers to the intensity distribution of light waves at different wavelengths, i.e., the spectral characteristics of light. Spectral information includes, but is not limited to, at least one of light intensity, wavelength, frequency, and spectral range.

[0049] For example, the electrical signal readout circuit in the spectral circuit layer 104 reads electrical signals from the pixel units in the pixel layer 102 that are connected to the spectral circuit layer 104, and the spectral logic circuit in the spectral circuit layer 104 processes the read electrical signals to obtain spectral signals.

[0050] The event camera circuit layer 106 is used to process electrical signals to obtain event information.

[0051] The event camera circuit layer 106 refers to the circuit layer used to determine event information. The event camera circuit layer 106 includes, but is not limited to, an electrical signal readout circuit and an event camera logic circuit. The electrical signal readout circuit in the event camera circuit layer 106 reads electrical signals from pixel units connected to the event camera circuit layer 106 in the pixel layer 102. The event camera logic circuit processes the electrical signals to obtain event information. Event information refers to information describing changes in light signal intensity, including but not limited to timestamps, locations, and change polarities. A timestamp can be the time marker corresponding to the moment the light signal intensity changes. A location can be the position where the light signal intensity changes. Change polarity can be an attribute describing whether the light signal intensity increases or decreases.

[0052] For example, the electrical signal readout circuit in the event camera circuit layer 106 reads electrical signals from the pixel units in the pixel layer 102 that are connected to the event camera circuit layer 106, and the event camera logic circuit in the event camera circuit layer 106 processes the read electrical signals to obtain event information.

[0053] In some exemplary embodiments, the image sensor includes a pixel layer 102, a spectral circuit layer 104, an event camera circuit layer 106, and at least one other circuit layer. The number and circuitry of the other circuit layers can be configured according to actual needs and are not limited herein.

[0054] The aforementioned image sensor uses a pixel layer to acquire light signals and convert them into electrical signals. A spectral circuit layer processes these electrical signals to obtain spectral information, and an event camera circuit layer processes the electrical signals to obtain event information. This means the image sensor can acquire both spectral information for white balance adjustment and event information for flicker removal. Furthermore, because the event camera circuit is more complex and requires a larger circuit board area, while the image sensor's circuit board area is smaller, the event camera circuit and the spectral circuit cannot coexist on the same circuit board. By placing the spectral circuit in the spectral circuit layer and the event camera circuit in the event camera circuit layer, the image sensor can simultaneously possess both spectral and event camera circuits. This allows the image sensor to determine both spectral and event information, saving hardware costs and reducing the size of stacked hardware components, thus saving space within the electronic device. It also enables adjustments to image data based on spectral and event information, improving image data quality.

[0055] In some exemplary embodiments, pixel layer 102 includes a plurality of pixel units, each pixel unit being connected to at least one of the spectral circuit layer 104 and the event camera circuit layer 106.

[0056] Pixel units are the basic components of pixel layer 102. Pixel units are used to acquire light signals and convert them into electrical signals. Pixel units may include microlenses, filters, and photosensitive elements. Microlenses are used to focus light and improve the efficiency of light signal acquisition; filters are used to filter light of specific wavelengths; and photosensitive elements are used to convert light signals into electrical signals. Pixel units can be divided into spectral pixel units and EVS (Event-based Vision Sensor) pixel units. Spectral pixel units are used to acquire light signals in a specific wavelength band and convert them into electrical signals; event-based camera pixel units are used to acquire light signals across the entire spectral range and convert them into electrical signals. This can be understood as event-based camera pixel units acquiring light signals across the entire visible spectrum, or a broader spectrum (such as infrared or ultraviolet light). For example, a partial schematic diagram of the pixel layer is shown in Figure 2. CH1 to CH8 in the pixel layer are spectral pixel units, and the EVS in the pixel layer are event-based camera pixel units. Both spectral pixel units and event-based camera pixel units are pixel units. The number of pixel units in the pixel layer can be set according to actual needs and is not limited here.

[0057] For example, each pixel unit in pixel layer 102 is connected to at least one of the spectral circuit layer 104 and the event camera circuit layer 106 via a metal via. Further, each pixel unit in pixel layer 102 can be vertically connected, laterally connected, bent, or connected with conductive adhesive to the spectral circuit layer 104 or the event camera circuit layer 106 via a metal via. Lateral connections can include buried vias and blind vias. A hole is a hole connecting inner layer circuits; neither end of it extends to the surface of the circuit board, but it is completely "buried" between the layers inside the circuit board. A blind via is a hole connecting outer layer circuits and inner layer circuits; it has only one end open on the surface of the circuit board, and the other end connected to the inner layer circuit. In flexible printed circuit boards (FPCs), in addition to connection methods similar to vias, connections can also be achieved by bending the circuit board itself.

[0058] In this embodiment, the pixel unit is connected to at least one of the spectral circuit layer and the event camera circuit layer. This can be understood as follows: when the pixel unit is connected to the spectral circuit layer, the electrical signal converted by the pixel unit can be transmitted to the spectral circuit layer for processing to obtain spectral information; when the pixel unit is connected to the event camera circuit layer, the electrical signal converted by the pixel unit can be transmitted to the event camera circuit layer for processing to obtain event information, thereby realizing the conversion of electrical signals into spectral information or event information.

[0059] In some exemplary embodiments, the plurality of pixel units include spectral pixel units and event camera pixel units, the spectral pixel units being connected to spectral circuit layer 104 and the event camera pixel units being connected to event camera circuit layer 106.

[0060] The number of spectral pixel units and event camera pixel units included in pixel layer 102, as well as the relative positions of the spectral pixel units and event camera pixel units, can be set according to actual needs and are not limited here.

[0061] For example, each spectral pixel unit in pixel layer 102 is connected to spectral circuit layer 104 via a metal via, and each event camera pixel unit in pixel layer 102 is connected to event camera circuit layer 106 via a metal via. The connection method for connecting spectral pixel units to spectral circuit layer 104 via metal vias can be one of vertical connection, side connection, bending connection, or conductive adhesive connection, etc. Similarly, the connection method for connecting event camera pixel units to event camera circuit layer 106 via metal vias can be one of vertical connection, side connection, bending connection, or conductive adhesive connection, etc. For example, as shown in Figure 3, spectral pixel unit 302 is vertically connected to spectral circuit layer 306 via metal via 304, and event camera pixel unit 308 is vertically connected to event camera circuit layer 312 via metal via 310.

[0062] In this embodiment, the pixel unit is connected to the spectral circuit layer. The electrical signal converted by the pixel unit is transmitted to the spectral circuit layer for processing to obtain spectral information, thereby realizing the conversion of electrical signal into spectral information. The pixel unit is also connected to the event camera circuit layer. The electrical signal converted by the pixel unit is transmitted to the event camera circuit layer for processing to obtain event information, thereby realizing the conversion of electrical signal into event information.

[0063] In some exemplary embodiments, the pixel layer 102 is divided into multiple pixel regions, each including multiple spectral pixel units and at least one event camera pixel unit; the multiple spectral pixel units are used to acquire light signals of different wavelengths and convert the light signals into electrical signals.

[0064] In this context, a pixel region refers to a local area within pixel layer 102, which can be composed of multiple pixel regions. A pixel region can consist of at least one minimum repeating region, which includes spectral pixel units for acquiring optical signals at different wavelengths and at least one event camera pixel unit. For example, as shown in Figure 4, a partial schematic diagram of the pixel layer shows that the minimum repeating region 402 includes spectral pixel units CH1 to CH15 and one event camera pixel unit EVS. Minimum repeating region 402 can be a single pixel region, and 404, which includes two minimum repeating regions, can also be a single pixel region. The number of event camera pixel units included in a pixel region can be set according to actual needs. For example, as shown in Figure 5, a partial schematic diagram of the pixel layer shows that pixel region 502 includes four event camera pixel units.

[0065] For example, the pixel layer 102 includes multiple pixel regions, each pixel region including multiple spectral pixel units and at least one event camera pixel unit. The multiple spectral pixel units in the pixel region are used to acquire light signals of different wavelengths and convert the light signals into electrical signals. The event camera pixel unit in the pixel region is used to acquire light signals and convert the light signals into electrical signals.

[0066] In this embodiment, the pixel layer is divided into multiple pixel regions. Each pixel region includes multiple spectral pixel units and at least one event camera pixel unit. The multiple spectral pixel units are used to acquire light signals of different wavelengths and convert the light signals into electrical signals. The white balance parameters corresponding to the pixel region can be determined based on the spectral information corresponding to the spectral pixel units in the pixel region. The flicker detection results corresponding to the pixel region can be determined based on the event information corresponding to the event camera pixel units in the pixel region. Then, the local image data in the initial image data aligned with the pixel region is corrected using the white balance parameters and flicker detection results corresponding to the pixel region. In other words, the pixel layer of the image sensor is divided into multiple pixel regions, which allows for simultaneous partitioned color temperature detection and flicker detection on the same image sensor.

[0067] In some exemplary embodiments, each pixel unit is connected to the spectral circuit layer 104 and the event camera circuit layer 106; different pixel units are used to acquire light signals of different wavelengths and convert the light signals into electrical signals.

[0068] For example, pixel layer 102 contains multiple pixel units. Different pixel units are used to acquire light signals of different wavelengths and convert the light signals into electrical signals. Each pixel unit in pixel layer 102 is connected to spectral circuit layer 104 and event camera circuit layer 106 through two metal vias, respectively. The connection methods between pixel units and spectral circuit layer 104 and event camera circuit layer 106 can be the same or different. For example, as shown in Figure 6, pixel unit 602 is vertically connected to spectral circuit layer 606 through metal via 604 and vertically connected to event camera circuit layer 610 through metal via 608; pixel unit 612 is vertically connected to spectral circuit layer 606 through metal via 614 and vertically connected to event camera circuit layer 610 through metal via 616.

[0069] In this embodiment, the pixel layer contains multiple pixel units. Different pixel units are used to acquire optical signals of different wavelengths and convert the optical signals into electrical signals. It can be understood that each pixel unit is used to acquire optical signals of a specified wavelength and convert the optical signals into electrical signals. The electrical signals can be used to determine spectral information and event information, laying the foundation for time-division multiplexing of the electrical signals converted by the pixel layer. Each pixel unit in the pixel layer is connected to the spectral circuit layer and the event camera circuit layer, further laying the foundation for time-division multiplexing of the electrical signals converted by the pixel layer.

[0070] In some exemplary embodiments, the pixel unit is used to acquire an optical signal in a first time period of the time-division multiplexing cycle, convert the optical signal into an electrical signal, transmit the electrical signal to the spectral circuit layer 104, and transmit the electrical signal to the event camera circuit layer 106 in a second time period of the time-division multiplexing cycle.

[0071] The time-division multiplexing cycle refers to the interval between the start of one complete measurement operation and the start of the next measurement operation for the image sensor. It can be understood as the time it takes for the pixel layer 102, spectral circuit layer 104, and event camera circuit layer 106 of the image sensor to complete one operation. The time-division multiplexing cycle includes a first time period and a second time period. The time-division multiplexing cycle can be a shooting cycle, the first time period can be the exposure time period of the shooting cycle, and the second time period can be the interval time period of the shooting cycle. The interval time period refers to the time period during which no exposure occurs, i.e., the non-exposure time period.

[0072] For example, the pixel unit is used to acquire the optical signal in the first time period of the time-division multiplexing cycle and convert the optical signal into an electrical signal. The electrical signal readout circuit in the spectral circuit layer 104 reads the electrical signal from the pixel unit of the pixel layer 102. In the second time period of the time-division multiplexing cycle, the electrical signal readout circuit in the event camera circuit layer 106 reads the electrical signal from the pixel unit of the pixel layer 102.

[0073] In some exemplary embodiments, after the electrical signal readout circuit in the spectral circuit layer 104 reads the electrical signal from the pixel unit of the pixel layer 102, the spectral logic circuit in the spectral circuit layer 104 further processes the read electrical signal to obtain a spectral signal. That is, the spectral logic circuit in the spectral circuit layer 104 processes the read electrical signal, which may or may not be completed within the first time period.

[0074] In some exemplary embodiments, after the electrical signal readout circuit in the event camera circuit layer 106 reads electrical signals from the pixel units of the pixel layer 102, the method further includes: the event camera logic circuit in the event camera circuit layer 106 processing the read electrical signals to obtain event information. That is, the processing of the read electrical signals by the event camera logic circuit in the event camera circuit layer 106 can be completed within the second time period or not within the second time period.

[0075] In this embodiment, the pixel unit acquires optical signals during the first time period of the time-division multiplexing cycle, converts the optical signals into electrical signals, and transmits the electrical signals to the spectral circuit layer. During the second time period of the time-division multiplexing cycle, the electrical signals are transmitted to the event camera circuit layer. That is, the optical signals converted by all pixel units are transmitted to the spectral circuit layer in the first time period and to the event camera circuit layer in the second time period, thus achieving time-division multiplexing of the electrical signals. Furthermore, the electrical signals converted by all pixel units are used to determine spectral information and event information, thereby increasing the amount of information in both spectral and event information and providing a large amount of accurate basic data for subsequent determination of white balance parameters and flicker detection results.

[0076] In some exemplary embodiments, pixel layer 102 is located in a first layer, one of the spectral circuit layer 104 and event camera circuit layer 106 is located in a second layer, and the other of the spectral circuit layer 104 and event camera circuit layer 106 is located in a third layer.

[0077] In this design, the layer in the image sensor that receives light signals is the first layer, which can be understood as the top layer in the direction in which light enters the image sensor. The second layer is located below the first layer, and the third layer is located below the second layer.

[0078] For example, the image sensor consists of a pixel layer 102, a spectral circuit layer 104, and an event camera circuit layer 106. The pixel layer 102 is located in the first layer, one of the spectral circuit layer 104 and the event camera circuit layer 106 is located in the second layer below the first layer, and the other circuit layer is located in the third layer below the second layer.

[0079] In this embodiment, the pixel layer is located in the first layer, and the spectral circuit layer and the event camera circuit layer are located below the pixel layer. There is no obstruction layer above the pixel layer, so that the pixel units in the pixel layer can collect light signals. Moreover, the spectral circuit and the event camera circuit are located in different layers, that is, the spectral circuit is located in the spectral circuit layer and the event camera circuit is located in the event camera circuit layer. This solves the problem that the complex event camera circuit cannot be located on the same circuit board as the spectral circuit. Thus, the image sensor can simultaneously have both spectral circuit and event camera circuit, enabling the image sensor to determine spectral information and event information.

[0080] In some exemplary embodiments, the spectral circuit layer 104 is located on the second layer, and the event camera circuit layer 106 is located on the third layer.

[0081] For example, the image sensor consists of a pixel layer 102, a spectral circuit layer 104, and an event camera circuit layer 106. The pixel layer 102 is located in the first layer, the spectral circuit layer 104 is located in the second layer below the pixel layer 102, and the event camera circuit layer 106 is located in the third layer below the spectral circuit layer 104.

[0082] In an exemplary embodiment, when the number of spectral pixel units in pixel layer 102 is greater than the number of event camera pixel units, pixel layer 102 is located in the first layer, spectral circuit layer 104 is located in the second layer below pixel layer 102, and event camera circuit layer 106 is located in the third layer below spectral circuit layer 104.

[0083] In this embodiment, the spectral circuit layer is located in the second layer below the pixel layer, so the length of the metal via connecting the spectral pixel unit and the spectral circuit layer is relatively short. The event camera circuit layer is located in the third layer below the spectral circuit layer, so the length of the metal via connecting the event camera pixel unit and the event camera circuit layer is relatively long. Generally, the number of spectral pixel units in the pixel layer is greater than the number of event camera pixel units, and the number of shorter metal vias is greater than the number of longer metal vias, thereby reducing the manufacturing cost of the image sensor.

[0084] In some exemplary embodiments, the event camera circuit layer 106 is located on the second layer, and the spectral circuit layer 104 is located on the third layer.

[0085] For example, the image sensor consists of a pixel layer 102, a spectral circuit layer 104, and an event camera circuit layer 106. The pixel layer 102 is located in the first layer, the event camera circuit layer 106 is located in the second layer below the pixel layer 102, and the spectral circuit layer 104 is located in the third layer below the event camera circuit layer 106.

[0086] In an exemplary embodiment, when the number of spectral pixel units in pixel layer 102 is less than the number of event camera pixel units, pixel layer 102 is located in a first layer, event camera circuit layer 106 is located in a second layer below pixel layer 102, and spectral circuit layer 104 is located in a third layer below event camera circuit layer 106.

[0087] In this embodiment, the event camera circuit layer is located in the second layer below the pixel layer, and the spectral circuit layer is located in the third layer below the event camera circuit layer. Therefore, the length of the metal via connecting the spectral pixel unit and the spectral circuit layer is relatively long. When the number of spectral pixel units in the pixel layer is less than the number of event camera pixel units, the number of shorter metal vias is greater than the number of longer metal vias, thereby reducing the manufacturing cost of the image sensor.

[0088] In some exemplary embodiments, an imaging module includes:

[0089] An imaging lens, and an image sensor in any of the above embodiments; the image sensor is used to receive light signals passing through the imaging lens, convert the light signals into electrical signals, and process the electrical signals to obtain spectral information and event information.

[0090] An imaging module is a key component in an electronic device used to capture light signals and generate image or video data. An imaging module includes multiple sub-components, which may include, but are not limited to, an imaging lens, an image sensor, and auxiliary circuitry. An imaging lens is the lens that focuses light signals onto the image sensor.

[0091] For example, the imaging lens and the image sensor in any of the above embodiments constitute an imaging module. The pixel layer 102 of the image sensor receives the light signal passing through the imaging lens and converts the light signal into an electrical signal. The spectral circuit layer 104 of the image sensor processes the electrical signal to obtain spectral information. The event camera circuit layer 106 of the image sensor processes the electrical signal to obtain event information.

[0092] In this embodiment, the imaging lens and the image sensor in any of the above embodiments can be used to form an imaging module. The imaging module is used to capture light signals and generate image processing data including spectral information and event information, providing accurate basic data for subsequent white balance adjustment and flicker removal.

[0093] In some exemplary embodiments, an electronic device includes the imaging module described in the above embodiments.

[0094] Electronic devices refer to devices that utilize electronic technology and circuits to perform functions. Electronic devices may include multiple imaging modules. Electronic devices can be, but are not limited to, various personal computers, laptops, smartphones, tablets, in-vehicle devices, IoT devices, and portable wearable devices. Portable wearable devices can include smartwatches, smart bracelets, smart glasses, smart helmets, etc.

[0095] Exemplarily, an electronic device includes the imaging module described in the above embodiments.

[0096] In some exemplary embodiments, an electronic device includes a plurality of imaging modules, one of which is the imaging module described in the above embodiments.

[0097] In this embodiment, an electronic device including the imaging module of the above embodiments can acquire spectral information and event information through the imaging module of the above embodiments, providing accurate basic data for the electronic device to perform white balance adjustment and flicker removal processing in the future.

[0098] In some exemplary embodiments, as shown in FIG7, an image processing method includes:

[0099] Process 702 involves acquiring spectral information corresponding to multiple bands and multiple event information through an image sensor.

[0100] The image sensor is a sensor composed of a pixel layer 102, a spectral circuit layer 104, and an event camera circuit layer 106. The image sensor is located in the imaging module, which is located in the electronic device.

[0101] For example, an electronic device acquires spectral information corresponding to multiple bands, as well as multiple event information, through an image sensor.

[0102] Process 704 determines the white balance parameters based on the spectral information corresponding to multiple bands.

[0103] White balance parameters refer to parameters used to reduce or avoid color cast in images caused by the color temperature of the light source. White balance parameters can be either shooting parameters (adjusted during shooting based on the color temperature of the light source in the scene) or correction parameters (used after shooting to correct color cast caused by the color temperature of the light source).

[0104] For example, the electronic device determines the white balance parameters based on the spectral information corresponding to multiple bands.

[0105] Process 706 determines the flicker detection result based on multiple event information.

[0106] The flicker detection result refers to the result of detecting changes in the brightness of the light source in the shooting scene. The flicker detection result may include, but is not limited to, the presence of flicker and flicker-reduction parameters. Flicker-reduction parameters can be divided into flicker-reduction shooting parameters, which are the parameter values ​​obtained by adjusting flicker-related parameters according to changes in the brightness of the light source in the shooting scene during the shooting process; flicker-reduction shooting parameters are used during the shooting process. Flicker-reduction parameters can also be flicker-reduction correction parameters, which are parameter values ​​used after shooting to correct bright and dark stripes caused by changes in the brightness of the light source in the initial image data; flicker-reduction correction parameters are used after shooting.

[0107] For example, an electronic device determines the flicker detection result based on multiple event information.

[0108] In some exemplary embodiments, the flicker detection result is determined based on multiple event information, including:

[0109] Event information includes timestamps and locations. The number of events with timestamps within the same time period and the same location is counted to obtain the frequency corresponding to each location. Based on the frequencies corresponding to each location, a time series signal is obtained. The time period refers to the time during which event information is collected. The timestamp refers to the time when the event occurred, and the location refers to the position of the event camera pixel unit where the event occurred. The peak frequency refers to the frequency corresponding to the peak value in the frequency signal. The flicker frequency range refers to a preset frequency range used to determine the flicker frequency. The flicker frequency range can be determined based on the power frequency; for example, if the power frequency is 50Hz, the flicker frequency range can be set to 45Hz to 55Hz.

[0110] In some exemplary embodiments, before obtaining the time series signal based on the frequencies corresponding to different locations, the method further includes: for each location, accumulating the frequencies corresponding to that location across multiple information collection time periods to obtain the accumulated frequency for that location; and obtaining the time series signal based on the accumulated frequencies corresponding to different locations. That is, by accumulating event information over a longer period, the influence of random noise is reduced while preserving the overall trend of change, thereby improving the accuracy of the time series signal.

[0111] In some exemplary embodiments, after determining that flicker exists, the method further includes: determining flicker-free shooting parameters based on the peak frequency, the flicker-free shooting parameters being used during the shooting process.

[0112] In process 708, the target image data is determined based on the white balance parameters and flicker detection results.

[0113] For example, the electronic device determines the target image data based on white balance parameters and flicker detection results.

[0114] In this embodiment, the electronic device acquires spectral information and event information corresponding to multiple bands through an image sensor, determines white balance parameters based on the spectral information corresponding to multiple bands, and determines flicker detection results based on the event information. The electronic device can simultaneously perform color temperature detection and flicker detection through the image sensor, and determine target image data based on white balance parameters and flicker detection results. That is, the target image data has undergone white balance adjustment and flicker removal processing at the same time, eliminating image color shift caused by light source color temperature and bright and dark stripes in the image caused by changes in light source brightness, thereby improving the quality of the target image data.

[0115] In some exemplary embodiments, spectral information corresponding to multiple bands and multiple event information are acquired through an image sensor, including:

[0116] The light signal is acquired by the spectral pixel unit in the pixel layer 102 of the image sensor and converted into a first electrical signal; the first electrical signal is processed by the spectral circuit layer 104 of the image sensor to obtain spectral information corresponding to multiple bands; the light signal is acquired by the event camera pixel unit in the pixel layer 102 and converted into a second electrical signal; the second electrical signal is processed by the event camera circuit layer 106 of the image sensor to obtain multiple event information.

[0117] The first electrical signal refers to the electrical signal converted from the light signal collected by the spectral pixel unit by the photosensitive element. The second electrical signal refers to the electrical signal converted from the light signal collected by the event camera pixel unit by the photosensitive element.

[0118] For example, the electronic device acquires light signals through multiple spectral pixel units in the pixel layer 102 of the image sensor and converts the light signals into first electrical signals. The first electrical signals are read from the multiple spectral pixel units through the spectral circuit layer 104 of the image sensor and processed to obtain spectral information corresponding to multiple bands. Light signals are also acquired through the event camera pixel units in the pixel layer 102 and converted into second electrical signals. The second electrical signals are read from the multiple event camera pixel units through the event camera circuit layer 106 of the image sensor and processed to obtain multiple event information.

[0119] In some exemplary embodiments, the electronic device acquires light signals through multiple spectral pixel units in the pixel layer 102 of the image sensor and converts the light signals into a first electrical signal. The electrical signal readout circuit in the spectral circuit layer 104 reads the first electrical signal from the spectral pixel units in the pixel layer 102 through a metal via. The spectral logic circuit in the spectral circuit layer 104 processes the read first electrical signal to obtain spectral information corresponding to multiple bands. The electronic device also acquires light signals through the event camera pixel units in the pixel layer 102 and converts the light signals into a second electrical signal. The electrical signal readout circuit in the event camera circuit layer 106 reads the second electrical signal from the event camera pixel units in the pixel layer 102 through a metal via. The event camera logic circuit in the event camera circuit layer 106 processes the read second electrical signal to obtain multiple event information.

[0120] In this embodiment, the electronic device determines spectral information corresponding to multiple bands through the spectral pixel units and spectral circuit layer in the pixel layer of the image sensor, and determines multiple event information through the event camera pixel units and event camera circuit layer in the pixel layer, providing accurate basic data for subsequent determination of white balance parameters and flicker detection results.

[0121] In some exemplary embodiments, the target image data is determined based on white balance parameters and flicker detection results, including:

[0122] Based on the white balance parameters and flicker detection results, the shooting parameters are determined; based on the shooting parameters, the target image data is acquired.

[0123] Shooting parameters refer to the parameters used to capture image data. They can be understood as the parameter values ​​that control how an electronic device captures light and generates image data during the image capture process. Shooting parameters include, but are not limited to, white balance shooting parameters and flicker removal shooting parameters.

[0124] For example, the electronic device determines the shooting parameters based on the white balance parameters and the flicker detection results, and uses the shooting parameters to capture target image data.

[0125] In this embodiment, the electronic device determines the shooting parameters based on the white balance parameters and the flicker detection results, and uses the shooting parameters to capture the target image data. That is, the electronic device performs white balance adjustment and flicker removal processing on the image data during the shooting process, thereby improving the quality of the target image data.

[0126] In some exemplary embodiments, the pixel layer 102 in the image sensor is divided into multiple pixel regions; white balance parameters are determined based on spectral information corresponding to multiple bands; flicker detection results are determined based on multiple event information; and target image data is determined based on the white balance parameters and the flicker detection results, including:

[0127] For each pixel region, the white balance parameters corresponding to the pixel region are determined based on the spectral information of the band corresponding to the spectral pixel unit in the pixel region; for each pixel region, the flicker detection result corresponding to the pixel region is determined based on the event information of the event camera pixel unit in the pixel region; and the target image data is determined based on the white balance parameters and flicker detection results corresponding to each pixel region.

[0128] For example, for each pixel region, the electronic device determines the white balance parameter corresponding to the pixel region based on the spectral information of the bands corresponding to the multiple spectral pixel units in the pixel region, determines the flicker detection result corresponding to the pixel region based on the event information corresponding to the multiple event camera pixel units in the pixel region, and processes the initial image data based on the white balance parameter and flicker detection result corresponding to each pixel region to obtain the target image data corresponding to the initial image data.

[0129] In this embodiment, the electronic device determines the white balance parameters and flicker detection results corresponding to the pixel region based on the spectral information of the bands corresponding to multiple spectral pixel units in the pixel region and the event information corresponding to multiple event camera pixel units, thereby realizing partitioned color temperature detection and flicker detection; then, based on the white balance parameters and flicker detection results corresponding to each pixel region, the target image data is determined, eliminating the image color shift caused by the light source color temperature and the bright and dark stripes in the image caused by the change in light source brightness, thus improving the quality of the target image data.

[0130] In some exemplary embodiments, the target image data is determined based on the white balance parameters and flicker detection results corresponding to each pixel region, including:

[0131] For each pixel region, determine the local image data aligned with the pixel region in the initial image data; based on the white balance parameters and flicker detection results corresponding to the pixel region, correct the local image data to obtain the corrected image data corresponding to the local image data; based on each corrected image data, obtain the target image data corresponding to the initial image data.

[0132] The initial image data refers to the image data to be corrected. This can be understood as the spectral information and event information from multiple bands acquired by the image sensor, used to correct the initial image data. Local image data refers to a portion of the initial image data. Specifically, the local image data aligned with pixel regions within the initial image data refers to the image data that needs correction using the white balance parameters and flicker detection results corresponding to the pixel regions.

[0133] For example, for each pixel region, local image data aligned with the pixel region in the initial image data is determined, and the local image data is corrected using the white balance parameters corresponding to the pixel region and the flicker detection results to obtain corrected image data corresponding to the local image data. The corrected image data are then combined to form the target image data.

[0134] In this embodiment, by using the white balance parameters corresponding to the pixel region and the flicker detection results, the local image data aligned with the pixel region in the initial image data is subjected to white balance adjustment and flicker removal processing to obtain the corrected image data corresponding to the local image data. This eliminates the image color shift caused by the light source color temperature and the bright and dark stripes in the image caused by the change in light source brightness, thereby improving the quality of the corrected image data. Based on each corrected image data, the target image data corresponding to the initial image data is obtained, thereby improving the quality of the target image data.

[0135] In some exemplary embodiments, spectral information corresponding to multiple bands and multiple event information are acquired through an image sensor, including:

[0136] During the exposure period of the shooting cycle, light signals are collected by the pixel units in the pixel layer 102 of the image sensor and converted into electrical signals. During the exposure period, the electrical signals are transmitted to the spectral circuit layer 104 in the image sensor to obtain spectral information corresponding to multiple bands. During the interval period of the shooting cycle, the electrical signals are transmitted to the event camera circuit layer 106 in the image sensor to obtain event information.

[0137] The shooting cycle refers to the time required to complete one full image acquisition and processing cycle. The shooting cycle includes an exposure period and an interval period. The exposure period refers to the time during which images are exposed within the shooting cycle, while the interval period refers to the time during which images are not exposed, i.e., the non-exposure period.

[0138] For example, during the exposure time period of the shooting cycle, the electronic device acquires light signals through the pixel units in the pixel layer 102 of the image sensor and converts the light signals into electrical signals. The electrical signal readout circuit in the spectral circuit layer 104 reads the electrical signals from the pixel units in the pixel layer 102, and the spectral logic circuit in the spectral circuit layer 104 processes the read electrical signals to obtain spectral information corresponding to multiple bands. During the interval time period of the shooting cycle, the electrical signal readout circuit in the event camera circuit layer 106 reads the electrical signals from the pixel units in the pixel layer 102, and the event camera logic circuit in the event camera circuit layer 106 processes the read electrical signals to obtain event information.

[0139] In some exemplary embodiments, during the exposure time period of the shooting cycle, the electronic device collects light signals through the pixel units in the pixel layer 102 of the image sensor in the secondary camera and converts the light signals into electrical signals; during the exposure time period, the electrical signals are transmitted to the spectral circuit layer 104 of the image sensor in the secondary camera to obtain spectral information corresponding to multiple bands; during the interval time period of the shooting cycle, the electrical signals are transmitted to the event camera circuit layer 106 of the image sensor in the secondary camera to obtain event information.

[0140] In this embodiment, the light signals converted from all pixel units in the pixel layer of the image sensor are transmitted to the spectral circuit layer during the exposure time period and to the event camera circuit layer during the interval time period, realizing time-division multiplexing of electrical signals and enabling simultaneous color temperature detection and flicker detection. Furthermore, the electrical signals converted from all pixel units are used to determine spectral information and event information, thereby increasing the amount of information in both spectral and event data and providing a large amount of accurate basic data for subsequent determination of white balance parameters and flicker detection results.

[0141] In some exemplary embodiments, the target image data is determined based on white balance parameters and flicker detection results, including:

[0142] During the exposure period of the shooting cycle, initial image data is acquired; based on white balance parameters and flicker detection results, correction parameters are determined; the initial image data is corrected based on the correction parameters to obtain target image data.

[0143] Among them, the correction parameters refer to the parameter values ​​used to correct the color shift of the image caused by the color temperature of the light source in the initial image data, and the parameter values ​​used to correct the bright and dark stripes caused by the change in the brightness of the light source in the initial image data. They can be understood as the parameter values ​​used to correct the color shift and bright and dark stripes of the initial image data.

[0144] For example, during the exposure period of a shooting cycle, the electronic device acquires initial image data through the main camera, as well as the corresponding white balance parameters and flicker detection results. Based on the white balance parameters and flicker detection results, correction parameters are determined, and the initial image data is corrected using these correction parameters to obtain the target image data. That is, the white balance parameters and flicker detection results used are those determined within the current shooting cycle.

[0145] In some exemplary embodiments, during the exposure period of the shooting cycle, the electronic device acquires initial image data through the main camera, as well as the white balance parameters and flicker detection results corresponding to the previous adjacent frame of the initial image data. Based on the white balance parameters and flicker detection results, correction parameters are determined, and the initial image data is corrected using these correction parameters to obtain the target image data. That is, the white balance parameters and flicker detection results are determined within the shooting cycle. Specifically, the white balance parameters and flicker detection results used are those determined in the previous shooting cycle.

[0146] In this embodiment, correction parameters are determined based on white balance parameters and flicker detection results. The correction parameters are then used to correct the initial image data to obtain the target image data. This eliminates the color shift caused by the light source color temperature and the bright and dark stripes in the image caused by changes in light source brightness in the initial image data, thereby improving the quality of the target image data.

[0147] In some exemplary embodiments, an image sensor is provided, comprising a pixel layer, a spectral circuit layer, and an event camera circuit layer. The pixel layer is located in a first layer, the spectral circuit layer is located in a second layer below the pixel layer, and the event camera circuit layer is located in a third layer below the spectral circuit layer. The pixel layer includes multiple spectral pixel units and multiple event camera pixel units. The spectral pixel units are connected to the spectral circuit layer through metal vias, and the event camera pixel units are connected to the event camera circuit layer through metal vias.

[0148] The electronic device includes a main camera and a secondary camera, with the image sensor located in the secondary camera's camera module. During shooting, the electronic device acquires initial image data through the main camera; it collects light signals through multiple spectral pixel units in the pixel layer of the image sensor and converts the light signals into a first electrical signal. The electrical signal readout circuit in the spectral circuit layer reads the first electrical signal from the spectral pixel units in the pixel layer through metal vias. The spectral logic circuit in the spectral circuit layer processes the read first electrical signal to obtain spectral information corresponding to multiple bands. It also collects light signals through the event camera pixel units in the pixel layer and converts the light signals into a second electrical signal. The electrical signal readout circuit in the event camera circuit layer reads the second electrical signal from the event camera pixel units in the pixel layer through metal vias. The event camera logic circuit in the event camera circuit layer processes the read second electrical signal to obtain multiple event information, including timestamps, locations, and polarities.

[0149] When the pixel layer is not divided into multiple pixel regions, the above image processing method includes:

[0150] Electronic devices determine white balance parameters based on spectral information corresponding to multiple bands.

[0151] The electronic device determines the flicker detection result based on multiple event information. For each location, the frequencies corresponding to that location over multiple information acquisition time periods are accumulated to obtain the cumulative frequency for that location. Based on the cumulative frequencies corresponding to different locations, a time series signal is obtained. The time series signal is subjected to Fourier transform to obtain a frequency domain signal. The peak frequency is determined based on the frequency domain signal. If the peak frequency is within the flicker frequency range, the flicker detection result is determined to be flickering; otherwise, the flicker detection result is determined to be flickering-free.

[0152] The electronic device captures target image data based on white balance parameters and flicker detection results.

[0153] In the above image processing method, the electronic device determines the shooting parameters based on the white balance parameters and the flicker detection results, and uses the shooting parameters to capture the target image data. That is, the electronic device performs white balance adjustment and flicker removal processing on the image data during the shooting process, thereby improving the quality of the target image data.

[0154] When the pixel layer is divided into multiple pixel regions, the above image processing method also includes:

[0155] For each pixel region, the electronic device determines the white balance parameter corresponding to the pixel region based on the spectral information of the bands corresponding to multiple spectral pixel units in the pixel region, and determines the flicker detection result corresponding to the pixel region based on the event information corresponding to multiple event camera pixel units in the pixel region; determines the local image data aligned with the pixel region in the initial image data, and corrects the local image data using the white balance parameter corresponding to the pixel region and the flicker detection result to obtain the corrected image data corresponding to the local image data, and combines the corrected image data into the target image data.

[0156] In the above image processing method, the electronic device uses the white balance parameters corresponding to the pixel region and the flicker detection results to perform white balance adjustment and flicker removal processing on the local image data aligned with the pixel region in the initial image data, thereby obtaining the corrected image data corresponding to the local image data. This eliminates the image color shift caused by the light source color temperature and the bright and dark stripes in the image caused by the change in light source brightness, thus improving the quality of the corrected image data. Based on each corrected image data, the target image data corresponding to the initial image data is obtained, thereby improving the quality of the target image data.

[0157] It should be understood that although the processes in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these processes are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these processes, and they can be executed in other orders. Moreover, at least some of the processes in the flowcharts of the embodiments described above may include multiple processes or multiple stages. These processes or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these processes or stages is not necessarily sequential, but may be performed alternately or in turn with other processes or at least a portion of processes or stages in other processes.

[0158] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0159] In some exemplary embodiments, as shown in FIG8, an image processing apparatus is provided, including: an acquisition module 802, a first determination module 804, a second determination module 806, and a processing module 808, wherein:

[0160] The acquisition module 802 is used to acquire spectral information corresponding to multiple bands and multiple event information through the image sensor;

[0161] The first determining module 804 is used to determine white balance parameters based on spectral information corresponding to multiple bands;

[0162] The second determining module 806 is used to determine the flicker detection result based on multiple event information;

[0163] The processing module 808 is used to determine the target image data based on the white balance parameters and flicker detection results.

[0164] In some exemplary embodiments, the acquisition module 802 is further configured to: acquire light signals through spectral pixel units in the pixel layer of the image sensor and convert the light signals into a first electrical signal; process the first electrical signal through the spectral circuit layer of the image sensor to obtain spectral information corresponding to multiple bands; acquire light signals through event camera pixel units in the pixel layer and convert the light signals into a second electrical signal; process the second electrical signal through the event camera circuit layer of the image sensor to obtain multiple event information.

[0165] In some exemplary embodiments, the processing module 808 is further configured to: determine shooting parameters based on white balance parameters and flicker detection results; and acquire target image data based on the shooting parameters.

[0166] In some exemplary embodiments, the first determining module 804 is further configured to: for each pixel region, determine the white balance parameter corresponding to the pixel region based on the spectral information of the band corresponding to the spectral pixel unit in the pixel region; the second determining module 806 is further configured to: for each pixel region, determine the flicker detection result corresponding to the pixel region based on the event information corresponding to the event camera pixel unit in the pixel region; the processing module 808 is further configured to: determine the target image data based on the white balance parameter and flicker detection result corresponding to each pixel region.

[0167] In some exemplary embodiments, the processing module 808 is further configured to: for each pixel region, determine local image data aligned with the pixel region in the initial image data; correct the local image data based on the white balance parameters and flicker detection results corresponding to the pixel region to obtain corrected image data corresponding to the local image data; and obtain target image data corresponding to the initial image data based on each corrected image data.

[0168] In some exemplary embodiments, the processing module 808 is further configured to: during the exposure time period of the shooting cycle, acquire light signals through pixel units in the pixel layer of the image sensor and convert the light signals into electrical signals; during the exposure time period, transmit the electrical signals to the spectral circuit layer in the image sensor to obtain spectral information corresponding to multiple bands; and during the interval time period of the shooting cycle, transmit the electrical signals to the event camera circuit layer in the image sensor to obtain event information.

[0169] In some exemplary embodiments, the processing module 808 is further configured to: acquire initial image data during the exposure time period of the shooting cycle; determine correction parameters based on white balance parameters and flicker detection results; and correct the initial image data based on the correction parameters to obtain target image data.

[0170] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0171] In an exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 9. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0172] Those skilled in the art will understand that the structure shown in Figure 9 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0173] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the processes described in the above method embodiments.

[0174] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the processes described in the above method embodiments.

[0175] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the processes described in the above method embodiments.

[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image sensor, characterized by, The image sensor includes at least: A pixel layer is used to acquire light signals and convert the light signals into electrical signals; A spectral circuit layer is used to process the electrical signal to obtain spectral information; and The event camera circuit layer is used to process the electrical signals to obtain event information.

2. The image sensor of claim 1, wherein, The pixel layer includes a plurality of pixel units, each of which is connected to at least one of the spectral circuit layer and the event camera circuit layer.

3. The image sensor of claim 2, wherein, The plurality of pixel units include spectral pixel units and event camera pixel units. The spectral pixel units are connected to the spectral circuit layer, and the event camera pixel units are connected to the event camera circuit layer.

4. The image sensor of claim 3, wherein, The pixel layer is divided into multiple pixel regions, each pixel region including multiple spectral pixel units and at least one event camera pixel unit; the multiple spectral pixel units are used to acquire light signals of different wavelengths and convert the light signals into electrical signals.

5. The image sensor of claim 2, wherein, Each pixel unit is connected to the spectral circuit layer and to the event camera circuit layer; different pixel units are used to acquire light signals of different wavelengths and convert the light signals into electrical signals.

6. The image sensor of claim 5, wherein, The pixel unit is used to acquire an optical signal in the first time period of the time-division multiplexing cycle, convert the optical signal into an electrical signal, transmit the electrical signal to the spectral circuit layer, and transmit the electrical signal to the event camera circuit layer in the second time period of the time-division multiplexing cycle.

7. The image sensor of claim 1, wherein, The pixel layer is located in the first layer, one of the spectral circuit layer and the event camera circuit layer is located in the second layer, and the other of the spectral circuit layer and the event camera circuit layer is located in the third layer.

8. The image sensor of claim 7, wherein, The spectral circuit layer is located on the second layer, and the event camera circuit layer is located on the third layer.

9. The image sensor of claim 7, wherein, The event camera circuit layer is located on the second layer, and the spectral circuit layer is located on the third layer.

10. An imaging module, characterized by include: Imaging lens, and The image sensor as described in any one of claims 1 to 9; the image sensor is used to receive light signals passing through the imaging lens, convert the light signals into electrical signals, and process the electrical signals to obtain spectral information and event information.

11. An electronic device, comprising: Including the imaging module as described in claim 10.

12. An image processing method, characterized by, include: The image sensor acquires spectral information corresponding to multiple bands, as well as information on multiple events. Based on the spectral information corresponding to the multiple wavebands, the white balance parameters are determined; Based on the aforementioned event information, the flicker detection result is determined; and Based on the white balance parameters and the flicker detection results, the target image data is determined.

13. The method according to claim 12, characterized in that, The acquisition of spectral information corresponding to multiple wavebands and multiple event information through an image sensor includes: The light signal is acquired by the spectral pixel unit in the pixel layer of the image sensor, and the light signal is converted into a first electrical signal; The first electrical signal is processed by the spectral circuit layer of the image sensor to obtain spectral information corresponding to multiple bands; Light signals are acquired through the event camera pixel units in the pixel layer, and the light signals are converted into a second electrical signal; and The second electrical signal is processed by the event camera circuit layer of the image sensor to obtain multiple event information.

14. The method according to claim 12 or 13, characterized in that, The determination of target image data based on the white balance parameters and the flicker detection results includes: Based on the white balance parameters and the flicker detection results, the shooting parameters are determined; and Based on the shooting parameters, the target image data is acquired.

15. The method of claim 13, wherein, The pixel layer in the image sensor is divided into multiple pixel regions; determining the white balance parameters based on the spectral information corresponding to the multiple bands includes: For each pixel region, the white balance parameter corresponding to the pixel region is determined based on the spectral information of the band corresponding to the spectral pixel unit in the pixel region. The determination of the flicker detection result based on the multiple event information includes: For each pixel region, based on the event information corresponding to the event camera pixel unit in the pixel region, determine the flicker detection result corresponding to the pixel region; and The determination of target image data based on the white balance parameters and the flicker detection results includes: The target image data is determined based on the white balance parameters and flicker detection results corresponding to each pixel region.

16. The method of claim 15, wherein, The determination of target image data based on the white balance parameters and flicker detection results corresponding to each pixel region includes: For each pixel region, determine local image data in the initial image data that is aligned with the pixel region; Based on the white balance parameters and flicker detection results corresponding to the pixel region, the local image data is corrected to obtain the corrected image data corresponding to the local image data; and Based on each of the corrected image data, the target image data corresponding to the initial image data is obtained.

17. The method of claim 12, wherein, The acquisition of spectral information corresponding to multiple wavebands and multiple event information through an image sensor includes: During the exposure period of the shooting cycle, light signals are collected by pixel units in the pixel layer of the image sensor and converted into electrical signals; During the exposure period, the electrical signal is transmitted to the spectral circuit layer in the image sensor to obtain spectral information corresponding to multiple bands; and During the intervals of the shooting cycle, the electrical signal is transmitted to the event camera circuit layer in the image sensor to obtain event information.

18. The method of claim 17, wherein, The determination of target image data based on the white balance parameters and the flicker detection results includes: During the exposure time period of the shooting cycle, initial image data is acquired; Based on the white balance parameters and the flicker detection results, the correction parameters are determined; and The initial image data is corrected based on the correction parameters to obtain the target image data.

19. An image processing apparatus characterized by comprising: The device includes: The acquisition module is used to acquire spectral information corresponding to multiple bands and multiple event information through the image sensor; The first determining module is used to determine the white balance parameters based on the spectral information corresponding to the multiple bands; The second determining module is used to determine the flicker detection result based on the multiple event information; and The processing module is used to determine the target image data based on the white balance parameters and the flicker detection results.

20. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the method of any one of claims 12 to 18.

21. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it performs the process of implementing the method of any one of claims 12 to 18.

22. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it performs the process of implementing the method of any one of claims 12 to 18.

Images