Image sensing apparatus, imaging apparatus and image sensing method

Abstract

Provided in the embodiments of the present application are an image sensing apparatus, an imaging apparatus and an image sensing method. The image sensing apparatus comprises: a pixel array, which comprises a plurality of readout units, wherein pixels of each readout unit share one floating diffusion region, which is connected to a readout line by means of a selection switch, and floating diffusion regions of every two adjacent readout units are connected to each other by means of a connection switch; and a control circuit, which is configured to, for a region to be binned in a current pixel binning mode, control connection switches to connect the floating diffusion regions in the readout units including pixels within the region to be binned, such that charges sensed by the pixels within the region to be binned are binned and accumulated in the connected floating diffusion regions, and to switch on the selection switch between the connected floating diffusion regions and any one of the connected readout lines, such that binned pixel information is outputted from a first readout line. By means of the embodiments of the present application, the same image sensing apparatus can support charge-domain binning of multiple sizes, so as to realize low-noise and high-definition imaging under multiple zoom ratios.

Description

Image sensing device, imaging device and image sensing method

[0001] This application claims priority to Chinese Patent Application No. 2025103779082, filed on March 26, 2025, entitled “Image Sensing Apparatus, Imaging Apparatus and Image Sensing Method”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of optical imaging technology, and in particular to image sensing devices, imaging devices, and image sensing methods. Background Technology

[0003] Pixel binning is a commonly used processing technique in the imaging field. Specifically, it combines multiple adjacent pixels within an image sensor into a single pixel for output. Depending on the specific implementation, pixel binning generally includes charge domain binning, analog domain binning, and digital domain binning. Among these, charge domain binning best guarantees the signal-to-noise ratio of the output signal.

[0004] In practical applications, pixel binning technology can be used to merge pixels of different sizes. Common binning sizes include merging 2×2 pixels into one pixel or 3×3 pixels into one pixel. In some application scenarios, there may be a need for pixel binning technology for different sizes simultaneously. One example is the In-Sensor Zoom (ISZ) function, which achieves digital zoom by directly cropping and processing the data inside the image sensor.

[0005] The ISZ function offers users a variety of zoom levels and specifically incorporates pixel binning technology to output different numbers of pixels as a single pixel at different zoom levels. As mentioned earlier, charge domain binning best guarantees the signal-to-noise ratio of the output signal. Therefore, the ISZ function also preferably uses charge domain binning to process pixel information within the image sensor to ensure low-noise signal output at all zoom levels.

[0006] However, current image sensor designs can only achieve 2×2 charge domain merging or 3×3 charge domain merging, and cannot support multi-size charge domain merging. As a result, they cannot achieve low-noise, high-definition imaging at multiple zoom magnifications, making it difficult to meet diverse user needs. Summary of the Invention

[0007] The purpose of this application is to provide an image sensing device, an imaging device, and an image sensing method, so as to achieve low-noise, high-resolution imaging at multiple zoom levels by using the same image sensing device to accommodate the merging of charge domains of multiple sizes. The specific technical solution is as follows:

[0008] In a first aspect, embodiments of this application provide an image sensing device, including:

[0009] A pixel array includes multiple readout units arranged in an array, each readout unit including multiple pixels; each pixel in a readout unit shares a floating diffuser, and the charge transfer between each pixel and the floating diffuser is independently controlled; the floating diffusers of every two adjacent readout units are connected by a connection switch.

[0010] Multiple readout lines, with the floating diffusion section of each readout unit connected to the readout lines via a selection switch;

[0011] The control circuit supports multiple pixel merging modes. Each pixel merging mode outputs pixel information for each region to be merged within the pixel array using a pixel merging method. A region to be merged contains multiple pixels, and the regions to be merged under different pixel merging modes are not identical. The control circuit is configured to: for the region to be merged under the current pixel merging mode, control the connection switch to connect the floating diffusion sections within each first readout unit. Each first readout unit contains pixels within the region to be merged, causing the charge sensed by each pixel within the region to be merged to accumulate and merge in the connected floating diffusion section; and set the selection switch between the connected floating diffusion section and the first readout line to a conducting state, connecting the first readout line to the connected floating diffusion section, so that the pixel information generated by merging pixels within the region to be merged is output through the first readout line.

[0012] An information processing unit is connected to the plurality of readout lines and is configured to generate an output image based on the pixel information output from the readout lines in the current pixel merging mode.

[0013] The image sensing device provided in this application embodiment arranges multiple readout units in an array within a pixel array, such that each pixel within a readout unit shares a floating diffuser, and the floating diffusers of every two adjacent readout units are connected by a connection switch. Based on this structural design, the control circuit can control the connection switch to connect the floating diffusers in two or more adjacent readout units into a whole, so that the charges sensed by each pixel in the two or more readout units can be merged and accumulated in the same floating diffuser, thereby realizing charge domain merging between pixels across readout units.

[0014] Based on the above mechanism, the image sensing device provided in this application embodiment can theoretically support charge domain merging of any size. Therefore, in practical applications, various pixel merging modes of different sizes can be configured according to specific needs. For a region to be merged under the current pixel merging mode, the control circuit connects the floating diffusion sections of each readout unit containing pixels in the region to be merged by controlling the connection switch. This allows the charges sensed by each pixel in the region to be merged to be merged and accumulated in the connected floating diffusion section. The merged and accumulated charges can be converted into electrical signals in the connected floating diffusion section. By turning on the selection switch between any one of the readout lines connected to the connected floating diffusion section and the connected floating diffusion section, the control circuit can output the electrical signal formed by the connected floating diffusion section from that readout line, thereby realizing the output of pixel information in the region to be merged by charge domain merging.

[0015] As can be seen from the above, based on the embodiments of this application, it is possible to use the same image sensing device to be compatible with the merging of multiple charge domains of different sizes. When the image sensing device provided based on the embodiments of this application implements the ISZ function, compared with the technical solution of achieving multiple zoom magnifications through analog domain merging of pixels, the pixel information output by the image sensing device in pixel merging mode has a higher signal-to-noise ratio, thereby ensuring low-noise, high-definition imaging at various zoom magnifications, which helps to provide users with diverse zoom magnification options while ensuring user experience.

[0016] In one embodiment of this application, each pixel is connected to the floating diffuser via a transmission switch;

[0017] The control circuit is specifically configured to: for the region to be merged in the current pixel merging mode, control the connection switch to connect the floating diffusion section in the adjacent readout units, so that the first region formed by the interconnected adjacent readout units includes the region to be merged; for each pixel in the first region, turn on the transmission switch between the pixel included in the region to be merged and the floating diffusion section, and turn off the transmission switch between the other pixels and the floating diffusion section, so that the charge sensed by each pixel in the region to be merged is merged and accumulated in the connected floating diffusion section.

[0018] In this embodiment of the application, for a region to be merged, the total capacitance of the connected floating diffusion section can be adjusted by adjusting the number of readout units contained in the first region used to output pixel information in the region to be merged, thereby controlling the conversion gain of the pixel information output from the region to be merged, and thus achieving higher output flexibility.

[0019] In one embodiment of this application, in a pixel merging mode, the number of readout units contained in the first region corresponding to each region to be merged is the same.

[0020] This application embodiment addresses each region to be merged in a one-pixel merging mode. By controlling that each region to be merged contains the same number of readout units within its corresponding first region, the total capacitance of the (connected) floating diffuser within each first region can be maintained at a substantially the same level. This helps ensure that the conversion gain of each pixel information output from this one-pixel merging mode is substantially the same, thereby helping to ensure the image quality of the final generated image.

[0021] In one embodiment of this application, in a pixel merging mode, the number of first connection switches in the conducting state and the number of second connection switches in the conducting state are the same in the first region corresponding to each region to be merged; the first connection switch is the connection switch of the floating diffusion part in two adjacent readout units in the row direction, and the second connection switch is the connection switch of the floating diffusion part in two adjacent readout units in the column direction.

[0022] In this embodiment, by controlling the number of first and second connection switches in the conducting state in each first region to be the same, it helps to maintain the total capacitance of the floating diffusion section in each first region at the same level with higher precision, so that the conversion gain of each pixel information output from a pixel merging mode is basically the same, thereby helping to further improve image quality.

[0023] In one embodiment of this application, the current pixel merging mode includes multiple groups to be merged, and each group to be merged includes one or more regions to be merged; each region to be merged in the same group corresponds to a different first region, and there is no overlap between any two first regions.

[0024] The control circuit is specifically configured to: synchronously output the pixel information of each region to be merged within the same merge group through different readout lines, and interleave the pixel information of the regions to be merged within different merge groups in terms of timing.

[0025] By grouping regions to be merged into the same merge group that correspond to different first regions and have no overlap between any two of them, the pixel information of each merged region in the same merge group is output synchronously, which helps to improve the output efficiency of pixel information.

[0026] In one embodiment of this application, the readout line is a column line, and the floating diffusion portion of each readout unit in the same column is connected to the same column line, while the floating diffusion portion of readout units in different columns is connected to different column lines.

[0027] The group to be merged includes a first region to be merged and a second region to be merged located in the same column. The first region corresponding to at least one of the first region to be merged and the second region to be merged covers at least two readout units in the row direction. The control circuit is specifically configured to output the pixel information of the first region to be merged and the pixel information of the second region to be merged synchronously through different column lines.

[0028] By synchronously outputting pixel information from different regions to be merged within the same column using different column lines, the output efficiency of pixel information can be improved.

[0029] In one embodiment of this application, the pixels in the pixel array are arranged as multiple extended units in an array, each extended unit includes 2×2 color units arranged in a Bayer pattern, and each color unit includes 3×3 pixels of the same color; the control circuit supports: a first pixel merging mode at 1x zoom, a second pixel merging mode at 2x zoom, and a full pixel mode.

[0030] In the first pixel merging mode, the pixels in the pixel array are combined into multiple 3×3 regions to be merged;

[0031] The second pixel merging mode includes a first sub-mode and / or a second sub-mode. In the first sub-mode, the pixel array contains a first output area, and the pixels in the first output area are combined into multiple 2×2 merged areas. In the second sub-mode, the pixels in each color unit in the first output area are combined into a 2×2 merged area and five 1×1 single pixel readout areas.

[0032] In the full-pixel mode, the pixel array includes a second output area, and each pixel in the second output area serves as a single-pixel readout area.

[0033] In the embodiments of this application, the image sensing device can realize a first pixel merging mode at a 1x zoom ratio, a second pixel merging mode at a 2x zoom ratio, and a full pixel mode at a 3x zoom ratio, providing users with a consistent zoom ratio selection and clear image display at each zoom ratio.

[0034] In one embodiment of this application, each readout unit includes 2×4 pixels, or each readout unit includes 2×2 pixels;

[0035] In the first pixel merging mode, each color unit within an expansion unit constitutes a region to be merged; the control circuit is specifically configured to: for an expansion unit, control the connection switch to sequentially connect to form the first region corresponding to each region to be merged, and output the corresponding pixel information.

[0036] In one embodiment of this application, each readout unit includes 2×4 pixels. In the first pixel merging mode, two adjacent extension units in the column direction are a first type of extension unit and a second type of extension unit, respectively. The first pixel of the first row in the first type of extension unit is the first pixel of the first row in the first readout unit; the first pixel of the first row in the second type of extension unit is the first pixel of the third row in the second readout unit; the first readout unit and the second readout unit are two adjacent readout units in the column direction.

[0037] In the first type of extended unit, the first region corresponding to the first region to be merged in the first row includes: the first three readout units in the first row within the scope of the first type of extended unit, and the first readout unit in the second row;

[0038] The first region corresponding to the second region to be merged in the first row includes: the first three readout units in the first row within the scope of the first type of extended unit, and the third readout unit in the second row;

[0039] The first region corresponding to the first region to be merged in the second row includes: the first two readout units in the first row within the scope of the first type of extended unit, and the first two readout units in the second row;

[0040] The first region corresponding to the second region to be merged in the second row includes: the second to third readout units in the first row within the scope of the first type of extended unit, and the second to third readout units in the second row;

[0041] In the second type of extended unit, the first region corresponding to the first region to be merged in the first row includes: the first two readout units of the first row within the scope of the second type of extended unit, and the first two readout units of the second row;

[0042] The first region corresponding to the second region to be merged in the first row includes: the second to third readout units in the first row within the scope of the second type of extended unit, and the second to third readout units in the second row;

[0043] The first region corresponding to the first region to be merged in the second row includes: the first read unit in the first row within the range of the second type of extended unit, and the first three read units in the second row;

[0044] The first region corresponding to the second region to be merged in the second row includes: the third readout unit in the first row within the scope of the second type of extended unit, and the first three readout units in the second row.

[0045] In the embodiments of this application, the number of readout units contained in the first region corresponding to each region to be merged is the same, which helps to ensure that the conversion gain of each pixel information output in the first pixel merging mode is basically the same, thereby helping to ensure the image quality of the final generated image.

[0046] In one embodiment of this application, in the first pixel merging mode, the control circuit is specifically configured to: synchronously output the pixel information of the first region to be merged in the first row of the first type of expansion unit and the pixel information of the second region to be merged in the first row of the second type of expansion unit through different readout lines in a first type of expansion unit and a second type of expansion unit in the same column and the next row of the first type of expansion unit; synchronously output the pixel information of the second region to be merged in the first row of the first type of expansion unit and the pixel information of the first region to be merged in the first row of the second type of expansion unit through different readout lines; synchronously output the pixel information of the first region to be merged in the second row of the first type of expansion unit and the pixel information of the second region to be merged in the second row of the second type of expansion unit through different readout lines; and synchronously output the pixel information of the second region to be merged in the second row of the first type of expansion unit and the pixel information of the first region to be merged in the second row of the second type of expansion unit through different readout lines.

[0047] In this embodiment of the application, by synchronously outputting the pixel information of the non-overlapping regions to be merged in the corresponding first regions, the output efficiency of pixel information can be improved.

[0048] In one embodiment of this application, in the first pixel merging mode, the number of first connection switches in the conducting state in the first region corresponding to each region to be merged in the pixel array is two, and the number of second connection switches in the conducting state is one.

[0049] In this embodiment, by controlling the number of first and second connection switches in the conducting state in each first region to be the same, it helps to maintain the total capacitance of the floating diffusion section in each first region at the same level with higher precision, so that the conversion gain of each pixel information output in the first pixel merging mode is basically the same, thereby helping to further improve image quality.

[0050] In one embodiment of this application, each readout unit includes 2×4 pixels. In the first sub-mode, a readout unit of the first output area includes two regions to be merged. The control circuit is specifically configured to: control all connection switches in the pixel array to be disconnected, and for each readout unit in the first output area, output the pixel information of the two regions to be merged in the readout unit in a staggered manner in time.

[0051] In this embodiment of the application, when each connection switch in the pixel array is in the off state, the pixel information of each region to be merged in the first sub-mode is output, which helps to achieve the highest possible conversion gain and low noise in the output pixel information.

[0052] In one embodiment of this application, each readout unit includes 2×2 pixels. In the first pixel merging mode, in an extended unit, the first region corresponding to the first region to be merged in the first row includes: the first two readout units in the first row and the first two readout units in the second row within the range of the extended unit; the first region corresponding to the second region to be merged in the first row includes: the second to third readout units in the first row and the second to third readout units in the second row within the range of the extended unit; the first region corresponding to the first region to be merged in the second row includes: the first two readout units in the second row and the first two readout units in the third row within the range of the extended unit; the first region corresponding to the second region to be merged in the second row includes: the second to third readout units in the second row and the second to third readout units in the third row within the range of the extended unit.

[0053] In the embodiments of this application, the number of readout units contained in the first region corresponding to each region to be merged is the same, which helps to ensure that the conversion gain of each pixel information output in the first pixel merging mode is basically the same, thereby helping to ensure the image quality of the final generated image.

[0054] In one embodiment of this application, the readout line is a column line, and the floating diffusion portion of each readout unit in the same column is connected to the same column line, while the floating diffusion portion of readout units in different columns is connected to different column lines.

[0055] In the first pixel merging mode, the control circuit is specifically configured to: for two adjacent expansion units in the column direction, synchronously output the pixel information of a region to be merged in one expansion unit and the pixel information of a region to be merged at the same position in the other expansion unit through different column lines.

[0056] In this embodiment of the application, synchronously outputting pixel information from different regions to be merged helps to improve the output efficiency of pixel information.

[0057] In one embodiment of this application, in the first pixel merging mode, the number of first connection switches in the conducting state in the first region corresponding to each region to be merged in the pixel array is two, and the number of second connection switches in the conducting state is one; or, the number of first connection switches in the conducting state in the first region corresponding to each region to be merged in the pixel array is one, and the number of second connection switches in the conducting state is two.

[0058] In this embodiment, by controlling the number of first and second connection switches in the conducting state in each first region to be the same, it helps to maintain the total capacitance of the floating diffusion section in each first region at the same level with higher precision, so that the conversion gain of each pixel information output in the first pixel merging mode is basically the same, thereby helping to further improve image quality.

[0059] In one embodiment of this application, each readout unit includes 2×2 pixels. In the first sub-mode, each readout unit in the first output area is a region to be merged. The control circuit is specifically configured to: control all connection switches in the pixel array to be disconnected, and output the pixel information of the corresponding region to be merged in each readout unit.

[0060] In this embodiment of the application, when each connection switch in the pixel array is in the off state, the pixel information of each region to be merged in the first sub-mode is output, which helps to achieve the highest possible conversion gain and low noise in the output pixel information.

[0061] In one embodiment of this application, each readout unit includes 1×3 pixels;

[0062] In the first pixel merging mode, for each region to be merged contained in the pixel array, the range of the first region corresponding to each region to be merged is consistent with the range of each region to be merged.

[0063] In one embodiment of this application, the readout line is a column line, and the floating diffusion portion of each readout unit in the same column is connected to the same column line, while the floating diffusion portion of readout units in different columns is connected to different column lines.

[0064] In the first pixel merging mode, the control circuit is specifically configured to: synchronously output the pixel information of each region to be merged in the same column through different column lines; and interleave the pixel information of each region to be merged in different columns in a timing sequence.

[0065] In this embodiment of the application, synchronously outputting pixel information from different regions to be merged helps to improve the output efficiency of pixel information.

[0066] In one embodiment of this application, under the first sub-mode, an expansion unit includes 3×3 regions to be merged; within an expansion unit, each region to be merged in the first column corresponds to the same first region, which includes: the first two readout units in the first row and the first two readout units in the second row within the scope of the expansion unit; each region to be merged in the second column corresponds to the same first region, which includes: the third to fourth readout units in the first row and the third to fourth readout units in the second row within the scope of the expansion unit; each region to be merged in the third column corresponds to the same first region, which includes: the fifth to sixth readout units in the first row and the fifth to sixth readout units in the second row within the scope of the expansion unit.

[0067] In this embodiment, the number of readout units contained in the first region corresponding to each region to be merged is the same, which helps to ensure that the conversion gain of each pixel information output in the first sub-mode is basically the same, thereby helping to ensure the image quality of the final generated image.

[0068] In one embodiment of this application, in the first sub-mode, the control circuit is specifically configured to: synchronously output the pixel information of three regions to be merged corresponding to different first regions for an expansion unit, and interleave the pixel information of different regions to be merged corresponding to the same first region in a timing manner.

[0069] In this embodiment of the application, synchronously outputting pixel information from different regions to be merged helps to improve the output efficiency of pixel information.

[0070] In one embodiment of this application, in the second sub-mode, the control circuit is specifically configured to: for the region to be merged, control the connection switch to connect the floating diffusion portion of the readout unit containing the pixels in the region to be merged, so as to output the pixel information of the region to be merged; for the single-pixel readout region, control the readout unit where the single-pixel readout region is located to disconnect from its adjacent readout units, and output the pixel information of the single-pixel readout region.

[0071] In this embodiment, the pixel information of the single-pixel readout area is output when the readout unit where the single-pixel readout area is located and its adjacent readout units are disconnected. This helps to make the conversion gain of the output single-pixel readout area pixel information as high as possible and the noise as low as possible.

[0072] In one embodiment of this application, each readout unit includes 3×3 pixels;

[0073] In the first sub-mode, the range of each first region to be merged within an expansion unit is consistent with the range of that expansion unit; the control circuit is configured to: for an expansion unit, output the pixel information of each region to be merged in an interleaved manner in time.

[0074] In this embodiment of the application, synchronously outputting pixel information from different regions to be merged helps to improve the output efficiency of pixel information.

[0075] In one embodiment of this application, each readout unit includes 3×3 pixels;

[0076] In the second sub-mode, the control circuit is specifically configured to: control all connection switches in the pixel array to be disconnected, and for each readout unit, output the pixel information of one region to be merged and five single-pixel readout regions in a staggered manner in time.

[0077] In this embodiment of the application, when each connection switch in the pixel array is in the off state, the pixel information of each region to be merged and the single pixel readout region in the second sub-mode is output, which helps to achieve the highest possible conversion gain and low noise in the output pixel information.

[0078] In one embodiment of this application, in the first pixel merging mode, each readout unit in the pixel array is a region to be merged, and the control circuit is specifically configured to: control all connection switches in the pixel array to be disconnected, and output pixel information of the corresponding region to be merged in each readout unit.

[0079] In this embodiment of the application, when each connection switch in the pixel array is in the off state, the pixel information of each region to be merged in the first pixel merging mode is output, which helps to achieve the highest possible conversion gain and low noise in the output pixel information.

[0080] In one embodiment of this application, in the second pixel merging mode, the information processing unit is configured to: obtain an initial image by performing pixel reordering processing on the pixel information output from the readout line, and convert the initial image into an output image conforming to the Bayer array arrangement.

[0081] In this embodiment of the application, by converting the initial image into an output image that conforms to the Bayer array arrangement, it is more in line with the general product architecture and facilitates subsequent image processing to generate a final image suitable for human eye observation.

[0082] In one embodiment of this application, in the second pixel merging mode, the information processing unit is configured to: input the initial image into a pre-trained image conversion model to obtain the output image;

[0083] The image conversion model is trained based on the following process:

[0084] Acquire a first sample image of the pixel array sensed for a calibrated scene in the first sub-mode or a second sample image sensed for a calibrated scene in the second sub-mode;

[0085] A reference image sensed by a reference pixel array for the calibrated scene is obtained; the reference pixel array conforms to the Bayer array arrangement and its resolution is consistent with the target resolution of the output image;

[0086] Using the first sample image or the second sample image as input data, and combining the reference image corresponding to the sample image, the initial image conversion model is trained to obtain the image conversion model corresponding to the first sub-mode or the second sub-mode.

[0087] In this embodiment of the application, the initial image in the second pixel merging mode is converted into an output image using artificial intelligence methods, which helps to ensure the accuracy of the resulting output image.

[0088] In one embodiment of this application, in the first sub-mode, the information processing unit is configured to: perform color conversion and up-interpolation conversion on the first initial image in the first sub-mode to obtain a first output image that conforms to the Bayer array arrangement and has the same resolution as the third output image; the third output image is: an image obtained by the information processing module based on the pixel information output by the pixel array in the first pixel merging mode.

[0089] In this embodiment of the application, by ensuring that the resolution of the first output image obtained in the first sub-mode is consistent with the resolution of the third output image, the image sensing device can output an output image with the same resolution in the first pixel merging mode, the full pixel mode and the first sub-mode, thereby providing users with a consistent image display effect at different zoom levels and ensuring user experience.

[0090] In one embodiment of this application, in the second sub-mode, the information processing unit is configured to: perform downsampling and mosaic rearrangement processing on the second initial image in the second sub-mode to obtain a second output image that conforms to the Bayer array arrangement and has the same resolution as the third output image; the third output image is: an image obtained by the information processing module based on the pixel information output by the pixel array in the first pixel merging mode.

[0091] In this embodiment of the application, by ensuring that the resolution of the first output image obtained in the second sub-mode is consistent with the resolution of the third output image, the image sensing device can output an output image with the same resolution in the first pixel merging mode, the full pixel mode and the second sub-mode, thereby providing users with a consistent image display effect at different zoom levels and ensuring user experience.

[0092] Secondly, embodiments of this application provide an imaging device, including the image sensing device described in any of the first aspects above.

[0093] Thirdly, embodiments of this application provide an image sensing method applied to a control circuit in an image sensing device. The image sensing device further includes a pixel array, multiple readout lines, and an information processing unit. The pixel array includes multiple readout units arranged in an array, each readout unit including multiple pixels. Each pixel in a readout unit shares a floating diffusion section, and the charge transfer between each pixel and the floating diffusion section is independently controlled. The floating diffusion sections of each two adjacent readout units are connected by a connection switch. The floating diffusion sections of each readout unit are respectively connected to a readout line via a selection switch. The information processing circuit is connected to the multiple readout lines. The control circuit supports multiple pixel merging modes, wherein the pixel merging mode indicates that the pixel information of each region to be merged in the pixel array is output separately by pixel merging, and a region to be merged contains multiple pixels. The regions to be merged under different pixel merging modes are not completely the same. The method includes:

[0094] The control circuit connects the floating diffusion sections within each first readout unit (which contains pixels within the area to be merged) via a connection switch for the region to be merged in the current pixel merging mode. This allows the charge sensed by each pixel within the region to be merged to accumulate and merge in the connected floating diffusion sections. The selection switch between the connected floating diffusion section and the first readout line is then set to the ON state, connecting the first readout line to the connected floating diffusion section. This allows the pixel information generated by merging pixels within the region to be merged to be output to the information processing unit via the first readout line, enabling the information processing unit to generate an output image based on the pixel information output from the readout line in the current pixel merging mode.

[0095] The beneficial effects of the second and third aspects can be referred to the first aspect, and will not be elaborated here. Attached Figure Description

[0096] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0097] Figure 1 is a schematic block diagram of an image sensing device provided in an embodiment of this application;

[0098] Figure 2 is a schematic diagram illustrating the circuit structure of the pixel array in the image sensing device provided in the embodiment of this application;

[0099] Figure 3 is another schematic block diagram of the image sensing device provided in the embodiments of this application;

[0100] Figure 4 is an exemplary block diagram of an imaging device provided in an embodiment of this application;

[0101] Figure 5 is a schematic block diagram of a terminal device as an imaging apparatus provided in an embodiment of this application;

[0102] Figure 6 is a schematic diagram of the software architecture of the terminal device as an imaging apparatus provided in an embodiment of this application;

[0103] Figure 7(a) is a schematic diagram of a pixel array in the related technology;

[0104] Figure 7(b) is a schematic diagram of the internal zoom of the 1× and 2× sensors based on the structure in Figure 7(a);

[0105] Figure 8(a) is a schematic diagram of another pixel array in the related technology;

[0106] Figure 8(b) is a schematic diagram of the 1× and 3× sensor internal zoom based on the structure in Figure 8(a);

[0107] Figure 9 is a circuit diagram of a pixel array in an image sensing device provided in an embodiment of this application;

[0108] Figure 10 is a schematic diagram of the switching states of the switches connected within the circuit structure in Figure 9;

[0109] Figure 11 is a timing diagram of a control signal for the circuit structure in Figure 9;

[0110] Figure 12 is a schematic diagram of the setting method of the first region provided in the embodiment of this application;

[0111] Figure 13 is a schematic diagram of the first region corresponding to each of the four 2×2 regions to be merged within the structure of Figure 12;

[0112] Figure 14 is a schematic diagram of the switching state of the internal connection switch in the structure of Figure 13;

[0113] Figure 15 is a schematic diagram of different regions to be merged within the same group to be merged, provided in an embodiment of this application.

[0114] Figure 16 is a schematic diagram of applying the structure in Figure 15 to a column-level ADC architecture.

[0115] Figure 17 is a schematic diagram of the pixel pattern of the pixel array in the image sensing device provided in the embodiment of this application;

[0116] Figure 18 is a schematic diagram of different readout modes of the image sensing device provided in the embodiments of this application;

[0117] Figure 19 is a schematic diagram of the setting method of the region to be merged and the single pixel readout region in the second sub-mode provided in the embodiment of this application;

[0118] Figure 20 is a schematic diagram of an application scenario where the image sensing device provided in the embodiments of this application is applied to a smartphone;

[0119] Figure 21 is a schematic diagram of the processing chain of the image sensing device provided in the embodiments of this application under different readout modes;

[0120] Figure 22 is a schematic diagram of the first structure of the readout unit provided in the embodiment of this application;

[0121] Figure 23 is a schematic diagram of applying the structure in Figure 22 to the pixel pattern in Figure 17;

[0122] Figure 24 is a circuit diagram of the structure in Figure 22;

[0123] Figures 25(a) to 25(d) are schematic diagrams of the first regions corresponding to each region to be merged contained in the structure in Figure 23 under the first pixel merging mode;

[0124] Figures 26(a) to 26(c) are schematic diagrams of the first region corresponding to each region to be merged contained in the structure in Figure 23 in the second sub-mode;

[0125] Figure 27 is a schematic diagram of a second structure of the readout unit provided in an embodiment of this application;

[0126] Figure 28 is a schematic diagram of applying the structure in Figure 27 to the pixel pattern in Figure 17;

[0127] Figures 29(a) to 29(d) are schematic diagrams of the first regions corresponding to each region to be merged contained in the structure in Figure 27 under the first pixel merging mode;

[0128] Figures 30(a) to 30(c) are schematic diagrams of the first region corresponding to each region to be merged contained in the structure in Figure 27 under the second sub-mode;

[0129] Figure 31 is a schematic diagram of a third structure of the readout unit provided in an embodiment of this application;

[0130] Figure 32 is a schematic diagram of applying the structure in Figure 31 to the pixel pattern in Figure 17;

[0131] Figure 33 is a circuit diagram of the structure in Figure 31;

[0132] Figure 34(a) is a schematic diagram of the first region corresponding to each region to be merged contained in the structure in Figure 31 under the first pixel merging mode;

[0133] Figure 34(b) is a schematic diagram of the readout process of each region to be merged in Figure 34(a);

[0134] Figure 35(a) is a schematic diagram of the first region corresponding to each region to be merged contained in the structure in the first sub-mode of Figure 31;

[0135] Figure 35(b) is a schematic diagram of the readout process of each region to be merged in Figure 35(a);

[0136] Figure 36 is a schematic diagram of the first region corresponding to each region to be merged included in the structure in Figure 31 in the second sub-mode;

[0137] Figure 37 is a schematic diagram of the fourth structure of the readout unit provided in the embodiments of this application;

[0138] Figure 38 is a schematic diagram of applying the structure in Figure 37 to the pixel pattern in Figure 17;

[0139] Figure 39 is a circuit diagram of the structure in Figure 37;

[0140] Figure 40(a) is a schematic diagram of the first region corresponding to each region to be merged contained in the structure in the first sub-mode of Figure 37;

[0141] Figure 40(b) is a schematic diagram of the process of reading out pixel information for each region to be merged in Figure 40(a);

[0142] Figure 41(a) is a first schematic diagram of the readout process of the structure in Figure 37 in the second sub-mode;

[0143] Figure 41(b) is a second schematic diagram of the readout process of the structure in Figure 37 in the second sub-mode;

[0144] Figure 41(c) is a third schematic diagram of the readout process of the structure in Figure 37 in the second sub-mode;

[0145] Figure 41(d) is a fourth schematic diagram of the readout process of the structure in Figure 37 in the second sub-mode. Detailed Implementation

[0146] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0147] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. For example, "first instruction" and "second instruction" are used to distinguish different user instructions and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0148] It should be noted that, in this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0149] This application provides specific embodiments and examples of an image sensing device, an imaging device including the image sensing device, and a pixel readout method, in order to overcome the technical bottleneck in the field that it is difficult to achieve low-noise, high-definition imaging at multiple zoom magnifications because it is difficult to combine multiple-size charge domains in the same image sensor.

[0150] To facilitate understanding of the technical solutions in the subsequent embodiments of this application, the basic architecture and principle of the image sensing device and imaging device involved in this application will be described by way of example below:

[0151] Image sensing devices specifically refer to devices used to sense external light signals and convert them into image signals that characterize the external scene through photoelectric conversion technology.

[0152] Figure 1 shows an exemplary block diagram of an image sensing device 10, including a control circuit 11, a pixel array 12, and a readout circuit 13. The control circuit 11 generates control signals to be sent to the pixel array 12. Under the control of these signals, the pixel array 12 converts sensed light into pixel information and outputs the generated pixel information to the readout line. The readout circuit 13 is connected to the pixel array via the readout line and reads the pixel information output by the pixel array 12, transmitting it to other peripheral devices for subsequent data processing. For ease of understanding, the working principles of the control circuit 11, pixel array 12, and readout circuit 13 are further explained in detail below:

[0153] The pixel array 12 includes multiple pixels arranged in an array, each of which can act as an independent unit to sense external light signals and convert the sensed light signals into electrical signals. Each pixel can be specifically constructed using a light-receiving device, such as a photodiode (PD). Figure 2 shows a schematic diagram of the circuit structure design of a single pixel within the pixel array 12. It should be understood that in practical applications, other circuit designs different from those shown in Figure 2 can be used depending on specific requirements. In the schematic of Figure 2, a single pixel is constructed from a photodiode PD, which is connected to a floating diffuser FD via a transmission switch TX. The floating diffuser FD is connected to an amplifier AMP, which is connected to a power supply voltage V. dd It is connected to the output line OUT via the selector switch SEL. Furthermore, the floating diffuser FD is also connected to the reset voltage V via the reset switch RST. rst .

[0154] In this context, the floating diffuser FD is not a physical device, but rather refers to a circuit structure that contains parasitic capacitance and can temporarily store the photogenerated charge generated by the photodiode PD and convert the stored photogenerated charge into a voltage signal. Specifically, in the structure illustrated in Figure 2, the floating diffuser FD can be understood as the circuit structure within the dashed box, including the circuit structure between the transmission switch TX, the amplifier AMP, and the reset switch RST.

[0155] For the circuit design described above, the control circuit 11 can control the actual working state of a single pixel by controlling the on / off states of the transmission switch TX, selection switch SEL, and reset switch RST. For example, the transmission switch TX can be connected to the transmission signal of the control circuit 11 via a transmission signal line. When the transmission signal is high, the transmission switch TX is on; when the transmission signal is low, the transmission switch TX is off. Similarly, the selection switch RST can be connected to the selection signal of the control circuit 11 via a selection signal line. When the selection signal is high, the selection switch RST is on; when the selection signal is low, the selection switch RST is off. Likewise, the reset switch RST can be connected to the reset signal of the control circuit 11 via a reset signal line. When the reset signal is high, the reset switch RST is on; when the reset signal is low, the reset switch RST is off. In practical applications, when multiple transmission switches TX are arranged within the pixel array circuit design, each transmission switch TX can be connected to a different transmission signal via a different transmission signal line, so that the on / off state of each transmission switch TX can be independently controlled by the control circuit 11. The selection switch SEL and reset switch RST operate similarly.

[0156] Taking the structure in Figure 2 as an example, during the actual operation of the pixel array 12, the photodiode PD can generate photocharge by sensing light. By turning on the transmission switch TX, the photocharge generated by the photodiode PD can be transferred to the floating diffuser FD to form a voltage signal. By turning on the selection switch SEL, the voltage signal generated by the floating diffuser FD can be amplified by the amplifier AMP and output from the readout line OUT. In addition, before performing a new signal sampling, it is generally necessary to turn on the reset switch RST to reset the photodiode PD to prevent residual charge from the previous sampling process from affecting the output data.

[0157] It should be noted that FIG2 only shows a schematic diagram of the circuit structure design of a single pixel of the pixel array 12, and does not show the connection relationship between different pixels, or the shared relationship of different pixels for components (e.g., the shared relationship of different pixels for the floating diffuser FD and amplifier AMP). FIG2 is provided only to illustrate the basic working principle of the image sensing device provided in this application, so as to understand the specific scheme of the subsequent embodiments of this application, and does not constitute a specific limitation on the circuit structure design of the pixel array 12 in the image sensing device provided in this application.

[0158] This application provides an image sensing device. Referring to the circuit structure design for a single pixel in Figure 2, multiple pixels are arranged in an array to form a pixel array 12. Specifically, the image sensing device provided in this application introduces a readout unit structure within the pixel array 12 (refer to the schematic diagram in Figure 9 below). Referring to the schematic diagram in Figure 2, the light receiving devices (PDs) of multiple pixels are connected to the same floating diffuser FD via transmission switches TX to form a single readout unit. This allows each pixel within a single readout unit to share the floating diffuser FD, reset switch RST, amplifier AMP, and connection switch SEL. Different readout units require their own floating diffuser FD, reset switch RST, amplifier AMP, and connection switch SEL. Furthermore, this application also introduces a connection switch SW structure in the pixel array 12. The connection switch SW connects the floating diffusers FD in adjacent readout units. The connection switch SW controls whether adjacent readout units use their respective floating diffusers FD independently or share the same floating diffuser FD, thereby enabling switching of the number of pixels sharing the same floating diffuser FD. The specific settings and functions of the readout unit and connection switch will be discussed in detail later.

[0159] In practical applications, the control circuit 11 shown in Figure 1 may actually consist of multiple sub-modules with different functions. Figure 3 provides an example where the control circuit 11 specifically includes a timing controller 111, a switch controller 112, and a row driver 113. The row driver 113 generates transmission signals to independently control each transmission switch within the pixel array 12, generates selection signals to independently control each selection switch within the pixel array 12, and generates reset signals to independently control each reset switch within the pixel array 12. The switch controller 112 generates connection signals to independently control each connection switch. The timing controller 111 controls the operating timing of the row driver 113 and the switch controller 112. Since the row driver 113 and the switch controller 112 generally follow the same operating timing, they can be connected to the same timing controller. In other embodiments, they can also be connected to different timing controllers for independent control.

[0160] Furthermore, for image sensing devices used to acquire color images, the electrical signals output by individual pixels can only represent light intensity information, not color information. Therefore, in order to acquire pixel information of different colors, a filter structure needs to be configured for each pixel in the pixel array 12, so that the pixel can be used to sense specific colors of light. According to typical product architectures, the filters generally use three colors: blue, red, and green. With the filter configured, the photodiode (PD) within the pixel performs photoelectric conversion on the filtered specific color of light, thereby outputting the corresponding color of pixel information.

[0161] The readout circuit 13 can read the pixel information output by each pixel in the pixel array 12 through the readout line OUT, and transmit the pixel information to other peripheral devices so that they can perform subsequent image processing to generate the final image information. In actual products, in addition to transmitting the pixel information output by the pixel array 12 to other peripheral devices through the readout circuit 13, various functional devices can also be integrated into the readout circuit 13, so that the readout circuit 13 can perform preliminary processing on the raw signal output by the pixel array 12 and transmit the processed data to the back-end image processing device.

[0162] In one example, an analog-to-digital converter (ADC), a buffer, and a pixel reordering module can be integrated into the readout circuit 13. The ADC converts the voltage signal output from the pixel array 12 into a digital signal for subsequent digital processing. The buffer temporarily stores the data output from the pixel array 12 for later processing such as pixel reordering. The pixel reordering module reorders the pixel information output from the pixel array 12. As mentioned earlier, each pixel in the pixel array 12 can act as an independent unit to convert external light signals into electrical signals. However, in practical applications, the output order of the pixel information generated by each pixel in the pixel array 12 is not fixed. Therefore, the pixel reordering module needs to rearrange the output pixel information according to the actual physical position of the pixels in the pixel array 12.

[0163] It should be understood that the architecture in Figure 1 does not constitute a specific limitation on the actual form of the product. In actual applications, the image sensing device 10 may contain more components than shown in Figure 1, and the aforementioned control circuit 11, pixel array 12, and readout circuit 13 may be deployed in different physical components for combined use. For example, in an actual product form, it is sufficient to ensure that the pixel array is integrated into the image sensor. The control circuit and readout circuit can be integrated either in the area surrounding the pixel array within the image sensor or on other chips used in combination with the image sensor. For example, the control circuit and readout circuit can be integrated inside the processor of the electronic device that houses the image sensor.

[0164] An imaging device specifically refers to a device used to capture images or videos based on external light signals. For example, an imaging device can be a digital camera or a camcorder.

[0165] Figure 4 shows an exemplary block diagram of the imaging device 20, which includes an image sensing device 10 and an image processor 14. The image processor 14 is used to perform image processing on the signal output by the image sensing device 10 to make the final image as close as possible to the effect of the human eye seeing the real scene. In one example, the image processor can specifically be an ISP (Image Signal Processor) chip, which can be configured with a defect pixel correction algorithm, a color interpolation algorithm, a color correction algorithm, etc.

[0166] In one example, the imaging device can also be a terminal device with imaging capabilities, such as a smartphone, tablet computer, or laptop computer. Figure 5 shows a schematic block diagram of a terminal device, which includes a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 130, a wireless communication module 140, a sensor module 150, a camera 162, and a display screen 161.

[0167] The sensor module 150 may include an image sensor 150A and a touch sensor 150B, etc.

[0168] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0169] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0170] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 150B, charger, flash, camera 162, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 150B through the I2C interface, enabling the processor 110 and the touch sensor 150B to communicate through the I2C bus interface, thereby realizing the touch function of the terminal device.

[0171] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 161 and the camera 162. The MIPI interface includes a Camera Serial Interface (CSI) and a Display Serial Interface (DSI). In some embodiments, the processor 110 and the camera 162 communicate via the CSI interface to enable the terminal device's shooting function. The processor 110 and the display screen 161 communicate via the DSI interface to enable the terminal device's display function.

[0172] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a structural limitation on the terminal device. In other embodiments, the terminal device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0173] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 130, wireless communication module 140, modem processor, and baseband processor.

[0174] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0175] The mobile communication module 130 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the first electronic device. In some embodiments, the mobile communication module 130 can be used to transmit call data between two electronic devices. For example, when acting as the called party device, it can obtain downlink audio stream data from the calling party device and transmit uplink audio stream data to the calling party device.

[0176] The wireless communication module 140 can provide solutions for wireless communication applications in electronic devices, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and infrared (IR) technologies.

[0177] For scenarios where the imaging device is the terminal device, the image processor specifically refers to the ISP chip within the processor 110. The control circuit and readout circuit can be integrated either on the logic circuit surrounding the pixel array within the image sensor 150A or within the processor 110.

[0178] The terminal device can achieve photo and video recording functions through the processor 110, camera 162, image sensor 150A, and display screen 161. For example, when taking a photo, the user opens the shutter, and light is transmitted through the camera 162 to the image sensor 150A. The image sensor 150A converts the light signal into an electrical signal and transmits the generated electrical signal to the processor 110. The ISP chip of the processor 110 processes the electrical signal, converts it into a visible image, and finally displays the image on the display screen 161. In some embodiments, the terminal device may include N cameras 162, where N is a positive integer greater than 0, and the terminal device may also include M displays 161, where M is a positive integer greater than 0, and M is less than or equal to N.

[0179] The software system of a terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of the terminal device.

[0180] Figure 6 is a software structure block diagram of a terminal device according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system library of the Android runtime, and the kernel layer.

[0181] The application layer can include a series of application packages. As shown in Figure 6, an application package can include applications such as a camera and a gallery.

[0182] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0183] As shown in Figure 6, the application framework layer may include a window manager, a content provider, a resource manager, etc.

[0184] The window manager is used to manage windowed applications. It can obtain the screen size, determine if a status bar is present, lock the screen, allow screen touch, drag the screen, and capture the screen, among other things.

[0185] Content providers are used to store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, and more.

[0186] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0187] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0188] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0189] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0190] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), and media libraries.

[0191] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0192] The media library supports playback and recording of various commonly used audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as: Moving Pictures Experts Group-4 (MPEG-4), H.264, Moving Picture Experts Group Audio Layer III (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR), Joint Photographic Experts Group (JPG), and Portable Network Graphics (PNG).

[0193] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0194] A 2D graphics engine is a drawing engine for 2D drawing.

[0195] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, and sensor drivers.

[0196] As mentioned earlier, in order to achieve multi-focal-length imaging and maintain high resolution with a single camera, current smartphones have generally introduced in-sensor focusing functionality. Furthermore, with in-sensor focusing, smartphones need to perform charge domain merging on pixel signals to ensure the signal-to-noise ratio (SNR) of the output signal at different zoom levels.

[0197] The image sensing device, imaging device, and pixel readout method provided in this application can specifically achieve low-noise, high-definition imaging at multiple zoom levels by using the same image sensing device to accommodate the merging of charge domains of multiple sizes, thus providing users with diverse imaging options.

[0198] To facilitate understanding, the following is a brief explanation of the basic concepts of the sensor's internal focusing function and charge domain merging:

[0199] In simple terms, sensor-internal zoom involves directly processing data within the sensor to change the size of pixels in an image, thus achieving a zoom effect. Charge domain combining is a special pixel readout method that combines the photogenerated charges sensed by multiple pixels into a single electrical signal. Common charge domain combining methods involve combining the photogenerated charges sensed by 2×2 adjacent pixels or 3×3 adjacent pixels; the 2×2 and 3×3 dimensions are generally referred to as the charge domain combining size.

[0200] The common magnification of the internal focusing function of the sensor is 1x, 2x or 3x. Currently, there are no products that can achieve 2x and 3x magnification at the same time. Therefore, products that support 1x and 2x magnification and products that support 1x and 3x magnification will be described separately as examples.

[0201] Referring to Figures 7(a) and 7(b), in products supporting 1× and 2× magnification, every 2×2 adjacent pixels in the pixel array are combined into a pixel group, and the four pixels in the same pixel group share the same floating diffuser. Thus, during the operation of the pixel array, the photogenerated charges generated by the photodiodes in the four pixels of the same pixel group can be merged and accumulated within the same floating diffuser. In this case, the electrical signal read from the floating diffuser by the readout circuit represents a pixel information item generated by the merging of the four pixels.

[0202] At 1x magnification, pixel information from all pixels within the entire photosensitive surface of the pixel array needs to be output. During the output process, the photogenerated charges of 2x2 pixels within each pixel group are combined and output as a single pixel information (i.e., 2x2 charge domain merging), essentially treating 2x2 pixels as a single pixel. At 2x magnification, pixel information from pixels within a specific area of ​​the photosensitive surface of the pixel array (typically a quarter area, i.e., half the height and half the width) can be output, and the pixel information generated for each pixel is output separately. In practical applications, the sensor's internal focusing function provides users with greater imaging flexibility. For example, 1x magnification can be used in low-light environments to suppress image noise and obtain higher-quality images; while 2x magnification can be used in well-lit environments to obtain magnified views containing more detailed information.

[0203] Referring to Figures 8(a) and 8(b), in products supporting 1× and 3× magnification, every 3×3 adjacent pixels in the pixel array are combined into a pixel group, and the nine pixels in the same pixel group share the same floating diffuser. Thus, during the operation of the pixel array, the photogenerated charges generated by the photodiodes in the nine pixels of the same pixel group can be merged and accumulated within the same floating diffuser. In this case, the electrical signal read from the floating diffuser by the readout circuit represents one pixel information generated by the merging of the nine pixels.

[0204] At 1x magnification, the pixel information of all pixels within the entire photosensitive surface of the pixel array needs to be output. During the output process, the photogenerated charges of 3×3 pixels in each pixel group are combined and output as a single pixel information (i.e., 3×3 charge domain merging), which is equivalent to using 3×3 pixels as a single pixel. At 3x magnification, the pixel information of pixels within a portion of the photosensitive surface of the pixel array (usually a 1 / 9 area, i.e., a region that is 1 / 3 high and 1 / 3 wide) can be output, and the pixel information generated by each pixel is output separately during the output process.

[0205] Those skilled in the art will understand that, since the photogenerated charges generated by the light-receiving devices within a pixel need to be converted into voltage signals within the floating diffuser, if charge domain merging is to be performed on multiple pixels, it is necessary to ensure that the light-receiving devices of these multiple pixels are all connected to the same floating diffuser in order to merge the photogenerated charges generated by each of the multiple pixels together and output a voltage signal. This voltage signal is the pixel information generated by merging the multiple pixels. Regarding the design in Figure 7(a), because its inherent circuit structure requires each 2×2 adjacent pixels to share a single floating diffuser, it is impossible to connect the light-receiving devices of each 3×3 adjacent pixels to the same floating diffuser. Therefore, 3×3 charge domain merging is not possible, thus preventing 3×x zoom within the sensor. As for the design in Figure 8(a), since its inherent circuit structure is that every 3×3 adjacent pixels share a floating diffuser, if we want to merge every 2×2 adjacent pixels into one pixel for output, some of the 2×2 adjacent pixels that need to be merged will actually be located in different pixel groups, and their light receiving devices will be connected to different floating diffusers. Therefore, 2×2 charge domain merging cannot be performed, and thus 2×x internal zoom of the sensor cannot be achieved.

[0206] Furthermore, some image sensor products incorporate analog pixel merging technology to merge pixel information in the analog domain, as an alternative to charge domain merging, enabling the product to support different zoom ratios. For example, an image sensor product might use analog domain merging to achieve pixel merging of one size to provide image display at a first zoom ratio, and use charge domain merging to achieve pixel merging of another size to provide image display at a second zoom ratio, thus offering users diverse zoom ratio options. However, compared to charge domain merging, analog domain merging results in a lower signal-to-noise ratio (SNR) in the output, leading to insufficient image quality at the first zoom ratio. Therefore, achieving multi-zoom ratio image display through analog domain merging cannot guarantee the final image's SNR and sharpness at each zoom ratio, thus failing to meet user needs.

[0207] In summary, ISZ functionality requires pixel charge domain merging technology to process data within the image sensor to achieve different zoom ratios, ensuring the signal-to-noise ratio and sharpness of the final image at each zoom level. Therefore, to enable image sensor products to offer diverse zoom ratio options and guarantee low-noise, high-resolution imaging at each zoom ratio, a product design that can accommodate multi-size charge domain merging within the same image sensor is urgently needed.

[0208] Furthermore, to avoid gaps in zoom ratios and image sharpness degradation that could negatively impact user experience, 2x and 3x zoom ratios remain the mainstream choices. Therefore, if a single product structure could be compatible with the merging of 2x2 and 3x3 charge domains to provide a choice of 1x, 2x, and 3x zoom ratios, it would be of great significance to the future design direction of image sensor products.

[0209] In view of this, the present application provides an image sensing device that is compatible with multi-size charge domain merging. Its basic architecture can be referred to Figure 1, and it will be described in detail below.

[0210] The image sensing device provided in this application embodiment specifically includes a pixel array 12, a readout line connected to the pixel array 12, and a control circuit 11 for controlling the pixel array 12.

[0211] The pixel array 12 includes multiple readout units arranged in an array, each readout unit including multiple pixels; each pixel in a readout unit shares a floating diffuser, and the charge transfer between each pixel and the floating diffuser is independently controlled.

[0212] From the perspective of pixel circuit design, in order for all pixels within the same readout unit to share the same floating diffuser, the light-receiving devices of each pixel within the same readout unit need to be connected to the same floating diffuser, and the light-receiving devices of pixels in different readout units need to be connected to different floating diffusers. The specific size of the readout unit can be set according to actual needs. In subsequent embodiments of this application, specific implementations with readout unit sizes of 2×4 (denoted as x×y, meaning a readout unit contains x rows × y columns of adjacent pixels), 2×2, 1×3, and 3×3 will be given. However, the actual selectable size is not limited to these; for example, the size of the readout unit can also be set to 1×2, 2×1, 4×2, etc.

[0213] In this embodiment, the floating diffusers between each pair of adjacent readout units are connected by a connection switch. The on / off state of the connection switch is controlled by the control circuit 11. When the connection switch between two adjacent readout units is off, the floating diffusers of the two readout units are isolated from each other, so that the photogenerated charges sensed by the pixels in the two readout units accumulate in their respective floating diffusers. When the connection switch between two adjacent readout units is on, the floating diffusers of the two readout units are connected to each other, and the connected floating diffusers are shared between the two readout units, so that the photogenerated charges sensed by the pixels in the two readout units can be merged and accumulated in the same floating diffuser.

[0214] In this embodiment, the control circuit 11 can independently control the on / off state of each connection switch. In one example, each connection switch can be connected to a different connection signal of the control circuit 11 via different connection signal lines. When the connection signal is high, the corresponding connection switch is turned on; when the connection signal is low, the corresponding connection switch is turned off.

[0215] In addition, each floating diffuser in the readout unit is connected to a readout line via a selection switch. The on / off state of the selection switch is controlled by the control circuit 11. The photogenerated charge sensed by the pixel in the readout unit accumulates in the floating diffuser to form a voltage signal. When the selection switch between the floating diffuser and the readout line is turned on, the formed voltage signal can be output to the subsequent readout circuit through the readout line.

[0216] The total number of readout lines in an image sensing device, and the correspondence between readout units and readout lines, are related to the specific product architecture. Generally, the readout lines output the readout voltage signal to the ADC module for analog-to-digital conversion. Assuming that the pixel array 12 includes X rows and Y columns of readout units, when using a column-level ADC, each column of readout units shares the same ADC module for analog-to-digital conversion. In this architecture, the number of readout lines is Y. The floating diffuser portions of each readout unit in the same column are connected to the same readout line via a selection switch, and then connected to the shared ADC module of that column via the readout line. The floating diffuser portions of readout units in different columns are connected to different readout lines. When using a pixel-level ADC, each readout unit is configured with its own ADC module for analog-to-digital conversion. In this architecture, the number of readout lines is X×Y. The floating diffuser portions of each readout unit are connected to different readout lines via selection switches, and then connected to their respective ADC modules via the readout lines.

[0217] Figure 9 shows a schematic diagram of the internal circuit structure of the pixel array 12 when the readout unit size is 2×2. It illustrates four adjacent readout units (readout units 1 to 4) arranged in two rows and two columns, and shows the specific arrangement of photodiodes PD (PD1 to PD16), transmission switches TX (TX1 to TX16), reset switches RST (RST1 to RST4), amplifier AMP, selection switches AEL (SEL1 to SEL4), readout lines OUT (OUT1 to OUT2), and connection switches SW (SW1 to SW8). It can be seen that each readout unit includes 2×2 pixels, each pixel has a photodiode PD, and each photodiode PD within the same readout unit is connected to the same floating diffuser FD via the transmission switch TX. Each readout unit has its own floating diffuser FD, and the floating diffuser FD of each readout unit is connected to a readout line OUT via the amplifier AMP and the selection switch SEL. In addition, the floating diffusion section (FD) of each readout unit is also connected to a reset voltage via a reset switch (RST).

[0218] In the structure shown in Figure 9, the floating diffusion section FD in a readout unit can be specifically understood as the circuit structure within the dashed box in the figure, including the line structure between each transmission switch TX and the amplifier AMP, between the transmission switch TX and the reset switch RST, and between the transmission switch TX and the connection switch SW.

[0219] In the structure shown in Figure 9, the floating diffuser FD of each readout unit is connected to the floating diffuser FDs of its four adjacent readout units (top, bottom, left, and right) via connecting switches SW. It can be seen that when the connecting switch SW between two adjacent readout units is open, the floating diffuser FDs within those two readout units are isolated from each other, and the photogenerated charges accumulated in the two floating diffuser FDs do not interfere with each other. However, when the connecting switch SW between two adjacent readout units is closed, the floating diffuser FDs of those two readout units connect to form a single unit. This is equivalent to all eight photodiodes PD in the two readout units being connected to the same floating diffuser FD, allowing the photogenerated charges sensed by these eight photodiodes PD to be combined and accumulated within the same floating diffuser.

[0220] Based on the above design, charge domain merging can be achieved for pixels within the same readout unit even when the connection switch SW is in the off state. Alternatively, by controlling the on / off state of the connection switch SW, the floating diffusers FD of two or more adjacent readout units can be connected to each other, allowing pixels within these two or more readout units to share the same floating diffuser, thereby enabling charge domain merging across readout units. Based on this principle, the image sensing device provided in this application embodiment can theoretically support charge domain merging of any size. Therefore, the specific size of the charge domain merging to be implemented by the image sensing device can be configured according to actual needs. For example, two charge domain merging modes can be configured for the image sensing device: 2×2 and 3×3. Alternatively, 4×4 and 5×5 charge domain merging modes can be further configured based on these two charge domain merging modes.

[0221] In this embodiment, the working mode of outputting pixel signals within the pixel array 12 through charge domain merging is referred to as the pixel merging mode. In this embodiment, the control circuit 11 can support multiple pixel merging modes. Under each pixel merging mode, the entire photosensitive area to be output within the pixel array 12 is divided into multiple mergeable areas. Each mergeable area contains multiple pixels, and the control circuit 11 needs to output the pixel information of each mergeable area under the current pixel merging mode through charge domain merging. Furthermore, the mergeable areas within the pixel array 12 are not entirely the same under different pixel merging modes. For example, a 2×2 or 3×3 pixel merging mode can be configured for the control circuit 11. In the 2×2 pixel merging mode, every 2×2 adjacent pixels within the photosensitive area to be output form a mergeable area; in the 3×3 pixel merging mode, every 3×3 adjacent pixels within the photosensitive area to be output form a mergeable area.

[0222] In this embodiment, the control circuit 11 can specifically implement charge domain merging in the current pixel merging mode through the following mechanism, and output the pixel information formed after merging:

[0223] For the area to be merged in the current pixel merging mode, if the pixels in the area to be merged belong to different readout units, the control connection switch connects the floating diffusion sections in each readout unit (first readout unit) containing the pixels in the area to be merged, so that the charge sensed by each pixel in the area to be merged is merged and accumulated in the connected floating diffusion section; and the selection switch between the connected floating diffusion section and the first readout line is set to the on state, and the first readout line is connected to the connected floating diffusion section, so that the pixel information generated by merging each pixel in the area to be merged is output through the first readout line.

[0224] The above mechanism specifically describes the process of merging the charge domains of pixels within a region to be merged and outputting the merged pixel information. In practical scenarios, in a pixel mode, the pixel array 12 typically includes multiple regions to be merged. It is necessary to perform charge domain merging on pixels within some or all of these regions and output the merged pixel information separately. Since the process of merging the charge domains for pixels within each region to be merged is essentially the same, each region can be processed separately using the same mechanism.

[0225] Specifically, when pixels within a region to be merged belong to several different readout units (first readout units), the control circuit 11 connects the floating diffusion sections of these different readout units via a control connection switch, allowing the floating diffusion sections of these different readout units to accumulate charge as a whole. Based on this, the photogenerated charges sensed by each pixel within the region to be merged can be merged and accumulated in the connected floating diffusion section, forming a voltage signal (pixel information generated by merging pixels within the region to be merged) in the connected floating diffusion section.

[0226] As mentioned earlier, each readout unit's floating diffusion section is connected to a readout line via a selection switch. Therefore, after connecting the floating diffusion sections of several readout units, one or more different readout lines will be connected to the connected floating diffusion section. In this embodiment, the control circuit 11 can turn on the selection switch between one of the readout lines (the first readout line) and the connected floating diffusion section, so that the voltage signal formed in the floating diffusion section can be output through this readout line. Thus, the output of pixel information within the region to be merged is completed through charge domain merging. The pixel information can be transmitted to the readout circuit 13 via the readout line, and then from the readout circuit 13 to the peripheral devices for further processing.

[0227] In practical applications, for some regions to be merged, there may be situations where the pixels within the region belong to only one readout unit. In this case, since the pixels within the region to be merged are already connected to the same floating diffuser, there is no need to turn on the connection switch. The pixels within the region to be merged can be directly merged and accumulated within the floating diffuser of their respective readout units, and the voltage signal formed by the floating diffuser in the readout unit is output through the readout line.

[0228] Assuming that the structure in Figure 9 is configured with two pixel merging modes, namely 2×2 pixel merging and 3×3 pixel merging, the working mechanism of the structure in Figure 9 under these two pixel merging modes will be illustrated separately:

[0229] In the 2×2 pixel merging mode, the 2×2 pixels in each readout unit form a region to be merged. Therefore, charge domain merging can be performed with all connection switches SW open. Taking a region to be merged consisting of four pixels, PD1 to PD4, as an example, by setting the transmission switches TX1 to TX4 to the on state, the photogenerated charges induced by PD1 to PD4 can be merged and accumulated in the floating diffusion section FD of readout unit 1 to form a voltage signal. By turning on the selection switch SEL1, the pixel information generated by the merging of PD1 to PD4 can be read from the readout line OUT1.

[0230] In the 3×3 pixel merging mode, taking a region to be merged consisting of nine pixels, namely PD1 to PD5 and PD7, PD9, PD10, and PD13, as an example, these nine pixels belong to four different readout units, namely readout units 1 to 4. Therefore, the floating diffuser sections (FD) of these four readout units need to be connected to achieve charge domain merging of the pixels in the region to be merged. In one example, referring to the schematic diagram in Figure 10, the control circuit 11 turns on the connecting switches SW1, SW2, and SW5, so that the floating diffuser sections of readout units 1 to 4 are connected to form a whole, and TX1 to TX5, as well as TX7, TX9, TX10, and TX13 are turned on. Thus, the photogenerated charges induced by PD1 to PD5, as well as PD7, PD9, PD10, and PD13 can be merged and accumulated in the connected floating diffuser section (FD), and a voltage signal is formed in the connected floating diffuser section (FD). In the schematic diagram of Figure 10, the floating diffuser FD of readout unit 1 and readout unit 2 are both connected to readout line OUT1, while readout unit 2 and readout unit 4 are both connected to readout line OUT2. With this structure, either the selection switches SEL1 and / or SEL3 can be turned on by the control circuit 11, allowing the pixel information generated by the merging of the nine photodiodes PD to be output through readout line OUT1, or the selection switches SEL2 and / or SEL4 can be turned on by the control circuit 11, allowing the pixel information generated by the merging of the nine photodiodes PD to be output through readout line OUT2.

[0231] In one example, to ensure the smooth output of pixel information from the region to be merged via the readout line, the control circuit 11 can be configured to control the on / off states of the transmission switch TX, connection switch SW, selection switch SEL, and reset switch RST according to the control timing sequence shown in Figure 11. Figure 11 uses nine pixels—PD1 to PD5, PD7, PD9, PD10, and PD13—as a region to be merged in Figure 9, and illustrates the process by which the control circuit 11 outputs the pixel information of this region through the floating diffusion section FD connected to readout units 1 to 4. The figure shows... <txi>To characterize the transmission signal of the access transmission switch TXI, through <rsti>To characterize the reset signal of the access reset switch RSTi, through <seli>To characterize the selection signal of the access selection switch SELi, through <swi>This is used to characterize the transmission signal of the access connection switch SWi.

[0232] Referring to Figure 11, to output pixel information of the nine pixels to be merged, namely PD1 to PD5, and PD7, PD9, PD10 and PD13, the control circuit 11 first sets the reset signal... <rst1>to <rst4>The signal is set to high level, which resets the floating diffusion section FD stage ① of readout units 1 to 4 to prevent residual charge from interfering with the subsequent charge accumulation process; then, the control circuit 11 selects the signal. <sel1>Set to high level to turn on selector switch SEL1 and connect the signal. <sw1> ， <sw2>and <sw5>Set to high level, turning on the connecting switches SW1, SW2, and SW5, connecting the floating diffuser sections FD of readout unit 1 to readout unit 4 to form a whole; in stage ②, the control circuit 11 transmits the signal. <tx1>to <tx5>,as well as <tx7> ， <tx9> ， <tx10>and <tx11>The signal is set to high level, which turns on the transmission switches TX1 to TX5, as well as TX7, TX9, TX10 and TX11. This allows the photogenerated charges induced by photodiodes PD1 to PD5, as well as PD7, PD9, PD10 and PD13, to be combined, accumulated and converted into voltage signals in the connected floating diffuser FD. In stage ③, the voltage signal generated by the connected floating diffuser FD is output through the readout line OUT1. The process of outputting pixel information in the above-mentioned area to be merged is now complete.

[0233] Furthermore, the method illustrated in Figure 10 is not the only way to connect the floating diffusion sections FD within readout units 1 to 4. Other connection switch SW control schemes can also connect the floating diffusion sections FD of the four readout units. For example, the floating diffusion sections FD of the four readout units can be connected by turning on connection switches SW2, SW4, and SW5, or by turning on connection switches SW1, SW2, SW4, and SW5.

[0234] As can be seen from the previous explanation, the number of connected switches SWs required to connect a specific number of floating diffuser sections (FDs) of readout units may not be fixed. In a preferred embodiment, when pixels within a region to be merged belong to several different readout units, the number of connected switches SWs set to the on state to connect the floating diffuser sections (FDs) of these readout units should be minimized when connecting the floating diffuser sections (FDs) of these readout units into a single unit. This design is based on the fact that when connecting several floating diffuser sections (FDs) into a single unit by turning on the connected switches SWs, the connected switches SWs themselves also provide a portion of parasitic capacitance. In other words, the parasitic capacitance of the connected switches SWs in the on state constitutes a part of the total capacitance of the connected floating diffuser sections (FDs). The conversion gain (specifically, the ratio of accumulated charge to the voltage value) of the photogenerated charge accumulated in the floating diffuser FD is inversely proportional to the capacitance of the floating diffuser FD. A larger capacitance in the floating diffuser FD results in a smaller conversion gain, which is detrimental to ensuring the signal-to-noise ratio (SNR) of the voltage signal output from the readout line OUT. Therefore, by minimizing the number of connection switches SW used to connect the floating diffusers FD in each first readout unit, the capacitance of the connected floating diffusers FD can be minimized, thus helping to ensure a low SNR signal output.

[0235] The above description, combined with the specific circuit design of pixel array 12, exemplifies the process of outputting pixel information within a region to be merged through charge domain merging. As mentioned earlier, in practical applications, pixel array 12 typically includes multiple regions to be merged, requiring the output of pixel information for each region through charge domain merging. In one possible implementation, the pixel information of each region to be merged can be output sequentially according to a certain order, based on the mechanism described in this embodiment. In another possible implementation, pixel information of several regions to be merged can be output synchronously through different readout lines, achieving the output of pixel information within each region to be merged in a parallel + serial manner. It should be noted that for synchronous output of pixel information from two regions to be merged, the floating diffuser connected to the photodiodes of each pixel in one region to be merged must be isolated from the floating diffuser connected to the photodiodes of each pixel in the other region to be merged, so that the photogenerated charge induced by the pixels in the two regions to be merged can accumulate separately in different floating diffusers.

[0236] In summary, the image sensing device provided in this application embodiment arranges multiple readout units in an array within a pixel array, such that each pixel within a readout unit shares a floating diffuser, and the floating diffusers of every two adjacent readout units are connected by a connection switch. Based on this structural design, the control circuit can control the connection switch to connect the floating diffusers in two or more adjacent readout units into a whole, allowing the charges sensed by each pixel in the two or more readout units to be merged and accumulated within the same floating diffuser, thereby achieving charge domain merging between pixels across readout units.

[0237] Based on the above mechanism, the image sensing device provided in this application embodiment can theoretically support charge domain merging of any size. Therefore, in practical applications, various pixel merging modes of different sizes can be configured according to specific needs. For a region to be merged under the current pixel merging mode, the control circuit connects the floating diffusion sections of each readout unit containing pixels in the region to be merged by controlling the connection switch. This allows the charges sensed by each pixel in the region to be merged to be merged and accumulated in the connected floating diffusion section. The merged and accumulated charges can be converted into electrical signals in the connected floating diffusion section. By turning on the selection switch between any one of the readout lines connected to the connected floating diffusion section and the connected floating diffusion section, the control circuit can output the electrical signal formed by the connected floating diffusion section from that readout line, thereby realizing the output of pixel information in the region to be merged by charge domain merging.

[0238] As can be seen from the above, based on the embodiments of this application, it is possible to use the same image sensing device to be compatible with the merging of multiple charge domains of different sizes. When the image sensing device provided based on the embodiments of this application implements the ISZ function, compared with the technical solution of achieving multiple zoom magnifications through analog domain merging of pixels, the pixel information output by the image sensing device in pixel merging mode has a higher signal-to-noise ratio, thereby ensuring low-noise, high-definition imaging at various zoom magnifications, which helps to provide users with diverse zoom magnification options while ensuring user experience.

[0239] In one example, the image sensing device provided in this application embodiment can be configured with 2×2 and 3×3 pixel binning modes, combined with a full-pixel mode that outputs pixel information for each individual pixel, to achieve 1×, 2×, and 3× internal zoom functions within the sensor. Specifically, 1× zoom can be achieved based on the 3×3 pixel binning mode, 2× zoom can be achieved based on the 2×2 pixel binning mode, and 3× zoom can be achieved based on the full-pixel mode. The specific implementation for each zoom level can be found in the preceding descriptions of Figures 7(b) and 8(b). By configuring these three modes, users can be provided with a consistent zoom level selection, and clear image display can be achieved at each zoom level. Therefore, the image sensing device provided in this application embodiment is of significant importance for the future design direction of image sensor products.

[0240] In practical applications, for a region to be merged under the current pixel merging mode, in order to connect the floating diffusion sections of the readout units containing the pixels in the region to be merged into a whole, and to merge and accumulate the charges sensed by each pixel in the region to be merged in the connected floating diffusion section, the control circuit 11 can specifically control the connection switch and the transmission switch through the following mechanism:

[0241] For the region to be merged in the current pixel merging mode, the control connection switch connects the floating diffusion section in the adjacent readout units, so that the first region formed by the interconnection of the adjacent readout units includes the region to be merged; for each pixel in the first region, the transmission switch between the pixel included in the region to be merged and the floating diffusion section is turned on, and the transmission switch between the other pixels and the floating diffusion section is turned off, so that the charge sensed by each pixel in the region to be merged is merged and accumulated in the connected floating diffusion section.

[0242] Specifically, in the process of connecting the floating diffusion sections of adjacent readout units via connection switches so that all pixels in the area to be merged are connected to the same floating diffusion section, the readout units that connect their floating diffusion sections via connection switches are not limited to the first readout unit (i.e., the readout unit that includes the pixels in the area to be merged). In practical applications, the floating diffusion sections of each first readout unit and other readout units (excluding the first readout unit) can also be connected into a whole. As long as the first area formed by the interconnected readout units includes the area to be merged, it can be ensured that all pixels in the area to be merged are connected to the same floating diffusion section. During data output, for each pixel in the first area, the transmission switch between the light receiving device of the pixel included in the area to be merged and the connected floating diffusion section is turned on, while the transmission switch between the light receiving device of other pixels and the connected floating diffusion section is turned off, so that the photogenerated charges sensed by each pixel in the area to be merged are merged and accumulated in the connected floating diffusion section.

[0243] Figure 12 provides an example, showing 16 readout units arranged in four rows and four columns, with each readout unit containing 2×2 pixels. Taking the 3×3 pixel merging mode as an example, the 3×3 pixels to be merged in the upper left corner forming the merging region, Figure 12 illustrates three different control methods for the connection switches, all of which can achieve charge domain merging for pixels within this merging region. Method 1: Connect the floating diffusers of the first two readout units in the first row and the first two readout units in the second row to form a first region A. Turn on the transmission switches between the 9 pixels to be merged and the floating diffusers, and turn off the transmission switches between the other pixels in the first region A and the floating diffusers, allowing the charges sensed by the 9 pixels to be merged to be merged and accumulated in the connected floating diffusers. Method 2: Connect the floating diffusers in the first three readout units in the first row, the first three readout units in the second row, and the first three readout units in the third row to form a first region B. Connect the floating diffusers of the 9 pixels to be merged and the first three readout units in the second row to form a first region B. Method 3: Connect the floating diffusion sections of all 16 readout units shown in the figure to form a first region C. Connect the transmission switches between the 9 pixels to be merged and the floating diffusion section, and disconnect the transmission switches between the other pixels in the first region C and the floating diffusion section. This allows the charges sensed by the 9 pixels to be merged to be merged and accumulated in the connected floating diffusion section. It should be understood that Figure 12 is only an example. In practical applications, the first region formed by connecting adjacent readout units is not limited to the example in Figure 12, and the shape of the first region can be irregular in addition to the rectangle shown in the figure.

[0244] As mentioned earlier, the conversion gain of the voltage signal converted from the accumulated charge in the floating diffuser is inversely proportional to the capacitance of the floating diffuser. When the floating diffusers of adjacent readout units are connected by a connecting switch, the total capacitance of the connected floating diffusers consists of two components: the parasitic capacitance of each connected floating diffuser and the parasitic capacitance of the connecting switch itself. In this embodiment, for a region to be merged, the total capacitance of the connected floating diffusers can be adjusted by changing the number of readout units in the first region used to output pixel information within that region, thereby controlling the conversion gain of the pixel information output from the region to be merged and achieving greater output flexibility.

[0245] In one embodiment of this application, for each region to be merged in a pixel merging mode, during the process of outputting the pixel information of each region to be merged, it should be ensured that the number of readout units contained in the first region corresponding to each region to be merged is the same.

[0246] Among them, the first region corresponding to a region to be merged specifically refers to the first region used to output the pixel information within the region to be merged, and the first region includes the region to be merged.

[0247] Figure 13 provides an example, where (a) to (d) show a total of nine readout units arranged in three rows and three columns, respectively. Each readout unit contains 2×2 pixels, for a total of 6×6 pixels. In the 3×3 pixel merging mode, the 6×6 pixels shown are specifically combined into four regions to be merged, which are shown in (a) to (d) respectively. Each region to be merged contains 3×3 pixels to be merged. In the example of Figure 13, as shown in Figure 13(a), the upper left corner of the region to be merged corresponds to the first region a, which includes the first two readout units of the first row and the first two readout units of the second row, for a total of four readout units; as shown in Figure 13(b), the upper right corner of the region to be merged corresponds to the first region b, which includes the last two readout units of the first row and the last two readout units of the second row, for a total of four readout units; as shown in Figure 13(c), the lower left corner of the region to be merged corresponds to the first region c, which includes the first two readout units of the second row and the first two readout units of the third row, for a total of four readout units; as shown in Figure 13(d), the lower right corner of the region to be merged corresponds to the first region d, which includes the last two readout units of the second row and the last two readout units of the third row, for a total of four readout units. As can be seen, in the 3×3 pixel merging mode, the number of readout units contained in the first region corresponding to the four regions to be merged shown in Figure 13 is the same, which is four.

[0248] As can be easily understood from the preceding explanation of parasitic capacitance, the total capacitance of the interconnected floating diffusers within the first region is related to the parasitic capacitance of each interconnected floating diffuser (i.e., the floating diffuser of each readout unit within the first region). According to the design of this embodiment, each readout unit within the pixel array 12 employs the same circuit structure design; therefore, the size of the parasitic capacitance of the floating diffusers within each readout unit is theoretically consistent. Based on this, it can be concluded that the total capacitance of the interconnected floating diffusers within the first region is actually related to the number of readout units within the first region.

[0249] Therefore, in this embodiment of the application, by controlling that each region to be merged in a pixel merging mode contains the same number of readout units within its corresponding first region, the total capacitance of the (connected) floating diffuser portions within each first region can be maintained at a substantially the same level. The capacitance of the floating diffuser portions within the first region determines the conversion gain of the charge to voltage value within the floating diffuser portion. If the capacitance of the floating diffuser portions in two first regions is different, even if the floating diffuser portions in the two first regions accumulate the same amount of photogenerated charge, they will ultimately form different voltage values. This means that the image sensing device unreasonably converts the received light of the same intensity into voltage values ​​representing different intensities, adversely affecting the image quality of the final image generated based on the pixel information output by the image sensing device. Therefore, by controlling the total capacitance of the floating diffuser portions within each first region to be substantially the same, this embodiment of the application helps ensure that the conversion gain of each pixel information output from the pixel merging mode is also substantially the same, thereby helping to ensure the image quality of the final generated image.

[0250] Furthermore, in practical applications, the number of connecting switches that need to be turned on to form a specific first region may not be fixed. For example, as discussed in the example in Figure 10 above, to connect readout units 1 to 4, connecting switches SW1, SW2, SW4, and SW5 can all be turned on, or only three of them can be turned on.

[0251] In one embodiment of this application, for each region to be merged in a pixel merging mode, in addition to ensuring that each region to be merged contains the same number of readout units in the first region, it should also be ensured that the number of connection switches in the on state in each first region is also the same.

[0252] Figure 14 shows an example of the on / off states of the connecting switches in each of the first regions in Figure 13. Referring to Figure 14(a), the first region a contains four readout units, each connected to one another by four connecting switches: SWA, SWb, SWc, and SWd. Based on the previous explanation of Figure 10, it is easy to understand that to form the first region a, all four connecting switches can be turned on, or only three of them can be turned on. The same logic applies to the first regions b through d.

[0253] Figure 14 shows the on / off state of each connecting switch in the first region a to the first region d. It can be seen that in the example in Figure 14, a total of three connecting switches, SWA, SWb and SWd, are turned on in the first region a, and three connecting switches are also turned on in each of the first regions b to d. The number of connected switches turned on is the same in each of the first regions.

[0254] As mentioned earlier, the parasitic capacitance inherent in the connecting switch itself is also part of the total capacitance of the floating diffusion section in the first region. Although the parasitic capacitance provided by the connecting switch constitutes only a small component of the total capacitance of the connected floating diffusion section compared to the parasitic capacitance inherent in the floating diffusion section itself, it generally does not have a significant impact on the size of the total capacitance. However, in this embodiment, in order to maximize the improvement of image quality, for the first region corresponding to each region to be merged in a pixel merging mode, by controlling the number of readout units contained in each first region to be consistent with the number of connecting switches in the on state, it helps to further maintain the total capacitance of the floating diffusion section in each first region at the same level, so that the conversion gain of each pixel information output in a pixel merging mode is also basically the same, thereby helping to further improve image quality.

[0255] In a preferred embodiment of this application, for the first region corresponding to each region to be merged in a pixel merging mode, while ensuring that the number of connection switches in the on state contained in each first region is the same, it should specifically ensure that: the number of first connection switches in the on state contained in each first region is the same, and the number of second connection switches in the on state contained in each first region is the same.

[0256] The first connection switch is used to connect the floating diffusers in two adjacent readout units in the row direction, and the second connection switch is used to connect the floating diffusers in two adjacent readout units in the column direction.

[0257] Taking the first region a shown in Figure 14(a) as an example, in which connecting switches SWA and SWd are the first connecting switches, and connecting switches SWb and SWc are the second connecting switches. In the example of Figure 14, the first region a includes two first connecting switches, SWA and SWd, in the on state, and one second connecting switch, SWb, in the on state; the first regions b to d also each contain two first connecting switches in the on state and one second connecting switch in the on state. Therefore, in the first regions a to d, the number of first connecting switches in the on state and the number of second connecting switches in the on state are the same in each of the first regions.

[0258] Specifically, considering the difference in connection direction between the first and second connecting switches, their circuit connections and layouts differ, potentially leading to differences in their parasitic capacitances. In this embodiment, by ensuring that the number of first and second connecting switches in the conducting state within each first region is consistent, it helps maintain the total capacitance of the floating diffuser in each first region at the same level with higher precision. This ensures that the conversion gain of each pixel information output from a pixel merging mode is essentially the same, thereby further improving image quality.

[0259] In one embodiment of this application, to facilitate the image sensing device in outputting pixel information under the current pixel merging mode in a parallel + serial manner, thereby improving the output efficiency of pixel information, for each pixel merging mode, in the functional design stage for the pixel merging mode, the regions to be merged under that pixel merging mode can be grouped into multiple groups to be merged, and each group to be merged includes one or more regions to be merged. The regions to be merged within the same group correspond to different first regions, and there is no overlap between any two first regions.

[0260] The control circuit 11 can control the output order of pixel information of each region to be merged through the following mechanism: synchronously outputting the pixel information of each region to be merged in the same group through different readout lines, and interleaving the pixel information of the regions to be merged in different groups in terms of timing.

[0261] For each region to be merged within the same merge group, the control circuit 11 can control the connection switch to form a first region corresponding to each merged region within the pixel array 12; and control the selection switch to output the voltage signal formed by the floating diffuser in each first region synchronously through different readout lines. The specific control process can be found in the preceding description.

[0262] Specifically, since the floating diffusion sections of each readout unit within a first region are interconnected, these floating diffusion sections accumulate charge as a whole and convert it into a voltage signal. Therefore, for a given first region, only one voltage value can be output at a time (i.e., only pixel information of one region to be merged can be output from one first region at a time). Thus, in this embodiment, for each region to be merged within the same merging group (i.e., each region to be merged whose pixel information needs to be output synchronously), it must be ensured that the first regions corresponding to these regions are all different, and that there is no overlap between any two first regions, in order to achieve synchronous output of pixel information for each region to be merged within the merging group.

[0263] The following is an illustrative example using Figures 13 and 15: Regarding the 6×6 pixel structure shown in Figure 13, in the 3×3 pixel merging mode, it contains four 3×3 regions to be merged. The first regions corresponding to these regions are first region a to first region d. Referring to Figures 13(a) to (d), it can be seen that any two of these four first regions overlap (e.g., the pixels in rows 1-3 and columns 3-4 are the overlap of first region a and second region b, etc.). Therefore, it is impossible to synchronously form first regions a to first region d within the pixel array 12. Consequently, any two regions to be merged shown in (a) to (d) cannot be grouped into the same merge group. Instead, the pixel information of the four regions to be merged shown in (a) to (d) needs to be output in a staggered manner. Figure 15 shows an example of regions to be merged that can be grouped into the same merge group. It illustrates two 3×3 merge regions, Region I and Region II, each corresponding to a different first region, designated Region I and Region II respectively. It can be seen that there is no overlap between Region I and Region II; therefore, Region I and Region II can be grouped into the same merge group, and their pixel information can be output synchronously.

[0264] In a product architecture employing a column-level ADC, the readout lines within the image sensing device are specifically column lines. The floating diffusers of each readout unit in the same column are connected to the same column line, while the floating diffusers of readout units in different columns are connected to different column lines. The example in Figure 9 illustrates the case where the readout lines are column lines, showing column lines OUT1 and OUT2. The floating diffusers FD of readout units 1 and 3 in the first column are both connected to column line OUT1, and the floating diffusers FD of readout units 2 and 4 in the second column are both connected to column line OUT2.

[0265] Regarding this product architecture, it's important to note that since each readout unit in the same column shares the same column line, and only one voltage value can be output from one column line at a time, to synchronously output pixel information for two regions to be merged in the same column, it's crucial to ensure that the first regions corresponding to these two regions do not overlap, and that there are sufficient column lines available for data output. If the floating diffuser of the first region corresponding to each of the two regions to be merged is only connected to the same column line, synchronous output of pixel information for these two regions cannot be achieved, and therefore, these two regions cannot be grouped into the same merging group.

[0266] In one example, if the first region corresponding to at least one of the two regions to be merged in the same column covers at least two readout units in the row direction, that is, if the floating diffuser of the first region corresponding to at least one of the two regions to be merged is connected to two different column lines, then the two regions to be merged can be classified into the same merge group. In this case, the control circuit 11 can specifically output the pixel information of the two regions to be merged synchronously through different column lines.

[0267] The specific column line from which pixel information formed in a first region is output can be controlled by a selection switch. For the corresponding details, please refer to the previous explanation of Figures 9-10.

[0268] Referring to Figure 16, which illustrates the connection relationship between each readout unit and the column lines when the pixel structure in Figure 15 adopts a column-level ADC architecture, the example in Figure 16 shows the connection relationship between each readout unit and the column lines. In the example of Figure 16, the floating diffusion sections of the readout units in the first column are all connected to column line 1, the floating diffusion sections of the readout units in the second column are all connected to column line 2, and the floating diffusion sections of the readout units in the third column are all connected to column line 3. It can be seen that the regions to be merged I and II in the same column correspond to the first region I and the first region II, respectively. The floating diffusion section of the first region I is connected to both column lines 1 and 2, and the floating diffusion section of the first region II is also connected to both column lines 1 and 2. Therefore, the control circuit 11 can synchronously output the pixel information of the two regions to be merged as shown in the figure through column lines 1 and 2, respectively. For example, the control circuit 11 can turn on the selection switch between the floating diffuser of readout unit a and / or readout unit b and column line 2, so that the pixel information formed in the first region I is output from column line 2; and turn on the selection switch between the floating diffuser of readout unit c and / or readout unit d and column line 1, so that the pixel information to be formed in the first region II is output from column line 1.

[0269] In this embodiment of the application, for a product architecture that uses a column-level ADC, the output efficiency of pixel information is further improved by using different column lines to synchronously output the pixel information of the regions to be merged in the same column.

[0270] Based on the image sensing device provided in the foregoing embodiments of this application, by reasonably arranging pixels for sensing different colors of light within the pixel array 12, the image sensing device can be applied to the sensing of color images. A possible design is described below with reference to specific embodiments:

[0271] In one embodiment of this application, the pixels in the pixel array are arranged as a plurality of extended units in an array, each of the extended units including 2×2 color units arranged in a Bayer pattern.

[0272] The Bayer pattern is a common color arrangement scheme in image sensor design. Figure 17 provides a schematic diagram showing an extension unit comprising four color units (first to fourth color units) arranged in a Bayer pattern. The first color unit consists of first-color pixels, the second and third color units consist of second-color pixels, and the fourth color unit consists of third-color pixels. In one example, the first color pixel is a red pixel, the second color pixel is a green pixel, and the third color pixel is a blue pixel. In this embodiment, the red pixel specifically refers to a pixel configured with a red filter for sensing red light; the green and blue pixels are similarly defined.

[0273] To facilitate the merging of pixel charge domains, multiple pixels of the same color need to be arranged within the same color unit so that adjacent pixels of the same color can be merged together in pixel merging mode for pixel information output. In the illustration of Figure 17, each color unit specifically includes 3×3 pixels of the same color. When arranging the pixel pattern of the pixel array 12 within the image sensing device using the design of Figure 17, two pixel merging modes under different zoom ratios can be configured for the image sensing device, as illustrated in Figure 18:

[0274] One is the first pixel merging mode at a 1x zoom ratio (i.e., the 3×3 pixel merging mode mentioned earlier). In the first pixel merging mode, each pixel in the pixel array 12 is specifically combined into multiple 3×3 regions to be merged. Based on the illustration in Figure 18, it can be seen that in the first pixel merging mode, each color unit in a diffusion unit can constitute a region to be merged.

[0275] The second is the second pixel binning mode at a 2x zoom ratio (2×). The second pixel binning mode can be implemented in two different ways:

[0276] This application refers to the first implementation as the first sub-mode (i.e., the 2×2 pixel merging mode mentioned above). In the first sub-mode, the pixel information in the first output area of ​​the pixel array 12 (usually 1 / 4 of the entire photosensitive area of ​​the pixel array 12, i.e., 1 / 2 high and 1 / 2 wide) needs to be output, and the pixels in the first output area are specifically combined into multiple 2×2 regions to be merged. Based on the illustration in Figure 18, it can be seen that for 6×6 pixels in an extended unit, these 6×6 pixels are specifically combined into 9 2×2 regions to be merged.

[0277] However, as can be seen from the illustration in Figure 18, in the first sub-mode, some areas to be merged may contain pixels of different colors simultaneously. That is, in the first sub-mode, the photogenerated charges sensed by pixels of different colors are merged into a single pixel information for output. In this case, subsequent data processing is required to perform color conversion on the output pixel information to ensure that each pixel information in the final output is used to represent a specific color (the specific processing methods will be explained later).

[0278] In view of the above problems, this application embodiment also provides an implementation method that can avoid color mixing and merging, namely the second sub-mode. In the second sub-mode, for each color unit in the first output area containing 3×3 pixels, 2×2 adjacent pixels are combined into a 2×2 region to be merged, and the remaining five pixels are each used as a 1×1 single pixel readout region.

[0279] In the second sub-mode, the embodiments of this application do not specifically limit the relative positions of the 2×2 regions to be merged within the color units. Figure 19 shows several optional examples. In Figure 19(a), the 2×2 regions to be merged within the four color units of the extended unit are distributed at the four corners of the extended unit; in Figure 19(b), the regions to be merged within the four color units are distributed in the central area of ​​the extended unit; in Figure 19(c), the regions to be merged within the four color units are distributed on both sides in the row direction of the extended unit; and in Figure 19(d), the regions to be merged within the four color units are distributed on both sides in the column direction of the extended unit. It should be understood that for each color unit, the 2×2 regions to be merged contained therein can be located at any of the four corner positions of the color unit. Therefore, the layout of the regions to be merged and the single-pixel readout areas is not limited to the illustrations in Figure 19.

[0280] In one example, in addition to the first pixel merging mode and the second pixel merging mode described above, a full pixel mode at a 3x magnification can also be configured for the image sensing device. In full pixel mode, the image sensing device specifically outputs the pixel information of each pixel in the second output area of ​​the pixel array 12 (usually 1 / 9 of the entire photosensitive area of ​​the pixel array 12, i.e., 1 / 3 high and 1 / 3 wide).

[0281] Specifically, since it does not involve the merging of pixel charge domains, pixel information output in full-pixel mode can be performed with all connection switches within pixel array 12 open. In the actual implementation, because all pixels within a readout unit share a floating diffuser, only one voltage value can be read from one readout unit at a time. Therefore, for each readout unit, the pixel information of each pixel needs to be output in a staggered timing sequence, and the specific pixel information to be output can be controlled by a transmission switch.

[0282] By configuring the image sensing device with a first pixel merging mode, a second pixel merging mode, and a full pixel mode, three different zoom ratios—1×, 2×, and 3×—can be achieved, providing users with a seamless zoom ratio selection and ensuring clear image display at each zoom ratio. Specifically, to achieve a 2× zoom ratio, either the first sub-mode or the second sub-mode can be configured; alternatively, both can be configured.

[0283] Furthermore, in current product architectures, image sensing devices typically process the pixel information output from pixel array 12 into a Bayer array format image for output, so that subsequent devices (such as ISP chips) can generate a final image that is recognizable to the naked eye based on the Bayer array format image. Therefore, in one embodiment of this application, the image sensing device further includes an information processing unit configured to generate an output image based on the pixel information output from the readout line. Specifically, the output image refers to an image conforming to the Bayer array arrangement. Specifically, when the image sensing device operates in pixel merging mode, the information processing unit is specifically used to generate an output image in the current pixel merging mode based on the pixel information output from the readout line; when the image sensing device operates in full-pixel mode, the information processing unit is specifically used to generate an output image in full-pixel mode based on the pixel information output from the readout line.

[0284] Referring to the example in Figure 1, the information processing unit can be specifically understood as part of the readout circuit 13. For the raw voltage signal output from the readout lines of the pixel array 12, one possible processing logic is as follows: first, perform analog-to-digital conversion on the raw voltage signal output from each readout line of the pixel array 12 to obtain pixel information in digital signal form; then, based on the actual physical location of the pixel region corresponding to each piece of pixel information in the pixel array 12, perform pixel reordering on each piece of pixel information (pixel reordering is a common pixel processing technique in the field, and its specific implementation can be found in related technologies) to obtain an initial image; then, convert the initial image into an output image conforming to the Bayer array arrangement. In practical applications, the entire process of processing the raw voltage signal into an output image conforming to the Bayer array arrangement may involve more steps than the aforementioned logic, and the execution order of each step may also differ. Furthermore, in specific product forms, the information processing unit can be integrated either within the image sensor as part of the logic circuitry surrounding the pixel array 12, or within the circuitry of the back-end ISP chip.

[0285] As can be seen from Figure 18, in the first pixel merging mode, after the charge domain merging is completed, the charges sensed by each pixel of the same color in the same color unit are merged into one pixel information for output. Therefore, the initial image obtained after pixel reordering already conforms to the Bayer array arrangement, and thus the obtained initial image can be directly used as the output image (third output image) for output. In the first sub-mode, the initial image obtained after pixel reordering (first image) does not conform to the Bayer array arrangement, so it needs to be converted into the first output image that conforms to the Bayer array arrangement before output. In the second sub-mode, the initial image obtained after pixel reordering (second image) also does not conform to the Bayer array arrangement, so it needs to be converted into the second output image that conforms to the array arrangement before output. In the full pixel mode, the initial image obtained after pixel reordering also does not conform to the Bayer array arrangement, so it needs to be converted into the fourth output image that conforms to the array arrangement before output.

[0286] Since the distribution of different color pixels in the generated initial image differs between the first sub-mode, the second sub-mode, and the full-pixel mode, a corresponding information processing algorithm needs to be configured for each mode. For the initial image obtained in each mode, the information processing algorithm corresponding to that mode is used to convert it into an output image conforming to the Bayer array arrangement. The information processing algorithms for each mode are explained below:

[0287] For full-pixel mode, the initial image needs to be mosaicked to obtain a fourth output image conforming to the Bayer array arrangement. Mosaic rearrangement algorithms for 9in1 (referring to a pixel arrangement scheme that groups 3×3 pixels of the same color together) structures are relatively common; refer to relevant techniques.

[0288] For the first sub-mode, in one example, the first image can be converted into a first output image conforming to the Bayer array arrangement using artificial intelligence. In this example, after generating the first image, the information processing unit obtains the first output image by inputting the first image into a pre-trained first model (an image conversion model corresponding to the first sub-mode). The first model can be trained through the following process:

[0289] Step A1: Acquire the first sample image sensed by pixel array 12 for the calibration scene in the first sub-mode.

[0290] The first sample image is in the same form as the aforementioned first image, and is the initial image obtained after reordering the original pixel information output from the pixel array 12.

[0291] Step A2: Obtain the reference image sensed by the reference pixel array for the calibrated scene.

[0292] Specifically, to provide a reference image corresponding to the first sample image during training, this application can additionally fabricate another image sensor for acquiring the reference image, and arrange a reference pixel array in this other image sensor. The pixel arrangement in this reference pixel array itself conforms to the Bayer array arrangement; and the resolution of the reference pixel array is consistent with the target resolution of the first output image. That is, if it is expected that the first image will be converted into a first output image containing m×n pixels through the first model, then the reference pixel array should also contain m×n pixels. In addition, to ensure that the scene area sensed by the reference pixel array is consistent with the scene area contained in the first sample image, the size of the reference pixel array should also be consistent with the size of the first output area within the pixel array 12.

[0293] Based on the above design, for the first sample image sensed by the pixel array 12 in a calibration scene, the initial image sensed by the reference pixel array in the same calibration scene (obtained by reordering the original pixel information output by the reference pixel array) is used as the reference image corresponding to the first sample image. In practical applications, the first sample image and the corresponding reference image can also be collected in various different calibration scenes to construct a training sample set.

[0294] Step A3: Using the first sample image as input data, train the first model by combining it with the reference image corresponding to the first sample image to obtain the trained first model.

[0295] Specifically, by outputting the first sample image to the first model, the first model converts it into the corresponding predicted first output image; the model loss is calculated based on the difference between the first output image and the corresponding reference image, and the model parameters of the first model are updated according to the model loss, thus achieving the training of the first model.

[0296] In practical applications, the type of model for the first model can be selected according to actual needs. In one example, the first model can be an artificial intelligence network with ground truth, such as Convolutional Neural Network (CNN), U-net, R-CNN, Residual Neural Network (ResNet), or Pix2Pix. In another example, the first model can also be an artificial intelligence network without ground truth, such as Generative Adversarial Network (GAN) or self-supervised learning model.

[0297] The loss function used during training can be constructed based on the difference between the predicted first output image and the reference image. For example, it can be constructed based on the mean square error, mean absolute error, etc. between the pixel values ​​in the two images.

[0298] In a preferred embodiment of this application, for the first output image in the first sub-mode, its resolution can be made consistent with that of the third output image while conforming to the Bayer array arrangement.

[0299] To achieve the above objectives, the information processing unit can obtain the first output image by performing color conversion and up-interpolation on the first image. As can be seen from the illustration in Figure 18, through this processing, the pixel information read from an extended unit will be presented as 4×4 pixel information in the first output image.

[0300] The above processing can be implemented using the first model described above, by setting the resolution of the reference pixel array used in step A2 to be consistent with the resolution of the third output image. To ensure that the scene area sensed by the reference pixel array is consistent with the scene area contained in the first sample image, the side length ratio of the pixel array 12 to the individual pixels in the reference pixel array can be set to 1:1.5.

[0301] Those skilled in the art will readily recognize that after completing charge domain merging in the first pixel merging mode, the resolution of the third output image obtained is consistent with the resolution of the fourth output image obtained in the full pixel mode. In this embodiment, by ensuring that the resolution of the first output image obtained in the first sub-mode is consistent with the resolution of the third output image, the image sensing device can output output images with the same resolution in the first pixel merging mode, the full pixel mode, and the first sub-mode, thereby providing users with a consistent image display effect at zoom levels of 1×, 2×, and 3×, ensuring a good user experience.

[0302] For the second sub-mode, in one example, the second image can be converted into a first output image conforming to the Bayer array arrangement using artificial intelligence. In this example, after generating the second image, the information processing unit inputs the second image into a pre-trained second model (an image conversion model corresponding to the second sub-mode) to obtain the second output image. The second model can be trained through the following process:

[0303] Step B1: Acquire the second sample image sensed by the pixel array for the calibrated scene in the second sub-mode.

[0304] Step B2: Obtain a reference image sensed by the reference pixel array for the calibrated scene.

[0305] The reference pixel array has a pixel arrangement that conforms to the Bayer array arrangement, and the resolution of the reference pixel array is consistent with the target resolution of the second output image.

[0306] Step B3: Using the second sample image as input data, train the second model by combining the second sample image to obtain the trained second model.

[0307] The principles of steps B1-B3 above are the same as those of steps A1-A3 mentioned above. Please refer to the explanation above for details.

[0308] In a preferred embodiment of this application, for the second output image in the second sub-mode, its resolution can be made consistent with that of the third output image while conforming to the Bayer array arrangement.

[0309] To achieve the above objective, the information processing unit can obtain the second output image by downsampling and mosaic rearrangement of the first image. As can be seen from the illustration in Figure 18, through this processing, the pixel information read from an extended unit will be presented as 4×4 pixel information in the second output image.

[0310] The above processing can be implemented using the second model mentioned above. For specific implementation methods and beneficial effects, please refer to the previous explanation of the first sub-mode. As can be easily seen from the illustration in Figure 18, since the color distribution and resolution of pixels in the output images of the first and second sub-modes are the same, the reference pixel arrays used in steps A2 and B2 are actually the same.

[0311] For ease of understanding, the following example illustrates a possible application scenario of this application by using the image sensing device provided in the embodiments of this application in a smartphone, in conjunction with Figures 20 and 21:

[0312] In the example of Figure 20, the smartphone is configured with three different zoom levels: 1×, 2×, and 3×, with 1× being the default. After the user enters the camera app's photo-taking function, at the default 1× zoom level, the smartphone performs the imaging process according to the flow shown in Figure 21: The control circuit 11 controls the transmission switch TX, selection switch SEL, connection switch SW, and reset switch SEL according to the control scheme of the first pixel merging mode, so that the pixel information of each 3×3 area to be merged in the pixel array is output as data through the readout line; after reordering the pixel information, an initial image can be generated. The information processing unit directly outputs this initial image as the output image, and the ISP chip generates a final image that is recognizable to the naked eye based on the output image, which is then displayed on the phone screen.

[0313] Those skilled in the art will understand that the control circuit 11 controls the pixel information output process within the pixel array 12, specifically based on a specific sequence of control signals (transmission signals, selection signals, connection signals, and reset signals) sent by the control circuit 11 to the pixel array 12. Therefore, in practical applications, for each zoom level, developers can pre-plan a specific control scheme for each switch, and accordingly design and store the specific sequence of each control signal as a configuration file. Thus, the control circuit 11 can send corresponding control signals based on the configuration file at the current zoom level, achieving switch control at the corresponding zoom level.

[0314] Referring to the illustration in Figure 20(a), at the default 1x magnification, the user can instruct the smartphone to switch to 2x magnification through a first interactive operation with the smartphone. This application embodiment does not limit the specific implementation of the first interactive operation. For example, the first interactive operation can be set as dragging the 2x icon on the screen towards the center, or it can be set as clicking on different magnification icons in the interface, sliding the slider or semi-circular adjuster corresponding to different magnifications, etc. After sensing the first interactive operation, the processor can reload the control circuit 11 and the information processing unit to cause the control circuit 11 and the information processing unit to enter the first sub-mode / second sub-mode. At 2× magnification, the smartphone similarly implements the imaging process based on the flow shown in Figure 21. The difference between the flow at 2× magnification and that at 1× magnification is that the control circuit 11 performs switching control according to the control scheme of the first sub-mode or the second sub-mode (depending on the specific configuration), so that the pixel information of each 2×2 region to be merged in the first sub-mode is output through the readout line / the pixel information of each 2×2 region to be merged and the single pixel readout region in the second sub-mode is output through the readout line; and after generating the initial image based on pixel reordering, if it is the first sub-mode, the information processing unit needs to perform color conversion and up-interpolation conversion on the initial image to generate the output image, and if it is the second sub-mode, the information processing unit needs to perform downsampling and mosaic rearrangement on the initial image to generate the output image.

[0315] Referring to the illustration in Figure 20(b), at the default 1x magnification, the user can switch to 3x magnification through a second interactive operation with the smartphone. Similarly, as described above, the second interactive operation can be set to drag the 3x icon on the screen towards the center, or it can be set to click on an interface icon, slide along a slider, or perform a half-circle adjustment. After sensing the second interactive operation, the processor can reload the control circuit 11 and the information processing unit to put them into full-pixel mode. At 3x magnification, the smartphone similarly implements the imaging process based on the flow shown in Figure 21. The difference between the 3x magnification flow and the 1x magnification flow is that the control circuit 11 specifically follows the full-pixel mode control scheme for switching control, so that the pixel information of each pixel in the second output area is output separately through the readout line; and after generating the initial image based on pixel reordering, the information processing unit needs to perform mosaic rearrangement on the initial image to generate the output image.

[0316] As mentioned earlier, this application provides specific implementations of readout units with sizes of 2×4, 2×2, 1×3, and 3×3 within the image sensing device. The following description, in conjunction with specific embodiments, will illustrate these four schemes:

[0317] Option 1: Readout unit size is 2×4

[0318] In Scheme 1, the basic structure of the readout unit is shown in Figure 22. Each readout unit includes 2×4 pixels, and these 2×4 pixels share the same floating diffuser. When the structure of Figure 22 is applied to the pixel arrangement of Figure 17, the basic structure of the pixel array 12 is shown in Figure 23, wherein the floating diffusers of each pair of adjacent readout units are connected by a connection switch.

[0319] Figure 24 shows a schematic diagram of the circuit structure within the pixel array 12 when the structure in Figure 22 is applied. It illustrates four adjacent readout units arranged in two rows and two columns. Each readout unit contains 2×4 pixels, and the photodiodes PD of each 2×4 pixel are connected to the same floating diffuser FD via connection switches TX. The floating diffuser FD of each readout unit is connected to a readout line OUT via an amplifier AMP and a selection switch SEL, and is connected to a reset voltage via a reset switch RST. Furthermore, the floating diffusers FD of every two adjacent readout units are connected via a selection switch SW.

[0320] When applying a column-level ADC architecture, the floating diffuser FD in each readout unit of the same column will be connected to the same readout line. For example, in Figure 24, the floating diffuser FD in the readout unit of the first column is connected to readout line OUT1, and the floating diffuser FD in the readout unit of the second column is connected to readout line OUT2.

[0321] The specific implementation of different pixel output modes in Scheme 1 will be explained below.

[0322] First pixel merging mode:

[0323] As shown in Figure 18, in the first pixel merging mode, the four color units within an extended unit each constitute a region to be merged. Based on Figure 23, it can be seen that within the extended unit, pixels of four different colors share the same floating diffuser, thus preventing the simultaneous reading of pixel information from different regions to be merged within the extended unit. Therefore, in practical applications, for an extended unit, the control circuit 11 can specifically control the connection switches to sequentially form the first region corresponding to each region to be merged, causing the pixel information of the four regions to be merged within the extended unit to be output in an interleaved timing sequence.

[0324] As mentioned earlier, to ensure the consistency of conversion gain among the pixel information output in the same pixel merging mode, the first region corresponding to each region to be merged should contain the same number of readout units. To achieve this, this application provides a specific design for the first region in this mode. It should be noted that, as shown in Figure 23, the expansion units included in Scheme 1 can be divided into two types: a first type of expansion unit and a second type of expansion unit. The first pixel of the first row of the first type of expansion unit is the first pixel of the first row within a readout unit, while the first pixel of the first row of the second type of expansion unit is the first pixel of the third row within another readout unit in the column direction. Since the positional relationship between pixels and readout units in these two types of expansion units is not the same, different designs are needed for the first regions corresponding to each region to be merged within these two types of expansion units. The following explanation uses Figures 25(a) to 25(d) as examples. In Figures 25(a) to 25(d), the number of columns in the region to be merged and the number of columns in the readout unit are indicated by the letter 'k', where 'k' is a natural number. The updates proceed from left to right according to the rule 'k = k + 1'. The number of rows in the region to be merged and the number of rows in the readout unit are indicated by the letter 'j', where 'j' is a natural number. The updates proceed from top to bottom according to the rule 'j = j + 1'.

[0325] Referring to Figure 25(a), the first region corresponding to the region to be merged in row 4j+1 and column 2k+1 is formed by connecting the following readout units: the readout units in columns 3k+1 to 3k+3 of row 3j+1, and the readout unit in column 3k+1 of row 3j+2 (in the first type of extended unit, the first region corresponding to the first region to be merged in the first row includes: the first three readout units in the first row within the scope of the first type of extended unit, and the first readout unit in the second row); the first region corresponding to the region to be merged in row 4j+3 and column 2k+2 is formed by connecting the following readout units: the readout units in columns 3k+2 and 3k+3 of row 3j+2, and the readout units in columns 3k+2 and 3k+3 of row 3j+3 (in the second type of extended unit, the first region corresponding to the second region to be merged in the first row includes: the second to third readout units in the first row within the scope of the second type of extended unit, and the second to third readout units in the second row).

[0326] Referring to Figure 25(b), the first region corresponding to the region to be merged in row 4j+1 and column 2k+2 is formed by connecting the following readout units: the readout units in columns 3k+1 to 3k+3 of row 3j+1, and the readout unit in column 3k+3 of row 3j+2 (in the first type of extended unit, the first region corresponding to the second region to be merged in the first row includes: the first three readout units in the first row within the scope of the first type of extended unit, and the third readout unit in the second row); the first region corresponding to the region to be merged in row 4j+3 and column 2k+1 is formed by connecting the following readout units: the readout units in columns 3k+1 and 3k+2 of row 3j+2, and the readout units in columns 3k+1 and 3k+2 of row 3j+3 (the first region corresponding to the first region to be merged in the first row includes: the first two readout units in the first row within the scope of the second type of extended unit, and the first two readout units in the second row).

[0327] Referring to Figure 25(c), the first region corresponding to the region to be merged in row 4j+2, column 2k+1 is formed by connecting the following read units: the read units in columns 3k+1 and 3k+2 of row 3j+1, and the read units in columns 3k+1 and 3k+1 of row 3j+2 (in the first type of extended unit, the first region corresponding to the first region to be merged in the second row includes: the first two read units in the first row and the first two read units in the second row within the scope of the first type of extended unit); the first region corresponding to the region to be merged in row 4j+4, column 2k+2 is formed by connecting the following read units: the read unit in column 3k+3 of row 3j+2, and the read units in columns 3k+1 to 3k+3 of row 3j+3 (in the second type of extended unit, the first region corresponding to the second region to be merged in the second row includes: the third read unit in the first row and the first three read units in the second row within the scope of the second type of extended unit).

[0328] Referring to Figure 25(d), the first region corresponding to the region to be merged in row 4j+2, column 2k+2 is formed by connecting the following read units: the read units in columns 3k+2 and 3k+3 of row 3j+1, and the read units in columns 3k+2 and 3k+3 of row 3j+2 (in the first type of extended unit, the first region corresponding to the second region to be merged in the second row includes: the second to third read units in the first row and the second to third read units in the second row within the scope of the first type of extended unit); the first region corresponding to the region to be merged in row 4j+4, column 2k+1 is formed by connecting the following read units: the read unit in column 3k+1 of row 3j+2, and the read units in columns 3k+1 to 3k+3 of row 3j+3 (in the second type of extended unit, the first region corresponding to the first region to be merged in the second row includes: the first read unit in the first row within the scope of the second type of extended unit, and the first three read units in the second row).

[0329] As can be seen from the above, based on the designs in Figures 25(a) to 25(d), the number of readout units contained in the first region corresponding to each of the four regions to be merged in the first type of expansion unit and the first region corresponding to each of the four regions to be merged in the second type of expansion unit is 4. This helps to ensure the consistency of the conversion gain between the pixel information output in the first pixel merging mode, thereby helping to ensure the image quality of the final generated image.

[0330] Furthermore, Figures 25(a) to 25(d) specifically illustrate the connection switches in the on state within each first region. It can be seen that the above design involves two different types of first regions: the first type consists of four adjacent readout units arranged in two rows and two columns, while the second type consists of three adjacent readout units in the same row and another readout unit in an adjacent row. As easily understood from the accompanying drawings, to connect and form the second type of first region, two first connection switches and one second connection switch must be turned on. In a preferred embodiment of this application, to ensure that the number of first connection switches in the on state and the number of second connection switches in each first region are consistent, as illustrated in the examples in Figures 25(a) to 25(d), each first type of first region is connected and formed by turning on two first connection switches and one second connection switch.

[0331] As mentioned earlier, several regions to be merged that correspond to different first regions and do not overlap with each other can be grouped into the same merge group. The pixel information of each merge region within the same merge group can be output synchronously to improve output efficiency. To achieve this objective, in one possible implementation of this application, for the repeating unit consisting of a first type of readout unit and a second type of readout unit shown in FIG23, referring to the illustration in FIG25(a), the pixel information of the region to be merged in row 4j+1 and column 2k+1 and the region to be merged in row 4j+3 and column 2k+2 are synchronously output through different readout lines (the pixel information of the first region to be merged in the first row of the first type of extended unit and the pixel information of the second region to be merged in the first row of the second type of extended unit are synchronously output through different readout lines); referring to the illustration in FIG25(b), the pixel information of the region to be merged in row 4j+1 and column 2k+2 and the region to be merged in row 4j+3 and column 2k+1 are synchronously output through different readout lines (the pixel information of the second region to be merged in the first row of the first type of extended unit and the pixel information of the second region to be merged in the first row of the second type of extended unit are synchronously output through different readout lines). The pixel information of the first region to be merged in the first row of the first type of expansion unit is output synchronously through different readout lines; referring to the illustration in Figure 25(c), the pixel information of the region to be merged in the 4j+2 row and the 2k+1 column of the first type of expansion unit and the region to be merged in the 4j+4 row and the 2k+2 column of the second type of expansion unit are output synchronously through different readout lines (the pixel information of the first region to be merged in the second row of the first type of expansion unit and the pixel information of the second region to be merged in the second row of the second type of expansion unit are output synchronously through different readout lines); referring to the illustration in Figure 25(d), the pixel information of the region to be merged in the 4j+2 row and the 2k+2 column of the first type of expansion unit and the region to be merged in the 4j+4 row and the 2k+1 column of the second type of expansion unit are output synchronously through different readout lines (the pixel information of the second region to be merged in the second row of the first type of expansion unit and the pixel information of the first region to be merged in the second row of the second type of expansion unit are output synchronously through different readout lines).

[0332] The output processes shown in Figures 25(a) to 25(d) need to be staggered in timing. The specific order can be set as needed. For example, the output can be performed sequentially in the order of Figure 25(a) → Figure 25(b) → Figure 25(c) → Figure 25(d). Furthermore, Figures 25(a) to 25(d) actually illustrate the first region within two adjacent repeating units in a row. When using a column-level ADC product architecture, the illustrations in Figures 25(a) to 25(d) can be referenced to synchronously output the pixel information of the same position to be merged within each repeating unit in the same row through different column lines.

[0333] First sub-pattern:

[0334] As can be easily seen from Figures 18 and 23, when implementing the first sub-mode in the first application scheme, each readout unit of the first output region within the pixel array 12 contains two regions to be merged, and there is no situation where regions to be merged span across readout units. Therefore, to achieve the highest possible conversion gain, pixel information output in the first sub-mode can be performed even when all connection switches within the pixel array 12 are in the off state.

[0335] In the specific implementation process, it is necessary to output the pixel information of two regions to be merged within the same readout unit in a staggered manner. For the region to be merged within a readout unit, the transmission switch between the pixels in the region to be merged and the floating diffuser is turned on, while the transmission switches between other pixels and the floating diffuser are turned off, thus achieving the output of pixel information within the region to be merged.

[0336] Second sub-pattern:

[0337] As mentioned earlier, in the second sub-mode, the 3×3 pixels within each color unit are combined into a 2×2 region to be merged and five single-pixel readout regions. The specific switching control scheme for the pixel array 12 depends on the specific location of each 2×2 region to be merged within the pixel array. In practical applications, the possible distribution locations of the regions to be merged are quite diverse, and the corresponding switching control schemes will also vary. Based on this, this application embodiment first provides a generally applicable operating scheme:

[0338] For a 2×2 region to be merged within a pixel array, if the region spans multiple readout units, the control circuit 11 can control a connection switch to form a first region containing the region to be merged, and output the pixel information of the region to be merged from the first region based on the mechanism described above. For a single-pixel readout region, the pixel information of the single-pixel readout region can be output even when the readout unit containing the single-pixel readout region is disconnected from both its adjacent readout units.

[0339] In practical applications, when a readout unit contains multiple single-pixel readout regions, the pixel information of these multiple single-pixel readout regions needs to be output in a staggered manner in time. The specific single-pixel readout region whose pixel information should be output at any given time can be controlled by a transmission switch.

[0340] This application states that a readout unit and its adjacent readout units are disconnected, specifically meaning that the connection switches between the floating diffusion section of the readout unit and the floating diffusion section of its adjacent readout units are disconnected.

[0341] As mentioned earlier, to ensure consistent conversion gain for each pixel information output under the same mode, it is necessary to ensure that each pixel information is output from a floating diffuser with the same capacitance. However, for the second sub-mode, the pixel information to be output from the region to be merged is formed by merging and accumulating the photogenerated charges sensed by 2×2 pixels, while the pixel information to be output from the single-pixel readout region is formed by the photogenerated charges sensed by a single pixel. Ensuring consistent conversion gain for these two types of pixel information is meaningless. Therefore, in this embodiment, the pixel information of the single-pixel readout region is output when both the readout unit containing the single-pixel readout region and its adjacent readout units are disconnected. This helps to maximize the conversion gain and minimize noise of the output pixel information in the single-pixel readout region.

[0342] In a preferred embodiment, for each 2×2 region to be merged within the pixel array, the number of readout units contained in the first region corresponding to each region to be merged can still be set to be consistent, so as to ensure that the conversion gain of the pixel information of each region to be merged is basically consistent.

[0343] The following examples illustrate the four ways to set up the regions to be merged, as shown in (a) to (d) of Figure 19.

[0344] When the second sub-mode is implemented using the method shown in Figure 19(a), it can be seen from Figure 19(a) and Figure 23 that the four regions to be merged within each extended unit are located in separate readout units, and there is no situation where the regions to be merged span across readout units. Therefore, specifically, the pixel information of each region to be merged can be output separately when all the connection switches in the pixel array 12 are in the off state.

[0345] In practical implementation, when a single readout unit contains two or more regions to be merged and / or single-pixel readout regions, the pixel information of each region to be merged and / or single-pixel readout region within the readout unit needs to be output in a staggered time sequence. For the region to be merged or the single-pixel readout region to be output within a readout unit, the transmission switch between the pixel within that region and the floating diffuser is turned on, while the transmission switches between other pixels and the floating diffuser are turned off, thus enabling the individual output of the pixel information for that region to be merged or the single-pixel readout region.

[0346] When implementing the second sub-mode using the method shown in Figure 19(b), the specific settings of the first regions corresponding to each region to be merged within the repeating unit can be referenced to ① to ④ in Figure 26(a). It should be noted that although each of the first regions in ① to ④ of Figure 26(a) shows two regions to be merged, the pixel information of each region to be merged within the same first region needs to be read out in a staggered manner. Similarly, when implementing the second sub-mode using the method shown in Figure 19(c), the specific settings of the first regions corresponding to each region to be merged within the repeating unit can be referenced to ① to ② in Figure 26(b); and when implementing the second sub-mode using the method shown in Figure 19(d), the specific settings of the first regions corresponding to each region to be merged within the repeating unit can be referenced to ① to ② in Figure 26(c).

[0347] Full pixel mode:

[0348] In one example, in full-pixel mode, specifically with all connection switches of pixel array 12 in the off state, the pixel information of each pixel in the second output area of ​​pixel array 12 can be output separately. This helps to maximize the conversion gain and minimize the noise of the pixel information output in full-pixel mode.

[0349] In practical implementation, the output of pixel information for each pixel within the same readout unit needs to be staggered in timing. For the pixel to be output within a readout unit, the transmission switch between that pixel and the floating diffuser is turned on, while the transmission switches between other pixels and the floating diffuser are turned off, thus enabling the individual output of pixel information for that pixel.

[0350] Option 2: The readout unit size is 2×2

[0351] In Scheme 2, the basic structure of the readout unit is shown in Figure 27. Each readout unit includes 2×2 pixels, and these 2×2 pixels share the same floating diffuser. When the structure of Figure 27 is applied to the pixel arrangement of Figure 17, the basic structure of the pixel array 12 is shown in Figure 28, wherein the floating diffusers of each pair of adjacent readout units are connected by a connection switch.

[0352] The specific circuit design of the structure in Figure 28 can be found in Figure 9, and will not be shown again here.

[0353] The specific implementation of different pixel output modes in Scheme 2 will be explained below.

[0354] First pixel merging mode:

[0355] As mentioned earlier, in the first pixel merging mode, each color unit constitutes a region to be merged. Based on Figure 28, it can be seen that within the extended unit, pixels of four colors share the same floating diffuser, making it impossible to read the pixel information of different regions to be merged within the extended unit simultaneously. Therefore, in practical applications, for an extended unit, the control circuit 11 can specifically control the connection switches to sequentially form the first region corresponding to each region to be merged, so that the pixel information of the four regions to be merged within the extended unit is output in an interleaved manner.

[0356] This application provides a specific design for the first region corresponding to each region to be merged within an extended unit, ensuring that each region to be merged contains the same number of output units within its first region. The following explanation is based on Figures 29(a) to 29(d). In Figures 29(a) to 29(d), the number of columns in the regions to be merged and the number of columns in the readout units are identified by the letter 'k', where 'k' is a natural number and is updated from left to right according to the rule k = k + 1. The number of rows in the regions to be merged and the number of rows in the readout units are identified by the letter 'j', where 'j' is a natural number and is updated from top to bottom according to the rule j = j + 1.

[0357] Referring to Figure 29(a), the first region corresponding to the region to be merged in the 2j+1 row and the 2k+1 column is formed by connecting the following readout units: the readout units in the 3k+1 to 3k+2 columns of the 3j+1 row, and the readout units in the 3k+1 to 3k+2 columns of the 3j+2 row (in an extended unit, the first region corresponding to the first region to be merged in the first row includes: the first two readout units in the first row and the first two readout units in the second row within the range of the extended unit).

[0358] Referring to Figure 29(b), the first region corresponding to the region to be merged in row 2j+1 and column 2k+2 is formed by connecting the following readout units: the readout units in columns 3k+2 to 3k+3 of row 3j+1, and the readout units in columns 3k+2 to 3k+3 of row 3j+2 (in an extended unit, the first region corresponding to the second region to be merged in the first row includes: the second to third readout units in the first row and the second to third readout units in the second row within the range of the extended unit).

[0359] Referring to Figure 29(c), the first region corresponding to the region to be merged in the 2j+2nd row and the 2k+1st column is formed by connecting the following readout units: the readout units in the 3k+1st to 3k+2nd columns of the 3j+2nd row, and the readout units in the 3k+1st to 3k+2nd columns of the 3j+3rd row (in an extended unit, the first region corresponding to the first region to be merged in the second row includes: the first two readout units in the second row and the first two readout units in the third row within the range of the extended unit).

[0360] Referring to Figure 29(d), the first region corresponding to the region to be merged in the 2j+2nd row and the 2k+2nd column is formed by connecting the following readout units: the readout units in the 3k+2nd to 3k+3rd columns of the 3j+2nd row, and the readout units in the 3k+2nd to 3k+3rd columns of the 3j+3rd row (in an extended unit, the first region corresponding to the second region to be merged in the second row includes: the second to third readout units in the second row and the second to third readout units in the third row within the range of the extended unit).

[0361] As can be seen from the above, based on the design of Figures 29(a) to 29(d), the number of readout units contained in the first region corresponding to each region to be merged in the pixel array 12 is 4, which helps to ensure the consistency of the conversion gain between the pixel information output in the first pixel merging mode, thereby helping to ensure the image quality of the final generated image.

[0362] It is readily apparent that the shape of each first region in Figures 29(a) to 29(d) is consistent with that of first region a shown in Figure 14. As discussed earlier with respect to Figure 14, to connect four adjacent readout units in two rows and two columns, all four connection switches (including two first connection switches and two second connection switches) between these four readout units can be turned on, or only any three of the four connection switches can be turned on. In a preferred embodiment of this application, in order to ensure that the total capacitance of the floating diffusion section after the connection of each first region is consistent and minimized as much as possible, so as to ensure the consistency of the conversion gain between the pixel information output in the first pixel merging mode and reduce signal noise, referring to the schematic diagrams in Figures 29(a) to 29(d), each first region shown in the figure can be connected by turning on two first connection switches and one second connection switch, or each first region shown in the figure can be connected by turning on one first connection switch and two second connection switches.

[0363] It is readily apparent that Figures 29(a) to 29(d) actually depict 2×2 adjacent expansion units. In a column-level ADC product architecture, in one possible implementation of this application, referring to the illustrations in Figures 29(a) to 29(d), for two adjacent expansion units in the column direction, the pixel information of a region to be merged within one expansion unit is synchronously output with the pixel information of the region to be merged at the same position within another expansion unit through different column lines. The specific implementation of the synchronous output can be found in the preceding explanation of Figure 16. Alternatively, referring to the illustrations in Figures 29(a) to 29(d), the pixel information of the regions to be merged at the same position within each expansion unit in the same row can be synchronously output through different column lines.

[0364] The output processes shown in the four figures 29(a) to 29(d) need to be staggered in timing, and the specific order can be set as needed.

[0365] First sub-pattern:

[0366] Referring to Figures 18 and 28, it can be seen that in the first sub-mode, each readout unit within the first output area of ​​the pixel array 12 is a region to be merged. Therefore, charge domain merging of pixels within each region to be merged can be achieved without turning on the connection switches. Thus, in one embodiment of this application, pixel information of each region to be merged within the pixel array 12 can be output even when all connection switches within the pixel array 12 are in the off state.

[0367] Second sub-pattern:

[0368] Combining Figures 18 and 28, it can be seen that when the second sub-mode is implemented using the method in Figure 18(a), the range of the area to be merged within a color unit is exactly the same as the range of a readout unit, and there is no situation where the area to be merged crosses the readout unit. Therefore, even when all the connection switches in the pixel array 12 are in the off state, the pixel information of each area to be merged and each single pixel readout area can be output separately.

[0369] When the second sub-mode is implemented using the method in Figure 18(b), the first region corresponding to each region to be merged within the expansion unit can be specifically set up as shown in ① to ④ of Figure 30(a); when the second sub-mode is implemented using the method in Figure 18(c), the first region corresponding to each region to be merged within the expansion unit can be specifically set up as shown in ① to ② of Figure 30(b); when the second sub-mode is implemented using the method in Figure 18(d), the first region corresponding to each region to be merged within the expansion unit can be specifically set up as shown in ① to ② of Figure 30(c).

[0370] Similarly to Scheme 1, for the single-pixel readout area in the second sub-mode, the pixel information of the single-pixel readout area can be output when the readout unit where the single-pixel readout area is located and its adjacent readout units are disconnected. This helps to make the conversion gain of the output single-pixel readout area pixel information as high as possible and the noise as low as possible.

[0371] Full pixel mode:

[0372] The full-pixel mode in Scheme 2 is similar to that in Scheme 1, and can be referred to the description above.

[0373] Option 3: The readout unit size is 1×3

[0374] In Scheme 3, the basic structure of the readout unit is shown in Figure 31. Each readout unit includes 3×1 pixels, and these 3×1 pixels share the same floating diffuser. When the structure of Figure 31 is applied to the pixel arrangement of Figure 17, the basic structure of the pixel array 12 is shown in Figure 32, wherein the floating diffusers of each pair of adjacent readout units are connected by a connection switch.

[0375] Figure 33 shows a schematic diagram of the circuit structure within the pixel array 12 when the structure in Figure 31 is applied. It illustrates four adjacent readout units arranged in two rows and two columns. Each readout unit contains 3×1 pixels, and the photodiodes PD of each 3×1 pixel are connected to the same floating diffuser FD via connection switches TX. The floating diffuser FD of each readout unit is connected to a readout line OUT via an amplifier AMP and a selection switch SEL, and is connected to a reset voltage via a reset switch RST. Furthermore, the floating diffusers FD of every two adjacent readout units are connected via a selection switch SW.

[0376] When applying a column-level ADC architecture, the floating diffuser FD in each readout unit of the same column will be connected to the same readout line. For example, in Figure 33, the floating diffuser FD in the readout unit of the first column is connected to readout line OUT1, and the floating diffuser FD in the readout unit of the second column is connected to readout line OUT2.

[0377] The specific implementation of different pixel output modes in Scheme 3 will be explained below.

[0378] First pixel merging mode:

[0379] As can be seen from Figures 18 and 32, in the first pixel merging mode, each region to be merged is specifically composed of three adjacent readout units in the row direction. Therefore, in one embodiment of this application, referring to the illustration in Figure 34(a), for each region to be merged, the three readout units covered by the region to be merged are connected by a connection switch to form a first region consistent with the range of the region to be merged, so that the pixel information of each region to be merged is output from the corresponding first region.

[0380] In a column-level ADC product architecture, when outputting pixel information based on the method shown in Figure 34(a), in one example, to improve output efficiency, pixel information within the same column of adjacent rows to be merged can be output synchronously through different column lines. In another example, to facilitate row control, referring to the illustration in Figure 34(b), the pixel information of different columns to be merged can be output interleaved by alternately executing the readout processes shown in ① and ②. Alternatively, referring to the illustration in Figure 34(b), pixel information of each region to be merged within the same column can be output synchronously through different column lines.

[0381] First sub-pattern:

[0382] Combining Figures 18 and 32, it can be seen that in the first sub-mode, there is a situation where a region to be merged spans two rows and two columns of four adjacent readout units (for example, the central 2×2 pixel region to be merged in the structure of Figure 32). Therefore, in one embodiment of this application, to ensure that the first region corresponding to each region to be merged in the first sub-mode contains the same number of readout units, the first region corresponding to each region to be merged can be set as shown in Figure 35(a). Figure 35(a) uses the letter k to identify the number of columns of the region to be merged and the number of columns of the readout units, where k is a natural number and is updated from left to right according to the rule k = k + 1. It also uses the letter j to identify the number of rows of the region to be merged and the number of rows of the readout units, where j is a natural number and is updated from top to bottom according to the rule j = j + 1.

[0383] As shown in Figure 35(a), the regions to be merged in columns 3k+1 of rows 3j+1 to 3j+3 correspond to the same first region. This first region is formed by connecting the following readout units: the readout units in columns 6k+1 to 6k+2 of row 2j+1, and the readout units in columns 6k+1 to 6k+2 of row 2j+2 (within one extended unit, each region to be merged in the first column corresponds to the same first region, which includes: the first two readout units of the first row and the first two readout units of the second row within the extended unit's range); the regions to be merged in columns 3k+2 of rows 3j+1 to 3j+3 correspond to the same first region. This first region is formed by connecting the following readout units: the readout units in columns 6k+3 to 6k+4 of row 2j+1, and the readout units in columns 6k+1 to 6k+2 of row 2j+2. The readout units in columns 6k+3 to 6k+4 (within one extended unit, each region to be merged in the second column corresponds to the same first region, which includes: the third and fourth readout units in the first row and the third and fourth readout units in the second row within the extended unit); the regions to be merged in columns 3k+3 of rows 3j+1 to 3j+3 correspond to the same first region, which is formed by connecting the following readout units: the readout units in columns 6k+5 to 6k+6 of row 2j+1, and the readout units in columns 6k+5 to 6k+6 of row 2j+2 (within one extended unit, each region to be merged in the third column corresponds to the same first region, which includes: the fifth and sixth readout units in the first row and the fifth and sixth readout units in the second row within the extended unit).

[0384] In one possible implementation of this application, for each first region illustrated in FIG35(a), referring to the illustration in the figure, it can be connected by turning on one first connection switch and two second connection switches, or it can be connected by turning on two first connection switches and one second connection switch.

[0385] As mentioned earlier, only one pixel information item can be output from a given first region at a time. Therefore, in the pixel information output process based on the illustration in Figure 35(a), the pixel information of each region to be merged corresponding to the same first region needs to be output in a staggered time sequence. When using a column-level ADC product architecture, in one embodiment of this application, to improve output efficiency, the pixel information of one region to be merged contained in each first region of the same column can be output synchronously. For example, to facilitate row control, referring to the illustration in Figure 35(b), the pixel information of the regions to be merged at the same position in each first region of the same row can be output synchronously by staggering the execution processes illustrated in ①, ②, and ③.

[0386] Second sub-pattern:

[0387] As can be seen from Figures 19 and 32, in the second sub-mode, the area to be merged within each color unit spans two adjacent readout units in the row direction. Therefore, the first region can be formed by connecting these two readout units to output the pixel information of the corresponding area to be merged.

[0388] In one example, for the actual layout of the four regions to be merged in Figures 19(a) to (d), the first region corresponding to each region to be merged can be set by referring to the illustrations in Figures 36(a) to (d).

[0389] Similar to Scheme 1 and Scheme 2, for each single-pixel readout area, the pixel information of that single-pixel readout area can be output when the readout unit where each single-pixel readout area is located and its adjacent readout units are disconnected.

[0390] Furthermore, as mentioned in Scheme 1, when a single readout unit contains two or more single-pixel readout regions, the pixel information of each single-pixel readout region within the readout unit needs to be output in a staggered manner in terms of timing.

[0391] Full pixel mode:

[0392] The full-pixel mode in Scheme 3 is similar to that in Scheme 1, as described above.

[0393] Option 4: Readout unit size is 3×3

[0394] In Scheme 1, the basic structure of the readout unit is shown in Figure 37. Each readout unit includes 3×3 pixels, and these 3×3 pixels share the same floating diffuser. When the structure of Figure 37 is applied to the pixel arrangement of Figure 17, the basic structure of the pixel array 12 is shown in Figure 38, wherein the floating diffusers of each pair of adjacent readout units are connected by a connection switch.

[0395] Figure 39 shows a schematic diagram of the circuit structure within the pixel array 12 when the structure in Figure 22 is applied. It illustrates four adjacent readout units arranged in two rows and two columns. Each readout unit contains 3×3 pixels, and the photodiodes PD of each 3×3 pixel are connected to the same floating diffuser FD via connection switches TX. The floating diffuser FD of each readout unit is connected to a readout line OUT via an amplifier AMP and a selection switch SEL, and is connected to a reset voltage via a reset switch RST. Furthermore, the floating diffusers FD of every two adjacent readout units are connected via a selection switch SW.

[0396] When used in a column-level ADC architecture, the floating diffuser FD in each readout unit of the same column is connected to the same readout line. For example, in Figure 39, the floating diffuser FD in the readout unit of the first column is connected to readout line OUT1, and the floating diffuser FD in the readout unit of the second column is connected to readout line OUT2.

[0397] The specific implementation of different pixel output modes in Scheme 4 will be explained below.

[0398] First pixel merging mode:

[0399] As can be easily seen from Figures 18 and 38, in the first pixel merging mode, each readout unit within the pixel array 12 constitutes a region to be merged. Therefore, in one embodiment of this application, in order to maximize the conversion gain of each pixel information read out in the first pixel merging mode and ensure the output signal-to-noise ratio of each pixel information, it is specifically possible to control all connection switches within the pixel array 12 to be in the off state, so as to output the pixel information in each region to be merged.

[0400] First sub-pattern:

[0401] As can be seen from Figures 18 and 38, in the first sub-mode, there is a case where a region to be merged spans two rows and two columns, encompassing four adjacent readout units (for example, the 2×2 pixel region to be merged in the center of the structure in Figure 38). Therefore, in one embodiment of this application, to ensure that the first region corresponding to each region to be merged contains the same number of readout units, the first region corresponding to each region to be merged in the first sub-mode can be set with reference to the example in Figure 40(a).

[0402] Referring to Figure 40(a), each region to be merged within an expansion unit corresponds to the same first region, and the first region is consistent with the range of the expansion unit. In one example, referring to the illustration in Figure 40(a), each first region can be formed by connecting one first connection switch and two second connection switches, or by connecting two first connection switches and one second connection switch.

[0403] As mentioned earlier, only one pixel information item can be read from one first region at a time. Therefore, when setting the first regions corresponding to each region to be merged based on the schematic diagram in Figure 40(a), for each first region, the pixel information of the 3×3 regions to be merged contained therein needs to be output in a staggered time sequence. In one example, the pixel information of these 3×3 regions to be merged can be output sequentially in a zigzag order according to the arrow direction marked in Figure 40(a). Of course, the actual output order is not limited to this.

[0404] When using a column-level ADC product architecture, one possible implementation, as illustrated in Figure 40(b), involves synchronously outputting pixel information of the same position in the same column expansion unit of two adjacent rows through different column lines. Alternatively, also as illustrated in Figure 40(b), pixel information of the same position in the same column expansion unit can be synchronously output through different column lines.

[0405] Second sub-pattern:

[0406] As can be easily understood by referring to Figures 18 and 38, since the regions to be merged in the second sub-mode are each located within their corresponding color units, and in Scheme 4, each pixel within each color unit constitutes a readout unit, there is no situation where the regions to be merged cross readout units. Therefore, in one embodiment of this application, in order to minimize the signal noise of the pixel information output in the second sub-mode, the connection switches in the pixel array 12 can be controlled to be in an open state, and the pixel information of each region to be merged and the single pixel readout region is output in this open state.

[0407] Since only one pixel information item can be output from a single readout unit at a time, in practical applications, the region to be merged within the same readout unit needs to be output in a staggered timing manner along with the five single-pixel readout regions. The specific output order can be configured in various ways in practical applications, and this embodiment does not limit this.

[0408] In one example, under the product architecture of a column-level ADC, when the second sub-mode is implemented using the scheme in Figure 19(a), the pixel information of each region to be merged and the single-pixel readout region within an extended unit can be read in the order of ① to ⑥ in Figure 41(a). It should be noted that although ① shows both a region to be merged and a single-pixel readout region within the same color unit, the output of pixel information for these two regions should be interleaved, and the same applies to ⑤; similarly, although ③ shows three single-pixel readout regions within the same color unit, the output of pixel information for these three regions should be interleaved, and the same applies to ④. When the second sub-mode is implemented using the schemes in Figure 19(b), (c), or (d), the pixel information of each region to be merged and the single-pixel readout region within an extended unit can be read in the order of ① to ⑥ in Figure 41(b), Figure 41(c), or Figure 41(d), respectively, and the basic principle is similar to that in Figure 41(a).

[0409] Full pixel mode:

[0410] The full-pixel mode in Scheme 4 is similar to that in Scheme 1, and can be referred to the description above.

[0411] In summary, by configuring the readout unit based on schemes one through four of this application, charge domain merging under two different zoom magnifications—the first pixel merging mode and the second pixel merging mode—can be achieved, as well as a full-pixel mode that outputs pixel information for a single pixel individually. This provides different zoom magnifications of 1×, 2×, and 3× for the user to choose from. In practical applications, it should be noted that an image sensing device has the characteristic that the smaller the total capacitance of the connected floating diffuser, the greater the conversion gain of the pixel information output from the floating diffuser, and the more helpful it is in reducing the original signal-to-noise ratio (SNR) level. Therefore, among the four schemes mentioned above, schemes one and two are relatively more helpful in ensuring low SNR output at 2× magnification, while schemes three and four are relatively more helpful in ensuring low SNR output at 3× magnification. The specific design scheme can be determined according to the specific emphasis in the actual application. Furthermore, for the first and second sub-modes under the second pixel merging mode, in order to ensure high conversion gain and low signal-to-noise ratio of the output signal, Scheme 1 and Scheme 2 are preferably implemented using the first sub-mode or the second sub-mode shown in Figure 19(a), while Scheme 4 is preferably implemented using the second sub-mode.

[0412] Based on the same inventive concept, this application also provides an image sensing method applied to a control circuit in an image sensing device. The image sensing device further includes a pixel array, multiple readout lines, and an information processing unit. The pixel array includes multiple readout units arranged in an array, each readout unit including multiple pixels. Each pixel in a readout unit shares a floating diffuser, and the charge transfer between each pixel and the floating diffuser is independently controlled. The floating diffusers of every two adjacent readout units are connected by a connection switch. The floating diffusers of each readout unit are respectively connected to the readout lines by selection switches. The information processing circuit is connected to the multiple readout lines. The control circuit supports multiple pixel merging modes. A pixel merging mode means that the pixel information of each region to be merged in the pixel array is output separately by pixel merging. A region to be merged contains multiple pixels, and the regions to be merged in different pixel merging modes are not completely the same. The method includes:

[0413] The control circuit connects the floating diffusion sections in each first readout unit to the region to be merged in the current pixel merging mode via a connection switch. The first readout unit contains the pixels in the region to be merged, so that the charge sensed by each pixel in the region to be merged is merged and accumulated in the connected floating diffusion section. The selection switch between the connected floating diffusion section and the first readout line is set to the on state, and the first readout line is connected to the connected floating diffusion section, so that the pixel information generated by merging each pixel in the region to be merged is output to the information processing unit through the first readout line, so that the information processing unit generates an output image based on the pixel information output from the readout line in the current pixel merging mode.

[0414] For details regarding the pixel readout method and its beneficial effects, please refer to the preceding description of the embodiment of the image sensing device; it will not be repeated here.

[0415] It should be noted that some structural or methodological features may be shown in a specific arrangement and / or order in the accompanying drawings. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0416] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0417] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application. < / tx9> < / tx7> < / sw1> < / swi> < / seli> < / rsti> < / txi>

Claims

1. An image sensing device, characterized in that, include: A pixel array includes multiple readout units arranged in an array, each readout unit including multiple pixels; each pixel in a readout unit shares a floating diffuser, and the charge transfer between each pixel and the floating diffuser is independently controlled; the floating diffusers of every two adjacent readout units are connected by a connection switch. Multiple readout lines, with the floating diffusion section of each readout unit connected to the readout lines via a selection switch; The control circuit supports multiple pixel merging modes. Each pixel merging mode outputs pixel information of each region to be merged within the pixel array through pixel merging. A region to be merged contains multiple pixels, and the regions to be merged under different pixel merging modes are not identical. The control circuit is configured to: for the region to be merged under the current pixel merging mode, control the connection switch to connect the floating diffusion sections in each first readout unit. The first readout unit is a readout unit containing pixels within the region to be merged, so that the charge sensed by each pixel within the region to be merged is merged and accumulated in the connected floating diffusion sections. The selection switch between the connected floating diffusion section and the first readout line is set to the on state, and the first readout line is connected to the connected floating diffusion section, so that the pixel information generated by merging each pixel in the area to be merged is output through the first readout line; An information processing unit is connected to the plurality of readout lines and is configured to generate an output image based on the pixel information output from the readout lines in the current pixel merging mode.

2. The image sensing device according to claim 1, characterized in that, Each of the aforementioned pixels is connected to the floating diffuser via a transmission switch; The control circuit is specifically configured to: for the region to be merged in the current pixel merging mode, control the connection switch to connect the floating diffusion section in the adjacent readout units, so that the first region formed by the interconnected adjacent readout units includes the region to be merged; for each pixel in the first region, turn on the transmission switch between the pixel included in the region to be merged and the floating diffusion section, and turn off the transmission switch between the other pixels and the floating diffusion section, so that the charge sensed by each pixel in the region to be merged is merged and accumulated in the connected floating diffusion section.

3. The image sensing device according to claim 2, characterized in that, In a pixel merging mode, the number of readout units contained in the first region corresponding to each region to be merged is the same.

4. The image sensing device according to claim 3, wherein In a pixel merging mode, the number of first connection switches in the conducting state and the number of second connection switches in the conducting state are the same in the first region corresponding to each region to be merged; the first connection switch is the connection switch of the floating diffusion part in two adjacent readout units in the row direction, and the second connection switch is the connection switch of the floating diffusion part in two adjacent readout units in the column direction.

5. The image sensing device according to claim 2, wherein The current pixel merging mode contains multiple groups to be merged, and each group to be merged contains one or more regions to be merged; each region to be merged in the same group corresponds to a different first region, and there is no overlap between any two first regions. The control circuit is specifically configured to: synchronously output the pixel information of each region to be merged within the same merge group through different readout lines, and interleave the pixel information of the regions to be merged within different merge groups in terms of timing.

6. The image sensing device according to claim 5, wherein The readout line is a column line, and the floating diffusion section of each readout unit in the same column is connected to the same column line, while the floating diffusion section of readout units in different columns is connected to different column lines. The group to be merged includes a first region to be merged and a second region to be merged located in the same column. The first region corresponding to at least one of the first region to be merged and the second region to be merged covers at least two readout units in the row direction. The control circuit is specifically configured to output the pixel information of the first region to be merged and the pixel information of the second region to be merged synchronously through different column lines.

7. The image sensing device according to claim 2, characterized in that, The pixels in the pixel array are arranged as multiple extended units in an array, each of the extended units includes 2×2 color units arranged in a Bayer pattern, and each of the color units includes 3×3 pixels of the same color; the control circuit supports: a first pixel merging mode at 1x zoom, a second pixel merging mode at 2x zoom, and a full pixel mode at 3x zoom. In the first pixel merging mode, the pixels in the pixel array are combined into multiple 3×3 regions to be merged; The second pixel merging mode includes a first sub-mode and / or a second sub-mode. In the first sub-mode, the pixel array contains a first output area, and the pixels in the first output area are combined into multiple 2×2 merged areas. In the second sub-mode, the pixels in each color unit in the first output area are combined into a 2×2 merged area and five 1×1 single pixel readout areas. In the full-pixel mode, the pixel array includes a second output area, and each pixel in the second output area serves as a single-pixel readout area.

8. The image sensing device according to claim 7, characterized in that, Each of the readout units comprises 2×4 pixels, or each of the readout units comprises 2×2 pixels; In the first pixel merging mode, each color unit within an expansion unit constitutes a region to be merged; the control circuit is specifically configured to: for an expansion unit, control the connection switch to sequentially connect to form the first region corresponding to each region to be merged, and output the corresponding pixel information.

9. The image sensing device according to claim 8, characterized in that, Each readout unit includes 2×4 pixels. In the first pixel merging mode, two adjacent extension units in the column direction are a first type of extension unit and a second type of extension unit, respectively. The first pixel of the first row in the first type of extension unit is the first pixel of the first row in the first readout unit; the first pixel of the first row in the second type of extension unit is the first pixel of the third row in the second readout unit; the first readout unit and the second readout unit are two adjacent readout units in the column direction. In the first type of extended unit, the first region corresponding to the first region to be merged in the first row includes: the first three readout units in the first row within the scope of the first type of extended unit, and the first readout unit in the second row; The first region corresponding to the second region to be merged in the first row includes: the first three readout units in the first row within the scope of the first type of extended unit, and the third readout unit in the second row; The first region corresponding to the first region to be merged in the second row includes: the first two readout units in the first row within the scope of the first type of extended unit, and the first two readout units in the second row; The first region corresponding to the second region to be merged in the second row includes: the second to third readout units in the first row within the scope of the first type of extended unit, and the second to third readout units in the second row; In the second type of extended unit, the first region corresponding to the first region to be merged in the first row includes: the first two readout units of the first row within the range of the second type of extended unit, and the first two readout units of the second row; The first region corresponding to the second region to be merged in the first row includes: the second to third readout units in the first row within the range of the second type of extended units, and the second to third readout units in the second row; The first region corresponding to the first region to be merged in the second row includes: the first readout unit in the first row within the range of the second type of extended unit, and the first three readout units in the second row; The first region corresponding to the second region to be merged in the second row includes: the third readout unit in the first row within the range of the second type of extended unit, and the first three readout units in the second row.

10. The image sensing device according to claim 9, wherein In the first pixel merging mode, the control circuit is configured to: in a first type of expansion unit and a second type of expansion unit in the same column as the first type of expansion unit and located in the next row of the first type of expansion unit, synchronously output the pixel information of the first region to be merged in the first row of the first type of expansion unit and the pixel information of the second region to be merged in the first row of the second type of expansion unit through different readout lines; The pixel information of the second region to be merged in the first row of the first type of expansion unit and the pixel information of the first region to be merged in the first row of the second type of expansion unit are output synchronously through different readout lines; The pixel information of the first region to be merged in the second row of the first type of expansion unit and the pixel information of the second region to be merged in the second row of the second type of expansion unit are output synchronously through different readout lines; The pixel information of the second region to be merged in the second row of the first type of expansion unit and the pixel information of the first region to be merged in the second row of the second type of expansion unit are output synchronously through different readout lines.

11. The image sensing device according to claim 9, characterized in that, In the first pixel merging mode, the number of first connection switches in the conducting state in the first region corresponding to each region to be merged in the pixel array is two, and the number of second connection switches in the conducting state is one.

12. The image sensing device according to claim 8, wherein Each readout unit includes 2×4 pixels. In the first sub-mode, a readout unit of the first output area includes two regions to be merged. The control circuit is specifically configured to: control all connection switches in the pixel array to be disconnected, and for each readout unit in the first output area, output the pixel information of the two regions to be merged in the readout unit in a staggered manner in time.

13. The image sensing device according to claim 8, characterized in that, Each readout unit includes 2×2 pixels. In the first pixel merging mode, in an extended unit, the first region corresponding to the first region to be merged in the first row includes: the first two readout units in the first row within the range of the extended unit, and the first two readout units in the second row. The first region corresponding to the second region to be merged in the first row includes: the second to third readout units in the first row within the range of the extended unit, and the second to third readout units in the second row; The first region corresponding to the first region to be merged in the second row includes: the first two readout units of the second row within the range of the extended unit, and the first two readout units of the third row; The first region corresponding to the second region to be merged in the second row includes: the second to third readout units in the second row within the range of the extended unit, and the second to third readout units in the third row.

14. The image sensing device according to claim 13, characterized in that, The readout line is a column line, and the floating diffusion part of each readout unit in the same column is connected to the same column line, while the floating diffusion parts of readout units in different columns are connected to different column lines. In the first pixel merging mode, the control circuit is specifically configured to: for two adjacent expansion units in the column direction, synchronously output the pixel information of a region to be merged in one expansion unit and the pixel information of a region to be merged at the same position in the other expansion unit through different column lines.

15. The image sensing device according to claim 13, wherein In the first pixel merging mode, the number of first connection switches in the conducting state in the first region corresponding to each region to be merged in the pixel array is two, and the number of second connection switches in the conducting state is one; or, the number of first connection switches in the conducting state in the first region corresponding to each region to be merged in the pixel array is one, and the number of second connection switches in the conducting state is two.

16. The image sensing device according to claim 8, characterized in that, Each readout unit includes 2×2 pixels. In the first sub-mode, each readout unit in the first output area is a region to be merged. The control circuit is specifically configured to: control all connection switches in the pixel array to be disconnected, and output the pixel information of the corresponding region to be merged in each readout unit.

17. The image sensing device according to claim 7, wherein Each of the readout units comprises 1×3 pixels; In the first pixel merging mode, for each region to be merged contained in the pixel array, the range of the first region corresponding to each region to be merged is consistent with the range of each region to be merged.

18. The image sensing device of claim 17, wherein The readout line is a column line, and the floating diffusion section of each readout unit in the same column is connected to the same column line, while the floating diffusion section of readout units in different columns is connected to different column lines. In the first pixel merging mode, the control circuit is specifically configured to: synchronously output the pixel information of each region to be merged in the same column through different column lines; and interleave the pixel information of each region to be merged in different columns in a timing sequence.

19. The image sensing device of claim 17, wherein In the first sub-mode, an expansion unit includes 3×3 regions to be merged; within an expansion unit, each region to be merged in the first column corresponds to the same first region, which includes: the first two readout units in the first row and the first two readout units in the second row within the scope of the expansion unit; each region to be merged in the second column corresponds to the same first region, which includes: the third to fourth readout units in the first row and the third to fourth readout units in the second row within the scope of the expansion unit; each region to be merged in the third column corresponds to the same first region, which includes: the fifth to sixth readout units in the first row and the fifth to sixth readout units in the second row within the scope of the expansion unit.

20. The image sensing device of claim 19, wherein In the first sub-mode, the control circuit is specifically configured to: synchronously output the pixel information of three regions to be merged corresponding to different first regions for an expansion unit, and interleave the pixel information of different regions to be merged corresponding to the same first region in a timing manner.

21. The image sensing device according to claim 8 or 17, wherein In the second sub-mode, the control circuit is specifically configured to: for the region to be merged, control the connection switch to connect the floating diffusion section of the readout unit containing the pixels in the region to be merged, so as to output the pixel information of the region to be merged; for the single-pixel readout region, control the readout unit where the single-pixel readout region is located to disconnect from its adjacent readout units, and output the pixel information of the single-pixel readout region.

22. The image sensing device according to claim 7, wherein Each of the readout units comprises 3×3 pixels; In the first sub-mode, the range of the first region corresponding to each region to be merged within an expansion unit is consistent with the range of the expansion unit; the control circuit is specifically configured to: for an expansion unit, output the pixel information of each region to be merged in an interleaved manner in time.

23. The image sensing device according to claim 7, wherein Each of the readout units comprises 3×3 pixels; In the second sub-mode, the control circuit is specifically configured to: control all connection switches in the pixel array to be disconnected, and for each readout unit, output the pixel information of one region to be merged and five single-pixel readout regions in a staggered manner in time.

24. The image sensing device according to claim 22 or 23, wherein In the first pixel merging mode, each readout unit in the pixel array is a region to be merged, and the control circuit is specifically configured to: control all connection switches in the pixel array to be disconnected, and output pixel information of the corresponding region to be merged in each readout unit.

25. The image sensing device of claim 8, wherein In the second pixel merging mode, the information processing unit is configured to: obtain an initial image by performing pixel reordering processing on the pixel information output from the readout line, and convert the initial image into an output image conforming to the Bayer array arrangement.

26. The image sensing device of claim 25, wherein In the second pixel merging mode, the information processing unit is configured to: input the initial image into a pre-trained image conversion model to obtain the output image; The image conversion model is trained based on the following process: Acquire a first sample image of the pixel array sensed for a calibrated scene in the first sub-mode or a second sample image sensed for a calibrated scene in the second sub-mode; A reference image sensed by a reference pixel array for the calibrated scene is obtained; the reference pixel array conforms to the Bayer array arrangement and its resolution is consistent with the target resolution of the output image; Using the first sample image or the second sample image as input data, and combining the reference image corresponding to the sample image, the initial image conversion model is trained to obtain the image conversion model corresponding to the first sub-mode or the second sub-mode.

27. The image sensing device of claim 25, wherein In the first sub-mode, the information processing unit is configured to: perform color conversion and up-interpolation conversion on the first initial image in the first sub-mode to obtain a first output image that conforms to the Bayer array arrangement and has the same resolution as the third output image; the third output image is: an image obtained by the information processing module based on the pixel information output by the pixel array in the first pixel merging mode.

28. The image sensing device of claim 25, wherein In the second sub-mode, the information processing unit is configured to: perform downsampling and mosaic rearrangement processing on the second initial image in the second sub-mode to obtain a second output image that conforms to the Bayer array arrangement and has the same resolution as the third output image; the third output image is: an image obtained by the information processing module based on the pixel information output by the pixel array in the first pixel merging mode.

29. An imaging device, characterized by Includes the image sensing device as described in any one of claims 1-28.

30. An image sensing method characterized by comprising: A control circuit is applied in an image sensing device, the image sensing device further comprising a pixel array, multiple readout lines, and an information processing unit; the pixel array includes multiple readout units arranged in an array, each readout unit including multiple pixels; each pixel of a readout unit shares a floating diffuser, and the charge transfer between each pixel and the floating diffuser is independently controlled; the floating diffusers of every two adjacent readout units are connected by a connection switch; the floating diffusers of each readout unit are respectively connected to a readout line by a selection switch; the information processing circuit is connected to the multiple readout lines; the control circuit supports multiple pixel merging modes, the pixel merging mode indicating that the pixel information of each region to be merged in the pixel array is output separately by pixel merging, a region to be merged contains multiple pixels, and the regions to be merged in different pixel merging modes are not completely the same; the method includes: The control circuit connects the floating diffusion sections within each first readout unit to the region to be merged in the current pixel merging mode via the connection switch. The first readout unit contains pixels within the region to be merged, causing the charges sensed by each pixel in the region to be merged to accumulate and merge in the connected floating diffusion sections. The selection switch between the connected floating diffusion section and the first readout line is set to the on state, and the first readout line is connected to the connected floating diffusion section, so that the pixel information generated by merging each pixel in the region to be merged is output to the information processing unit through the first readout line, so that the information processing unit generates an output image based on the pixel information output from the readout line in the current pixel merging mode.

Images