Method of operating a pixel circuit, image sensor and method of operating an image sensor

Abstract

Embodiments relate to a method of operating a pixel circuit (120) comprising a first capacitor (126) and a second capacitor (127). The first capacitor (126) is configured to be connected to the second capacitor (127), and the second capacitor (127) is configured to be connected to a readout source follower (130). The method comprising a sampling process and a readout process. The sampling process comprises sampling signal charges (S130) to the first capacitor (126). The readout process comprises reading out (S150) first charges stored in the second capacitor (127), connecting (S160) the first capacitor (126) to the second capacitor (127) so as to distribute the signal charges stored in the first capacitor (126) between the first and second capacitors (126, 127), and reading out (S170) second charges stored in the second capacitor (127). A signal level stored in the second capacitor (127) is maintained after reading out the second charges and before sampling signal charges to the first capacitor for a next frame.

Description

[0001] METHOD OF OPERATING A PIXEL CIRCUIT , IMAGE SENSOR AND METHOD OF OPERATING AN IMAGE SENSOR

[0002] BACKGROUND

[0003] CMOS image sensors comprising a photosensitive element and a readout circuit are increasingly employed in a variety of applications , among others , in virtual or augmented reality (VR, AR) applications . Eye tracking in AR and smart glasses is extremely demanding in terms of power consumption . Since AR or smart glasses are continuously worn and often supplied with power using a battery, attempts are made in order to reduce the power consumption of these devices . In this regard, event-based sensors are emerging in the markets . Event-based sensors have a speciali zed pixel architecture that detect changes of the image and only transmit information about the changes . Generally, attempts are being made to reduce power consumption of event-based sensors and to reduce the latency .

[0004] It is an obj ect of the present invention to provide an improved method of operating a pixel circuit , an improved image sensor and an improved method of operating an image sensor .

[0005] SUMMARY

[0006] According to embodiments , the above obj ect is achieved by the claimed matter according to the independent claims . Further developments are defined in the dependent claims .

[0007] Embodiments relate to a method of operating a pixel circuit . The pixel circuit comprises a first capacitor and a second capacitor . The first capacitor is configured to be connected to the second capacitor, and the second capacitor is configured to be connected to a readout source follower . The method comprising a sampling process and a readout process . The sampling process comprises sampling signal charges to the first capacitor . The readout process comprises reading out first charges stored in the second capacitor, connecting the first capacitor to the second capacitor so as to distribute the signal charges stored in the first capacitor between the first and second capacitors , and reading out second charges stored in the second capacitor . A signal level stored in the second capacitor is maintained after reading out the second charges and before sampling signal charges to the first capacitor for a next frame .

[0008] According to further embodiments , an image sensor comprises a photosensitive element , and a pixel circuit . The pixel circuit comprises a first capacitor and a second capacitor . The first capacitor is configured to be connected to the second capacitor, and the second capacitor is configured to be connected to a readout source follower . The pixel circuit is configured to be electrically coupled to the photosensitive element . The image sensor further comprises a data processing device configured to process a column output signal output by the pixel circuit to generate output data, and a controller configured to operate the pixel circuit in a first acquisition mode . In the first acquisition mode , the controller is configured to control the pixel circuit to perform a sampling process and a readout process . During the sampling process the controller is configured to control the pixel circuit to perform sampling signal charges to the first capacitor . During the readout process the controller is configured to control the pixel circuit to perform reading out first charges stored in the second capacitor, connecting the first capacitor to the second capacitor so as to distribute the signal charges stored in the first capacitor between the first and second capacitors , and reading out second charges stored in the second capacitor . A signal level stored in the second capacitor is maintained after reading out the second charges and before sampling signal charges to the first capacitor for a next frame .

[0009] According to embodiments , the data processing device is configured to detect a change based on a first di f ference between the first charges and the second charges read out from the second capacitor . The output data may include a detection result .

[0010] According to further embodiments , the data processing device may be configured to detect a change based on a second di f ference which is based on first di f ferences between consecutive frames .

[0011] According to embodiments , the controller may be configured to further operate the pixel circuit in a second acquisition mode to generate image data .

[0012] For example , the controller may be configured to switch from the first acquisition mode to the second acquisition mode after a predetermined number of frames has been acquired .

[0013] According to embodiments , the image sensor may further comprise a quality monitoring unit . The controller may be configured to switch from the first acquisition mode to the second acquisition mode based on a quality detected by the quality monitoring unit .

[0014] For example , the data processing device may be configured to generate a detection signal indicating whether a change has been detected, and the controller is configured to switch to the second mode , when a change has been detected . According to further embodiments , an image sensor system comprises the image sensor which has been explained above and a system processing unit . The system processing unit is configured to receive the output data transmitted by the data processing device .

[0015] For example , the data processing device may be configured to further compress the output data before transmitting them to the system processing unit .

[0016] According to embodiments , the system processing unit may be configured to reconstruct an actual image based on a first di f ference between the first charges and the second charges read out from the second capacitor .

[0017] Further embodiments relate to a method of operating an image sensor . The image sensor comprises a photosensitive element , and a pixel circuit comprising a first capacitor and a second capacitor . The first capacitor is configured to be connected to the second capacitor, and the second capacitor is configured to be connected to a readout source follower . The pixel circuit is configured to be electrically coupled to the photosensitive element . The image sensor further comprises a processing device configured to process an output signal output by the pixel circuit . The method comprises operating the pixel circuit in a first acquisition mode to detect a change . In the first acquisition mode , the pixel circuit performs a sampling process and a readout process . During the sampling process the pixel circuit performs sampling signal charges to the first capacitor . During the readout process the pixel circuit performs reading out first charges stored in the second capacitor, connecting the first capacitor to the second capacitor so as to distribute the signal charges stored in the first capacitor between the first and second capacitors , and reading out second charges stored in the second capacitor . A signal level stored in the second capacitor is maintained after reading out the second charges and before sampling signal charges to the first capacitor for a next frame .

[0018] For example , the pixel circuit may be configured to be operated in a second acquisition mode . The second acquisition mode may comprise controlling the pixel circuit to perform a sampling process and a readout process . During the sampling process the pixel circuit performs sampling a reset level to the first capacitor and the second capacitor, and, thereafter, sampling signal charges to the first capacitor . During the readout process the pixel circuit performs reading out reset charges stored in the second capacitor, connecting the first capacitor to the second capacitor so as to distribute the signal charges stored in the first capacitor between the first and second capacitors , and reading out second charges stored in the second capacitor .

[0019] For example , a change may be detected based on a first di f ference between the first charges and the second charges read out from the second capacitor .

[0020] According to further embodiments , a change may be detected based on a second di f ference which is based on first di f ferences between consecutive frames .

[0021] For example , the method may further comprise monitoring a quality of an output signal and performing the second acquisition mode depending on the quality .

[0022] For example , the data processing device may generate a detection signal indicating whether a change has been detected, and the mode is changed to the second mode when a change has been detected .

[0023] A further method is directed to a method of operating an image sensor system comprising an image sensor . The image sensor comprises a photosensitive element and a pixel circuit comprising a first capacitor and a second capacitor . The first capacitor is configured to be connected to the second capacitor, and the second capacitor is configured to be connected to a readout source follower . The pixel circuit is configured to be electrically coupled to the photosensitive element . The image sensor further comprises a data processing device configured to process an output signal output by the pixel circuit to generate output data . The method comprises operating the pixel circuit in a first acquisition mode to detect a change . In the first acquisition mode , the pixel circuit performs a sampling process and a readout process . During the sampling process the pixel circuit performs sampling signal charges to the first capacitor . During the readout process the pixel circuit performs reading out first charges stored in the second capacitor, connecting the first capacitor to the second capacitor so as to distribute the signal charges stored in the first capacitor between the first and second capacitors , and reading out second charges stored in the second capacitor . A signal level stored in the second capacitor is maintained after reading out the second charges and before sampling signal charges to the first capacitor for a next frame . The image sensor system further comprises a system processing unit configured to receive the output data from the data processing device .

[0024] The method may further comprise reconstructing, in the system processing unit , an actual image based on a first di f ference between the first charges and the second charges read out from the second capacitor .

[0025] According to embodiments , the method further comprises , by the data processing device , further compressing the output data before transmitting them to the system processing unit .

[0026] Further embodiments relate to an electronic device comprising the image sensor or the image sensor system which have been explained above .

[0027] For example , the electronic device may be selected from an eye tracking device , an AR device , a VR device , smart glasses , a surveillance camera, an inspection camera and an interface .

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this speci fication . The drawings illustrate the embodiments of the present invention and together with the description serve to explain the principles . Other embodiments of the invention and many of the intended advantages will be readily appreciated, as they become better understood by reference to the following detai led description . The elements of the drawings are not necessarily to scale relative to each other . Like reference numbers designate corresponding similar parts .

[0030] Fig . 1 is a schematic equivalent circuit diagram of a pixel forming part of an image sensor according to embodiments .

[0031] Fig . 2A is a timing diagram of signals for driving the pixel circuit illustrated in Fig . 1 according to a first acquisition mode when performing a sampling process . Fig. 2B is a timing diagram of driving signals for driving the pixel circuit illustrated in Fig. 1 according to the first acquisition mode when performing a readout process.

[0032] Fig. 3 is a schematic view of an image sensor and an image sensor system according to embodiments.

[0033] Fig. 4A is a timing diagram of signals for operating the pixel circuit illustrated in Fig. 1 according to a second acquisition mode when performing a sampling process.

[0034] Fig. 4B is a timing diagram of signals for operating the pixel circuit illustrated in Fig. 1 according to the second acquisition mode when performing a readout process.

[0035] Fig. 5A illustrates an application example of a method according to embodiments.

[0036] Fig. 5B summarizes a method of operating an image sensor according to embodiments.

[0037] Fig. 5C shows a schematic diagram of an eye.

[0038] Fig. 6A is an equivalent circuit diagram of a portion of a pixel according to further embodiments.

[0039] Fig. 6B is an equivalent circuit diagram of a portion of a pixel according to further embodiments.

[0040] Fig. 7 is a schematic view of an electronic device according to embodiments. DETAILED DESCRIPTION

[0041] In the following detailed description reference is made to the accompanying drawings, which form a part hereof and in which are illustrated by way of illustration specific embodiments in which the invention may be practiced. In this regard, directional terminology such as "top", "bottom", "front", "back", "over", "on", "above", "leading", "trailing" etc. is used with reference to the orientation of the Figures being described. Since components of embodiments of the invention can be positioned in a number of different orientations, the directional terminology is used for purposes of illustration and is in no way limiting. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope defined by the claims.

[0042] The description of the embodiments is not limiting. In particular, elements of the embodiments described hereinafter may be combined with elements of different embodiments.

[0043] As employed in this specification, the terms "coupled" and / or "electrically coupled" are not meant to mean that the elements must be directly coupled together - intervening elements may be provided between the "coupled" or "electrically coupled" elements. The term "electrically connected" may describe a low- ohmic electric connection between the elements electrically connected together.

[0044] According to further embodiments and where appropriate, the term "electrically connected" may mean that the respective elements are "directly connected" or are "directly and permanently connected" . Fig. 1 is an equivalent circuit diagram of a pixel 10 forming part of an image sensor according to embodiments. As is shown, the pixel 10 illustrated in Fig. 1 comprises a photosensitive element 110 that is configured to generate charges from incident electromagnetic radiation 15. For example, the photosensitive element 110 may comprise a photodiode, for example, a pinned photodiode or any other suitable photosensitive element. The image sensor further comprises a pixel circuit 120 which is connected to the photosensitive element 110. A transfer transistor 121 is configured to control a transfer of the charges from the photosensitive element 110 to a floating diffusion 134. For example, the floating diffusion 134 may be implemented as or may comprise a capacitor and may implement a sense node. By applying a corresponding voltage to a gate electrode of the transfer transistor 121, charges generated by the photosensitive element 110 may be transferred to the floating diffusion 134.

[0045] The pixel circuit 120 further may comprise a reset transistor 122 for resetting the floating diffusion 134. Moreover, the pixel circuit 120 may comprise a source follower 123. A gate electrode of the source follower 123 may be electrically coupled to the floating diffusion 134. A terminal, e.g. a source terminal of the source follower 123 may be electrically connected to a node which may be connected to a precharge transistor 124. The node may be further electrically coupled to a first node 132 via a first sampling transistor 125. The first node 132 may be electrically coupled to a first capacitor 126 and to a second node 133 via a second sampling transistor 128. The second node 133 is electrically coupled to a second capacitor 127. The second node 133 may be further electrically coupled to a gate electrode of a readout source follower 130. A terminal, e.g. a source terminal of the readout source follower 130 may be electrically coupled to a column output 115 via a selection transistor 131. The pixel circuit 120 illustrated in Fig. 1 is only by way of example and various components may be dispensed with, e.g. the precharge transistor 124. As is shown in Fig. 1, the pixel circuit 120 comprises a first capacitor and a second capacitor 126,

[0046] 127. The first capacitor 126 is configured to be connected to the second capacitor 127, e.g. via the second sampling transistor

[0047] 128. Further, the second capacitor is configured to be connected to the readout source follower 130.

[0048] When the image sensor comprising elements illustrated in Fig. 1 is to be operated in a first acquisition mode, changes of the image are to be detected. As a result, only information about the changes is to be transmitted. For example, this may be useful, when a signal generated by the image sensor is to be used for performing event detection.

[0049] Fig. 2A illustrates signals that may be applied to the elements of the pixel circuit 120 when an image sensor comprising the pixel 10 as illustrated in Fig. 1 is to be operated in a first acquisition mode.

[0050] At S120, charges generated by the photosensitive element 110 are transferred to the floating diffusion 134. For example, the transfer transistor 121 may be switched on so as to accomplish the transfer. At S130, the signal charges are sampled to the first capacitor 126. The transfer transistor 121 may be switched off. The first sampling transistor 125 is maintained switched on. The second sampling transistor 128 is maintained switched off. Accordingly, the charges accumulated in the floating diffusion are transferred via the first sampling transistor 125 to the first capacitor 126.

[0051] Optionally, at S100, the floating diffusion 134 is reset before the charges generated by the photosensitive element 110 are transferred to the floating diffusion. To this end, a suitable voltage may be applied to the gate electrode of the reset transistor 122 resulting in a reset of the floating diffusion 134. As is further illustrated, the first sampling transistor 125 may be switched on, and the second sampling transistor 128 may be switched off. Further, according to implementations, the precharge transistor 124 may be switched on during S100 to enable a discharge of charges from the first capacitor 126.

[0052] Further optionally, at S110, the reset level stored in the floating diffusion 134 may be sampled. Accordingly, the reset transistor 122 is set to a non-conducting state, e.g. by switching off the reset transistor 122. Further, the first sampling transistor 125 may be maintained at a high level, so that the reset level may be sampled and stored in the first capacitor 126. As is clearly to be understood, resetting the floating diffusion (S100) and sampling (Slid) the reset level stored in the floating diffusion 134 may be dispensed with.

[0053] As has been described, the sampling process comprises sampling signal charges S120 to the first capacitor 126. Optionally, the sampling process further comprises sampling a reset level S110 to the first capacitor 126 before sampling signal charges to the fist capacitor. As has further been described with reference to Fig. 2A, for example, charges stored in the second capacitor 127 are maintained during the step S110 of sampling a reset level to the first capacitor 126. In more detail, since a second sampling transistor 128 is maintained at a non-conductive state during this sampling step, the reset level is not sampled to the second capacitor 127. Hence, according to embodiments, a signal level is maintained in the second capacitor 127.

[0054] Fig. 2B illustrates signals representing elements of a readout process performed by the pixel circuit 120 illustrated in Fig. 1, for example. S140 refers to processing of a previous row. At S150, the selection transistor 131 is switched on. Accordingly, at S150, first charges stored in the second capacitor 127 are read out. A column output signal 16 is transmitted via the column output 115. At S160, the second sampling transistor 128 is switched on. Accordingly, signal charges stored in the first capacitor 126 are distributed between the first and the second capacitors 126, 127. For example, 50% of the signal charges previously stored in the first capacitor 126 may now be stored in the second capacitor 127 and further 50 % of the signal charges may be stored in the first capacitor 126. Accordingly, at S160, the first capacitor is connected to the second capacitor. At S170, the second sampling transistor is switched off and the selection transistor 131 is maintained in a conducting state. Accordingly, reading out second charges stored in the second capacitor 127 is performed. A column output signal 16 is transmitted via the column output 115. At S180, the selection transistor 131 is switched off, and the next row is processed. Later on, the next frame will be processed. After the readout process and before a sampling process for the next frame, a stored signal level is maintained in the second capacitor 127. For example, even when optionally a reset level is sampled, S110, to the first capacitor, the stored signal level is maintained in the second capacitor 127.

[0055] The process illustrated in Figs. 2A and 2B is repeated for every frame. Since the stored signal level is maintained in the second capacitor 127 after the readout process and before a sampling process for the next frame, after S160, connecting the first capacitor to the second capacitor, a history of the last frames is stored in the second capacitor 127. In this context, the feature "last frames" refers to the frames read out when starting from the last reset step of the second capacitor 127. This will be explained in more detail below. Fig . 3 is a schematic view of an image sensor system 30 comprising an image sensor 20 according to embodiments . As is illustrated, the image sensor 20 comprises a plurality of pixels 10 that may be e . g . arranged in rows and columns . The pixel 10 may be implemented in the manner as has been illustrated in Fig . 1 . As is clearly to be understood, the pixel 10 may also be implemented in a di f ferent manner . Pixels of one column may be coupled to a column output 115 . The image sensor 20 may further comprise a row selection line driver 150 for selecting a line to be read out . The image sensor 20 may further comprise a controller 151 which is configured to operate the pixel circuit 120 in a first acquisition mode . Further, the controller 151 may be configured to operate the pixel circuit in a second acquisition mode to generate image data . For example , the controller 151 may be configured to control the pixel circuit to perform the methods that have been explained with reference to Figs . 2A and 2B .

[0056] The column output 115 is further connected to readout circuitry 154 comprising a plurality of analog digital converters (ADC ) 117 which are configured to convert the column output signals 16 transmitted via the column output 115 into digital signals , respectively . For example , also a gain may be set in the readout circuitry 154 . An output of the readout circuitry 154 may be input to a data proces sing device 152 . which further processes the digital signals generated by the ADCs 117 . The data processing device 152 generates output data .

[0057] The image sensor system 30 may comprise the image sensor 20 which has been described above , and a system processing unit 153 . The system processing unit 153 may e . g . receive output data transmitted by the data processing device 152 . For example , the system processing unit 153 may be implemented as an MCU (micro controlling unit ) . According to embodiments , the system processing unit 153 may be arranged remote from the image sensor 20 , e . g . on a di f ferent chip or may be arranged in a di f ferent device or may be arranged at a di f ferent location . For example , a data transmission line 157 , e . g . a digital bus or a wireless connection may be arranged between the image sensor 20 and the system processing unit 153 so as to enable data trans fer .

[0058] According to further embodiments , the system processing unit 153 may be integrated with the data processing device 152 or may be arranged on the same semiconductor chip or in the same device as the image sensor 20 .

[0059] For example , the system processing unit 153 may be configured to detect an event based on data received from the data processing device 152 . For example , when an event has been detected, the system processing unit 153 may send a corresponding signal to the controller 151 to change an acquisition mode to an image data generation mode or second acquisition mode . Further, a sensor system output signal 17 indicating that an event has been detected may be output by the image sensor system 30 . For example , the image sensor 20 or image sensor system 30 may further comprise a quality monitoring unit 118 that may be configured to monitor the quality of the generated data . For example , the quality monitoring unit 118 may be a component of the system processing unit 153 or of the data processing device 152 . This will be explained below while referring to Figs . 5A and 5B .

[0060] The system processing unit 153 may further process received pixel data, e . g . after a direct readout process . As will be explained in more detail below, the data processing device 152 may further compress data before they are transmitted to the system processing unit 153 . Using speci fic algorithms , the system processing unit 153 may be operable to reconstruct images from compressed data or may be operable to generate further detection results based on compressed data. As a result of processing data acquired during a direct readout process, image data or an event detection result may be output as the sensor system output signal 17.

[0061] In the following, examples of processes performed by the data processing device 152 will be explained in more detail. Readout and conversion of analog signals to digital signals may be performed by the readout circuitry 154.

[0062] 1. High-pass detection mode

[0063] As has been described with reference to Fig. 2A, a signal level stored in the second capacitor is maintained after a readout process and before performing a sampling process for a subsequent frame. Accordingly, the second capacitor 127 is not reset but stores an average of the previous signal charges y[n-l] . Further, as a result of the sampling of signal charges, S130, image information x[n] is stored in the first capacitor 126. The readout process S150 reads out first charges y[n-l] stored in the second capacitor. When subsequently processing the column output signal 16, e.g. in a corresponding ADC 117, the content of the second capacitor 127, i.e. the first charges y[n-l] are digitized.

[0064] Further, after S160, connecting the first capacitor to the second capacitor, the first and the second capacitors 126, 127 are shorted and their content is averaged. As a result, each of the first and the second capacitors 126, 127 stores second charges y[n] , wherein y[n] = 0.5 * (y[n- 1] + x[n] ) - After reading out the second charges y[n] stored in the second capacitor, S170, the content of the second capacitor 127 is digitized by the corresponding ADC 117. The digital output o [n] of the processing device 116 in the digital domain corresponds to a difference between the digitized content of the second charges y[n] and the digitized content of the first charges y[n-l] . o[n] = y[n]-y[n-l] = 0.5*y [n-1 ] +0.5*x [n] -y [n-1 ] = 0.5*x [n] -0.5*y [n-1 ]

[0065] Accordingly, o [n] corresponds to a high-pass filtered representation of the image sequence. Accordingly, an output of the image sensor 20 in this mode may be effectively a high-pass filtered image sequence which will only be different from zero when the content of the image changes over a number of previous frames. By way of example, o [n] is output by the readout circuitry 154, e.g. a corresponding ADC 117 to the data processing device 152.

[0066] Accordingly, the output of the image sensor may be used to detect regions of change. For example, a signal output by the data processing device 152 may indicate when the content of a pixel has changed, e.g. when a change has been detected. As a result, data to be transferred may be reduced.

[0067] 2. True differential detection mode

[0068] As has been described above, the first charges y[n-l] read out from the second capacitor 127 correspond to the history or average of previous images taken before the present frame. According to embodiments, by performing a "true differential mode" a difference between two successive frames may be detected. To this end, a second difference is determined which is based on first differences between consecutive frames. To be more specific, the data processing device 152 may determine the difference between the present frame and the frame before the present frame .

[0069] As has been discussed before, o [n] = y [n] -y [n-1 ] =0.5*x [n] -0.5*y [n-1 ] or o[n] = 0.5*x [n] -0.5* ( 0.5*x [n-1 ] -0.5*y [n-2 ] ) .

[0070] Further, o [n-1 ] =0.5*x [n-1 ] -0.5*y [n-2 ] .

[0071] Accordingly : o [n] -0.5*o [n-1] =0.5*x [n] -0.5*x [n-1]

[0072] Since the output signals of the present frame and of the frame before include a recursive low-pass filtered image sequence, by subtracting the output of two successive frames, the recursive portion may be eliminated. Accordingly, a true differential image may be reconstructed from the difference between two consecutive frames.

[0073] For example, this may be useful in cases in which the information provided by the high-pass filtered mode has too much delayed information .

[0074] Signals read out by the readout circuitry 154, e.g. when performing the high-pass detection mode or the differential readout mode as described above, may have a value e.g. in a range from -128 to 127. These signals may be further compressed at the data processing device 152. For example, the data processing device 152 may process the data to detect a change. As a result, for example, a signal processed by the data processing device 152 may have a value of -1, 0 or 1, depending on whether a negative change / no change / positive change has been detected. For example, a change may be detected using a threshold, e.g. an absolute threshold, an adaptive threshold or a relative threshold. For example, according to an adaptive threshold method, the threshold may depend on the data read out by the readout circuitry 154. As a consequence, intensity differences in images may be better handled.

[0075] Further compression methods may be employed by the data processing device 152. For example, quantization may be used. According to further examples, 1-bit or 4-bit encoding may be performed so as to compress the data.

[0076] Accordingly, a signal transmitted from the data processing device 152 to the system processing unit 153 may indicate a change of the content of captured images. According to further implementations, e.g. when the method explained below with reference to Figs. 4A and 4B is employed, a signal transmitted from the data processing device 152 to the system processing unit 153 may include image data.

[0077] Methods for data reduction may be employed to the data received by the data processing device 152. As a result, an amount of data to be transmitted to the system processing unit 153 is largely reduced resulting in a faster transmission speed and a reduced power.

[0078] 3. "Image fetch / direct readout" from high-pass image and capac- itor content

[0079] According to this process, the actual image may be reconstructed based on a high-pass filtered image and the content of the second capacitor, i.e. the second charges. For example, this reconstruction may be performed by the system processing unit 153.

[0080] For example, reading out the second charges ox[n] may be performed by initiating an additional data transfer with a different readout strategy, e.g. the method described with reference to Figs. 4A and 4B below. As has been explained above, o[n] = y [n] -y [n-1 ] =0.5*x [n] -0.5*y [n-1 ] ox [n] =y [n]

[0081] Accordingly, x[n]=o[n]+ ox[n] + offset

[0082] In more detail, the output, as has been explained above under section 1, corresponds to the difference of the digital signal corresponding to the second charges and the digital signal corresponding to the first charges. This output may be added to a digital signal corresponding to the second charges and an offset.

[0083] For example, this could be used in an adaptive algorithm that may be performed by the system processing unit 153. For example, the algorithm could decide that insufficient information is contained in the differential image and then initiate direct readout of the actual image to retrieve additional information. For example, according to this approach, a differential and a direct image readout may be interleaved without disturbing the state of the differential information in the storage capacitors.

[0084] To be more specific, as has been explained above, the image fetch / direct readout may be implemented in a manner so that a content stored on the second capacitor 127 is maintained. Consequently, without a reset being performed on the second capacitor 127, the storage history remains to be stored.

[0085] Fig. 4A shows signals to be applied to the pixel circuit 120, for illustrating an image readout process according to the second acquisition mode. For example, the signals may be applied to the pixel circuit 120 illustrated in Fig. 1. At S200, the floating diffusion may be reset by switching on the reset transistor 122. During this process, the first sampling transistor 125 and the second sampling transistor 128 are switched on. Further, during S200, the precharge transistor 124 is switched on. At S210, the reset level is sampled. As is shown, the reset transistor 122 is switched off. Since both the first sampling transistor 125 and the second sampling transistor 128 are switched on, the reset level is sampled to the first and to the second capacitors 126, 127. Thereafter, the second sampling transistor 128 is switched off. Further, at S220, a transfer of charges is performed. To this end, the transfer transistor 121 is switched on. As a result, charges generated in the photosensitive element 110 are transferred to the floating diffusion. At S230, sampling signal charges, the transfer transistor is switched off. At this point of time, the second sampling transistor 128 is switched off, and the first sampling transistor 125 is switched on. As a result, the signal charges are sampled to the first capacitor 126.

[0086] Fig. 4B shows an example of signals applied during a readout process. For example, the signals may be applied to the pixel circuit 120 illustrated in Fig. 1. At S240, the previous row is processed. At S250, the selection transistor 131 is switched on to initiate reading out a reset level S250 which is stored on the second capacitor 127. At S260, the second sampling transistor 131 is switched on. As a result, the signal charges stored on the first capacitor 126 are distributed between the first capacitor 126 and the second capacitor 127. After switching off the second transistor 128, at S270, the second charges stored in the second capacitor 127 are read out. At S280, the next row is processed. The processing device 116 may be operable to subtract the reset level from the second charges. Fig. 5A illustrates examples of frames that may be captured by an image sensor system 30. The image sensor has two or more acquisition modes: a first mode, e.g. a low-power, low-accuracy mode corresponding to the high-pass mode or the true differential mode as explained above, and a second mode, e.g. a high-power, high-accuracy mode, corresponding e.g. to the processes explained above with respect to Figs. 4A and 4B. The high-power mode is used for taking a normal image by the image sensor, e.g. 400x400x8 bit using a complex algorithm (e.g. large CNN, convolutional neural network) to achieve high accuracy. Processes performed by the second acquisition mode are e.g. illustrated in Figs. 4A and 4B.

[0087] Fig. 5A shows a timing diagram and the different images acquired by the image sensor. At position 1, a real frame 155 may be acquired using the second mode, at a high accuracy. For example, this may be accomplished using the processes illustrated in Figs. 4A and 4B. Thereafter, the image sensor switches to the first operating mode, low power mode, e.g. the high-pass mode or the true differential mode which have been explained above. As a result of this processing, frames indicative of changes between neighbouring frames may be output. For example, this may be accomplished by appropriating controlling the pixel circuit 120 as has been described above with reference to Figs. 2A and 2B. Accordingly, at position 2, information that only contains changes is represented.

[0088] For example, a quality metric may be generated that may be indicative of the data quality. For example, this could be part of a Kalman filter or RNN ("recursive neural network") . During the processes at position 2, energy may be saved. For example, instead of transmitting image information, frames including information about changes may be transmitted. Accordingly, for example, the pixel may be compressed to e.g. 4 bit to reduce the amount of transmitted data . Further, the transmission may be conditional on a line or block basis to avoid transmitting data when too few changes are detected . Since only sparse data are generated, less computational ef fort is needed for the evaluation .

[0089] For example , at position 3 , the quality metric may indicate that the tracking quality of the low-power acquired frames 156 is too bad . Accordingly, an image fetch may be initiated from the same frame to get high-accuracy tracking information . For example , this may be accompl ished using the image fetch / direct readout method as has been explained above . According to further embodiments , alternatively a normal direct readout as has been explained above with reference to Figs . 4A and 4B may be performed .

[0090] Thereafter, again frames 156 are acquired at low power using the first acquisition mode . According to further implementations , every nthframe , a full image fetch may be performed .

[0091] The method described may be used for eye tracking or for visual inspection when only a slight di f ference is to be detected . The method explained with reference to Fig . 5A may also be used for medical applications , e . g . for a sensor determining an alertness of a person . According to further implementations , this method may also be used for visual inspection, e . g . for quality monitoring . For example , the interpretation of the data may be performed in the system processing unit 153 .

[0092] Fig . 5B summari zes a method for operating an image sensor 20 or an image sensor system 30 according to embodiments . The image sensor system 30 comprises an image sensor 20 comprising a photosensitive element and a pixel circuit comprising a first capacitor and a second capacitor . The first capacitor is configured to be connected to the second capacitor, and the second capacitor is configured to be connected to a readout source follower. The pixel circuit is electrically coupled to the photosensitive element. The image sensor further comprises a processing device configured to process an output signal output by the pixel circuit. For example, the image sensor may have a configuration as has been explained with reference to Fig. 3. The pixel circuit may be implemented in the manner as has been illustrated in Fig. 1. The method comprises detecting a change (S310) between two consecutive frames comprising performing the first acquisition mode. For example, detecting a change may comprise processing (S320) a column output signal obtained according to the high pass mode or processing (S330) a column output signal obtained according to the true differential mode. Detecting a change (S310) may be repeated several times. For example, a quality monitoring process (S350) may be performed so as to monitor the quality of an output signal. When the quality monitoring process indicates that the quality of the output signal is not sufficient, detecting a change (S310) according to the first acquisition mode may be performed. This process may comprise a processing (S340) according to the direct readout mode so as to reconstruct an actual image. According to embodiments, instead, generating image data (S300) according to the second acquisition mode may be performed. Further, generating image data (S300) according to the second acquisition mode may be performed before detecting a change (S310) .

[0093] Fig. 5C shows a schematic view of an eye 160. Fig. 5C illustrates elements that may e.g. be inspected when performing eye tracking, e.g. adaptive eye tracking which may be used for AR ("augmented reality") applications. Fig. 5C shows the pupil 161 of the eye 160 which is surrounded by the iris 162. As is shown, glints 163 corresponding to specular reflections from an arrangement of light sources such as LEDs are illustrated. A movement of the eye will cause the pupil 161 and the glints 163 to change. Generally, the glints 163 are always saturated and the brightest part of the image. Further, the pupil 161 is the darkest part of the image .

[0094] Since the pupil 161 and the glints 163 are the largest / smallest signals in the frame, they will be most visible in the high-pass and differential images. This may be used for further compressing the data sent, e.g. from the data processing device 152 to the system processing unit 153. For example, there is a very low bit depth readout of changes possible. For example, even one bit for glints may be used (e.g. 1 = pixel changed from the non-saturated to saturated, or vice versa, 0 = pixel neither saturated before, nor after) .

[0095] Further, a readout may be performed only for locations of changed pixels representing glint locations.

[0096] A similar approach may be used for the part of the image of the lowest brightness, i.e. the pupil 161. Further, changes in pupil 161 and / or glint 163 locations could be used to activate full image reads to update an accurate eye tracking. Further, these changes may be used when the image sensor is used as an interface .

[0097] As has been explained in the foregoing, due to the specific first acquisition mode described, a simple and fast method for detecting changes between subsequent frames may be implemented in an image sensor 20. In an image sensor system 30 comprising the image sensor 20, further reduction of transmission time may be achieved using compression methods for compressing data to be transmitted from a data processing device 152, being a component of the image sensor 20, to a system processing unit 153, being a component of the image sensor system 30. Fig . 6A shows an equivalent circuit diagram of a pixel 10 comprising a di f ferent pixel circuit 120 which may as well be used for performing the method described herein above . As is illustrated in Fig . 6A, di f fering from embodiments illustrated in Fig . 1 , the first sampling transistor 125 is arranged between the first node 132 and the first capacitor 126 . Further, the second sampling transistor 128 is arranged between the second node 133 and the second capacitor 127 . Further, a third sampling transistor 129 may be arranged between the source follower 123 and the first node 132 . By appropriately adapting the signals applied to the respective components , the method described with reference to Figs . 2A and 2B may be performed . According to further implementations , the precharge transistor 124 may be dispensed with .

[0098] According to embodiments illustrated in Fig . 6B, two di f ferent capacitor branches may be connected to a third node 138 which is connected to a node between the source follower 123 and the precharge transistor 124 . For example , the first sampling transistor 125 may be arranged between the first node 132 and the third node 138 . Further, the first capacitor 126 may be connected to the first node 132 . The first node 132 is connected to the gate electrode of the readout source follower 130 . A terminal of the readout source follower 130 is connected to the column output 115 via the selection transistor 131 .

[0099] The second branch is implemented in a similar manner . In more detail , the second sampling transistor 128 is arranged between the third node 138 and the second node 133 . The second node 133 is connected to the second capacitor 127 . Further, the second node 133 is connected to the gate electrode of a second readout source follower 135 . A terminal , e . g . a source terminal of the second readout source follower 135 is connected to a second column output via a second selection transistor 136 . According to further implementations , the precharge transistor 124 may be dispensed with .

[0100] As is clearly to be understood, the equivalent circuit diagrams of Figs . 6A and 6B merely represent further examples to illustrate that the concepts disclosed may be applied to image sensors 20 implemented in a manner di f ferent from Fig . 1 . As is clearly to be understood, further variants of the image sensor 20 may be used, e . g . using further in-pixel capacitors .

[0101] Fig . 7 is a schematic view of an electronic device 25 according to embodiments . 18 . For example , the electronic device 25 may be selected from an eye tracking device , an AR device , a VR device , smart glasses , a surveillance camera, an inspection camera and an interface .

[0102] While embodiments of the invention have been described above , it is obvious that further embodiments may be implemented . For example , further embodiments may comprise any subcombination of features recited in the claims or any subcombination of elements described in the examples given above . Accordingly, this spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein .

[0103] LIST OF REFERENCES pixel incident electromagnetic radiation column output signal sensor system output signal image sensor electronic device photosensitive element column output analog digital converter quality monitoring unit pixel circuit trans fer transistor reset transistor source follower precharge transistor first sampling transistor first capacitor second capacitor second sampling transistor third sampling transistor readout source follower selection transistor first node second node floating di f fusion second readout source follower second selection transistor second column output third node row selection line driver controller data processing device 153 system processing unit

[0104] 154 readout circuitry

[0105] 155 real frame

[0106] 156 low power acquired frame

[0107] 157 data transmission line

[0108] 160 eye

[0109] 161 pupil

[0110] 162 iris

[0111] 163 glint

[0112] S100 reset floating diffusion

[0113] S110 sampling a reset level

[0114] S120 transfer of charges

[0115] S130 sampling signal charges

[0116] S140 processing previous row

[0117] S150 reading out first charges stored in the second capacitor

[0118] S160 connecting the first capacitor to the second capacitor

[0119] S170 reading out second charges stored in the second capacitor

[0120] S180 processing next row / frame

[0121] S200 reset floating diffusion

[0122] S210 sampling a reset level

[0123] S220 transfer of charges

[0124] S230 sampling signal charges

[0125] S240 processing previous row

[0126] S250 reading out a reset level

[0127] S260 connecting the first capacitor to the second capacitor

[0128] S270 reading out charges stored in the second capacitor

[0129] S280 processing next row

[0130] S300 generating image data (second acquisition mode)

[0131] S310 detecting a change (first acquisition mode)

[0132] S320 high pass mode

[0133] S330 true differential mode

[0134] S340 direct readout mode

[0135] S350 monitoring quality

Claims

CLAIMS1. A method of operating a pixel circuit (120) comprising: a first capacitor (126) and a second capacitor (127) , the first capacitor (126) being configured to be connected to the second capacitor (127) , and the second capacitor (127) being configured to be connected to a readout source follower (130) , the method comprising a sampling process and a readout process, the sampling process comprising sampling signal charges (S130) to the first capacitor (126) ; the readout process comprising: reading out (S150) first charges stored in the second capacitor ( 127 ) ; connecting (S160) the first capacitor (126) to the second capacitor (127) so as to distribute the signal charges stored in the first capacitor (126) between the first and second capacitors (126, 127) ; and reading out (S170) second charges stored in the second capacitor (127) , wherein a signal level stored in the second capacitor (127) is maintained after reading out the second charges and before sampling signal charges to the first capacitor for a next frame.

2. An image sensor (20) comprising: a photosensitive element (110) ; a pixel circuit (120) comprising a first capacitor(126) and a second capacitor (127) , the first capacitor (126) being configured to be connected to the second capacitor(127) , and the second capacitor (127) being configured to be connected to a readout source follower (130) , the pixel circuit (120) being configured to be electrically coupled to the photosensitive element (110) ;a data processing device (152) configured to process a column output signal (16) output by the pixel circuit (120) to generate output data; a controller (151) configured to operate the pixel circuit (120) in a first acquisition mode, wherein, in the first acquisition mode, the controller (151) is configured to control the pixel circuit (120) to perform a sampling process and a readout process, wherein during the sampling process the controller (151) is configured to control the pixel circuit (120) to perform: sampling signal charges (S130) to the first capacitor (126) ; and during the readout process the controller (151) is configured to control the pixel circuit (120) to perform: reading out (S150) first charges stored in the second capacitor ( 127 ) ; connecting (S160) the first capacitor (126) to the second capacitor (127) so as to distribute the signal charges stored in the first capacitor (126) between the first and second capacitors (126, 127) ; and reading out (S170) second charges stored in the second capacitor (127) , wherein a signal level stored in the second capacitor (127) is maintained after reading out the second charges and before sampling signal charges to the first capac- itor for a next frame.

3. The image sensor (20) according to claim 2, wherein the data processing device (152) is configured to detect a change based on a first difference between the first charges and the second charges read out from the second capacitor (127) , wherein the output data include a detection result.The image sensor (20) according to claim 3, wherein the data processing device (152) is configured to detect a changebased on a second difference which is based on first differences between consecutive frames.

5. The image sensor (20) according to any of claims 2 to4, wherein the controller (151) is configured to further operate the pixel circuit (120) in a second acquisition mode to generate image data.

6. The image sensor (20) according to claim 5, wherein the controller (151) is configured to switch from the first acquisition mode to the second acquisition mode after a predetermined number of frames has been acquired.

7. The image sensor (20) according to claim 5 or 6, further comprising a quality monitoring unit (118) , wherein the controller (151) is configured to switch from the first acquisition mode to the second acquisition mode based on a quality detected by the quality monitoring unit (118) .

8. The image sensor (20) according to any of claims 5 to 7, wherein the data processing device (152) is configured to generate a detection signal indicating whether a change has been detected, and the controller (151) is configured to switch to the second mode, when a change has been detected.

9. An image sensor system (30) comprising the image sensor(20) according to any of claims 2 to 8 and a system processing unit (153) , the system processing unit (153) being configured to receive the output data transmitted by the data processing device ( 152 ) .

10. The image sensor system (30) according to claim 9, wherein the data processing device (152) is configured tofurther compress the output data before transmitting them to the system processing unit (153) .

11. The image sensor system (30) according to claim 9 or 10, wherein the system processing unit (153) () is configured to reconstruct an actual image based on a first difference between the first charges and the second charges read out from the second capacitor (127) .

12. A method of operating an image sensor (20) comprising a photosensitive element (110) ; a pixel circuit (120) comprising a first capacitor(126) and a second capacitor (127) , the first capacitor (126) being configured to be connected to the second capacitor(127) , and the second capacitor (127) being configured to be connected to a readout source follower (130) , the pixel circuit (120) being configured to be electrically coupled to the photosensitive element (110) ; a data processing device (152) configured to process an output signal output by the pixel circuit (120) to generate output data; the method comprising: operating the pixel circuit (120) in a first acquisition mode to detect an change, wherein, in the first acquisition mode, the pixel circuit (120) performs a sampling process and a readout process, wherein during the sampling process the pixel circuit performs (120) : sampling (S130) signal charges to the first capacitor (126) ; and during the readout process the pixel circuit (120) performs : reading out (S150) first charges stored in the second capacitor ( 127 ) ;connecting (S160) the first capacitor (126) to the second capacitor (127) so as to distribute the signal charges stored in the first capacitor (126) between the first and second capacitors (126, 127) ; and reading out (S170) second charges stored in the second capacitor (127) , wherein a signal level stored in the second capacitor (127) is maintained after reading out the second charges and before sampling signal charges to the first capacitor for a next frame.

13. The method according to claim 12, wherein the pixel circuit (120) is configured to be operated in a second acquisition mode, the second acquisition mode comprising: controlling the pixel circuit to perform a sampling process and a readout process, wherein during the sampling process the pixel circuit performs: sampling (S210) a reset level to the first capacitor (126) and the second capacitor (127) ; and thereafter, sampling signal charges to the first capacitor (126) ; and during the readout process the pixel circuit (120) performs : reading out (S250) reset charges stored in the second capacitor ( 127 ) ; connecting (S260) the first capacitor (126) to the second capacitor (127) so as to distribute the signal charges stored in the first capacitor (126) between the first and second capacitors (126, 127) ; and reading out second charges stored in the second capacitor (127) .

14. The method according to claim 12 or 13, wherein a change is detected based on a first difference between thefirst charges and the second charges read out from the second capacitor ( 127 ) .

15. The method according to claim 12 or 13, wherein a change is detected based on a second difference which is based on first differences between consecutive frames.

16. The method according to any of claims 12 to 14, further comprising monitoring a quality of an output signal and performing the second acquisition mode depending on the quality.

17. The method according to any of claims 12 to 16, wherein the data processing device (153) generates a detection signal indicating whether a change has been detected, and the mode is changed to the second mode when a change has been detected.

18. A method of operating an image sensor system (30) comprising an image sensor (20) , the image sensor comprising a photosensitive element (110) ; a pixel circuit (120) comprising a first capacitor(126) and a second capacitor (127) , the first capacitor (126) being configured to be connected to the second capacitor(127) , and the second capacitor (127) being configured to be connected to a readout source follower (130) , the pixel circuit (120) being configured to be electrically coupled to the photosensitive element (110) ; a data processing device (152) configured to process an output signal output by the pixel circuit (120) to generate output data; the method comprising: operating the pixel circuit (120) in a first acquisition mode to detect a change, wherein, in the first acquisition mode, the pixel circuit(120) performs a sampling process and a readout process,wherein during the sampling process the pixel circuit (120) performs : sampling (S130) signal charges to the first capacitor (126) ; and during the readout process the pixel circuit (120) performs : reading out (S150) first charges stored in the second capacitor ( 127 ) ; connecting (S160) the first capacitor (126) to the second capacitor (127) so as to distribute the signal charges stored in the first capacitor (126) between the first and second capacitors (126, 127) ; and reading out (S170) second charges stored in the second capacitor (127) , wherein a signal level stored in the second capacitor (127) is maintained after reading out the second charges and before sampling signal charges to the first capacitor for a next frame, the image sensor system (30) further comprising a system processing unit (153) configured to receive the output data from the data processing device (152) .

19. The method according claim 18, comprising reconstructing, in the system processing unit (153) an actual image based on a first difference between the first charges and the second charges read out from the second capacitor (127) .

20. The method according to claim 18 or 19, further com- prising, by the data processing device (152) , further com- pressing the output data before transmitting them to the sys tern processing unit (153) .

21. An electronic device (25) comprising the image sensor(20) according to any of claims 2 to 8 or the image sensor system (30) according to any of claims 9 to 11.

22. The electronic device (25) according to claim 21, being selected from an eye tracking device, an AR device, a VR device, smart glasses, a surveillance camera, an inspection cam- era and an interface.

Images