CMOS image sensor-based pupil area measurement device

Abstract

A CMOS image sensor-based pupil area measurement device according to one embodiment of the present invention comprises: a pixel array including a plurality of pixels that have pixel regions arranged in a matrix form and are connected to each other, and for capturing the eyes of a user to acquire a plurality of pixel values; and a pupil region processing unit for binarizing, into black and white, the pixel values acquired from the respective pixels, and determining a pupil region by performing a region reduction operation of gradually whitening, multiple times, pixel regions having low black densities to acquire a remaining high-density core region, and a region expansion operation of expanding the core region by gradually blackening, multiple times, white pixel regions adjacent to the core region. Accordingly, the pupil area can be effectively measured with a simpler configuration.

Description

CMOS image sensor-based pupil area measuring device

[0001] The present invention relates to a pupil area measuring device, and more specifically, to a CMOS image sensor-based pupil area measuring device that determines the pupil area of ​​the eye through pixel-level data processing using a CMOS image sensor.

[0002] Changes in the size of a person's pupil are useful for estimating emotions, such as psychological state, and can also serve as an indicator of cognitive responses, such as thinking ability or concentration. Therefore, technology for measuring pupil area, including pupil size, is being utilized in various fields.

[0003] In measuring the pupil area, an image of the user's eye is captured by an image sensor such as a camera, and the pupil area is identified from the acquired image. In this process, processor-based software algorithm technology is used.

[0004] However, processors executing these algorithms consume a significant amount of power, and depending on the algorithm's performance, accurately measuring the pupil area can take a considerable amount of time. Furthermore, implementing the function to measure pupil size increases component size, which hinders product miniaturization and leads to higher manufacturing costs.

[0005] The present invention provides a CMOS image sensor-based pupil area measuring device capable of effectively measuring the pupil area with a simpler configuration to solve the above-mentioned problems.

[0006] A pupil area measurement device based on a CMOS image sensor according to one embodiment comprises: a pixel array having a plurality of interconnected pixels arranged in a matrix form and capturing a user's eye to obtain a plurality of pixel values; and a pupil area processing unit that determines the pupil area of ​​the eye by performing an area reduction operation to obtain a remaining high-density core area by progressively whitening a pixel area with low black density multiple times using the binary value of each pixel and surrounding pixel values, and an area expansion operation to expand the core area by progressively blackening a white pixel area adjacent to the core area multiple times. In this way, the center of a single pupil area is determined through the area reduction operation in the pixel array, and an operation to restore the pupil area at an accurate location is performed from the center of the pupil area through the area expansion operation, thereby providing information that can obtain the pupil area and the original pupil area.

[0007] The pupil region processing unit can perform a region expansion operation by changing the pixel value to black when, for each pixel, at least one of two or more surrounding pixels has a black value. By doing so, white noise within the black region of the eye image can be easily removed. Additionally, the pupil region processing unit can perform a region reduction operation by changing the pixel value to white when, for each pixel, at least one of two or more surrounding pixels has a white value. By doing so, the center of the pupil can be determined in a simple manner by finding a high-density single-core region with a high density of black pixels.

[0008] The pupil region processing unit includes a device (means) capable of storing the current pixel state, and this stored state changes according to region expansion and contraction operations. Here, the region expansion and contraction operations are performed at a certain clock (time). The output of the storage device is connected to surrounding pixels and the row and column registers of the corresponding pixels. The row and column register values ​​store a value of 1 if there is one or more pixels representing black in each row and column. The system may further include a termination decision unit that sums the values ​​of the row and column registers based on the output values ​​of the processing unit according to expansion and contraction, and terminates the expansion and contraction when the number of pixel columns or rows of the remaining core region is less than or equal to a predetermined value. By doing so, the core region at the time of termination can be determined as the center of the pupil region.

[0009] In the process of determining the center of the pupil region, the number of times a region reduction or region expansion operation is performed is stored, and after determining the center, a region expansion operation corresponding to the number of operations is performed to restore the pupil region at the time of pupil imaging and determine the size of the pupil region.

[0010] When the expansion and contraction operations are terminated, the system may further include a column selection unit that selects a column position of one of a plurality of black pixels in a register based on the final output value of the processing unit, and a row selection unit that selects a row position of one of a plurality of black pixels. Alternatively, the size of the pupil area may be determined based on the number of pixel columns or rows of a pixel group having black pixel values ​​among the pixels of the determined pupil area. Accordingly, the location and size of the pupil area can be simply determined using the pixel values ​​of the pupil area.

[0011] According to one embodiment of the present invention as described above, a pupil area measuring device capable of effectively measuring the pupil area with a simpler configuration is provided, which has the effect of obtaining changes in the pupil area at high speed and with low power consumption.

[0012] In addition, by implementing a pupil area measurement device based on a CMOS image sensor within the sensor, the number of parts can be minimized, product miniaturization can be easily achieved, and manufacturing costs can be reduced.

[0013] FIG. 1 is a block diagram illustrating the configuration of a pupil area measuring device according to one embodiment of the present invention.

[0014] FIG. 2 illustrates the configuration of a processing unit of a pixel and pupil area measuring device according to one embodiment of the present invention.

[0015] FIGS. 3 and FIGS. 4 are drawings illustrating examples in which a region reduction operation and a region expansion operation are performed according to an embodiment of the present invention.

[0016] FIG. 5 is a flowchart of the operation for determining the pupil area in a pupil area measuring device according to one embodiment of the present invention.

[0017] FIG. 6 is a diagram illustrating an example of determining the pupil area in an eye image according to one embodiment of the present invention.

[0018] FIG. 7 is a diagram relating to a region reduction operation and the determination of the end point of the region reduction operation according to one embodiment of the present invention.

[0019] FIG. 8 is a diagram relating to a region expansion operation and the determination of a pupil region according to a region expansion operation according to an embodiment of the present invention.

[0020] FIG. 9 is a diagram illustrating a configuration for determining the size of a pupil area according to one embodiment of the present invention.

[0021] FIG. 10 is a diagram relating to the acquisition of location information of a pupil area according to one embodiment of the present invention.

[0022] FIG. 11 is a drawing regarding the determination of the size of the pupil area according to one embodiment of the present invention.

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention.

[0024] FIG. 1 is a block diagram illustrating the configuration of a pupil area measuring device according to one embodiment of the present invention.

[0025] The pupil area measuring device (1) is mounted on an electronic device, such as a user terminal or a head-mounted display apparatus (HMD), and provides information on the pupil area.

[0026] According to one embodiment, the pupil area measuring device (1) is implemented as a device based on a single-chip CMOS (complementary metal oxide silicon) image sensor (CIS: CMOS image sensor). The CMOS image sensor is an image sensor manufactured using CMOS manufacturing technology, and since the circuit for image signal processing can be integrated into a single chip, it is easier to miniaturize the module through a System On Chip (SoC). In addition, the CMOS image sensor enables effective measurement of the pupil area with a simpler configuration and enables pixel-level image processing by the pixel array (100) described later.

[0027] According to one embodiment of the present invention, the pupil area measuring device (1) can perform two functions. First, the pupil area measuring device (1) performs the function of providing information on the pupil area based on image data, that is, an eye image, that captures the user's eye. Second, the pupil area measuring device (1) performs the function of outputting the captured image data through an image output unit (150). In one embodiment, the image output unit (150) includes a Correlated Double Sampling (CDS) circuit and an A / D converter (ADC) to convert the captured image data into a digital signal and output it.

[0028] The pupil area measuring device (1) includes a pixel array (100). Referring to FIG. 1, the pixel array (100) has a plurality of pixels having pixel areas arranged in a matrix form, and captures the user's eye to obtain a plurality of pixel values.

[0029] According to one embodiment, the pixel array (100) is configured in the form of a matrix having an M×N matrix with M pixel columns (hereinafter also referred to as columns) and N pixel rows (hereinafter also referred to as rows). Each pixel within the pixel array (100) is configured as a cellular neural network connected to surrounding cells. Each pixel of the pixel array (100) outputs a value representing its state, for example, black or white, and when a single pixel in each column and row represents a black value, this is stored in the column flag register (120) and the row flag register (110). That is, the row and column registers are like projections of the black image within the pixel array in the row and column directions, respectively. The values ​​stored in the column and row registers can be summed for each entire column and row.

[0030] Each pixel (200) in the pixel array (100) includes a capture unit (210) that captures image data of the eye image by pixel by photographing the user's eye. The capture unit can be implemented, for example, with photodiodes and transistors to capture image data by pixel.

[0031] According to one embodiment, the capture unit may be configured to capture image data pixel by pixel by adopting a structure for driving in a global shutter manner. However, since this is not limited to other examples, the capture unit may be configured for a rolling shutter method.

[0032] The pupil area measuring device (1) includes a processing unit (220) (hereinafter also referred to as a pupil area processing unit) that processes pixel values ​​obtained for each pixel (200) of the pixel array (100). In one embodiment, the processing unit (200) is provided for each individual pixel (200), and a separate determination unit (pupil area determination unit) that determines the pupil area based on the final pixel values ​​processed by each processing unit (200) may be further provided. However, the present invention is not limited to the disclosed embodiments, and the following operations may be performed by a configuration (e.g., a processing unit) that encompasses the processing unit and the determination unit.

[0033] According to one embodiment, the processing unit (220) binarizes a plurality of pixel values ​​obtained from the pixel capture unit (210) into black and white, and can selectively perform an area reduction operation in which pixels with black binarized pixel values ​​are changed to white, or an area expansion operation in which pixels with white binarized pixel values ​​are changed to black.

[0034] FIG. 2 illustrates the configuration of a processing unit of a pupil area measuring device according to one embodiment of the present invention. FIG. 3 and FIG. 4 are drawings illustrating examples in which an area reduction operation and an area expansion operation are performed according to one embodiment of the present invention.

[0035] Referring to FIG. 2, the processing unit (220) includes a binarization unit (230) that binarizes a plurality of pixel values ​​obtained from the pixel capture unit (210) into black and white.

[0036] According to one embodiment, the binarization unit (230) includes a comparator that compares pixel data (pixel value) of each pixel of image data obtained by photographing a user's eye with a threshold value and outputs binary data. The comparator can compare a plurality of pixel values ​​with a predetermined threshold value (external threshold) and output binary data corresponding to the comparison result, for example, 1 or 0.

[0037] In one embodiment, the binarization unit (230) outputs low, i.e., 0, when the pixel value is greater than the threshold value, and outputs high, i.e., 1, when the pixel value is less than or equal to the threshold value. Here, 0 can be defined as a binarized pixel value representing black, and 1 can be defined as a binarized pixel value representing white.

[0038] The processing unit (220) can perform a region reduction operation or a region expansion operation on the binarized pixel value output from the binarization unit (230).

[0039] The area reduction operation includes erosion or shrink, which changes black pixel values ​​to white based on surrounding pixel values. The area expansion operation includes dilation or expand, which changes white pixel values ​​to black based on surrounding pixel values.

[0040] According to one embodiment, the pupil area measuring device (1) may be configured to perform an area reduction operation or an area expansion operation on pixel values ​​obtained within a pixel array (100), and to determine the pupil area through such pixel-level interaction.

[0041] Referring to FIG. 2, the processing unit (220) may include a first multiplexer (240), a flip-flop unit (FF) (250), a reduction means (All-0 Detector) (260), an expansion means (All-1 Detector) (270), and a second multiplexer (280).

[0042] The first multiplexer (240) uses a selection signal (sel) to transmit the binarized pixel value (binary data) output from the binarization unit (230) to the flip-flop unit (250).

[0043] The flip-flop unit (250) can be implemented as a D flip-flop and stores a pixel value output from the first multiplexer (240) by synchronizing it with a clock signal (CLK). Here, the flip-flop unit (250) can store a value for one clock cycle. The value thus stored is provided to neighboring pixels, row register (110), and column register (120).

[0044] The value stored in the flip-flop section (250) is transmitted to the reduction means (260) and the expansion means (270) so that the area reduction operation or area expansion operation can be performed pixel by pixel.

[0045] Referring to FIG. 2, the reduction means (260) can be implemented as an n-in NOR gate. The reduction means (260) receives the pixel value (binary data) of the target pixel from the flip-flop unit (250) and receives a predetermined number (e.g., 4) of pixel values ​​of surrounding pixels from surrounding pixels.

[0046] According to one embodiment, the reduction means (260) performs an area reduction operation for each pixel whose pixel value is black (0), changing (converting) the corresponding pixel value to white (1) based on the pixel values ​​of two or more surrounding pixels.

[0047] Specifically, the reduction means (260) performs a region reduction operation by changing the target pixel value to white (1) when at least one pixel value among two or more (e.g., four) surrounding pixels of a target pixel (310, 320) whose pixel value is black is white (1). In other words, the reduction means (260) can operate as an All-0 detector that maintains the pixel value as black (0) only when the pixel values ​​(binary data) of a set number of surrounding pixels relative to the target pixel (310, 320) are all black (0).

[0048] For example, in the case of Example 1 of FIG. 3, since at least one of the four surrounding pixels (311, 312, 313, 314) of a target pixel (310) whose pixel value is black (0) is white (1), an area reduction operation is performed in the reduction means (260) to change the corresponding pixel value to white (1). According to this area reduction operation, the size of the area where the pixel value is black (0) in the entire pixel area can be reduced.

[0049] In contrast, in the case of Example 2 of FIG. 3, since the surrounding 4 pixels (321, 322, 323, 324) of the target pixel (320) whose pixel value is black (0) are all black (0), this is detected (All-0 detector), and the corresponding pixel value is kept as black (0).

[0050] However, this is an example and is not limited thereto, and as another example, the reduction means (260) may be implemented so that the area reduction operation is performed when more than a predetermined number of surrounding pixels (e.g., more than 2, which is half of the 4 surrounding pixels) are white (1).

[0051] Referring to FIG. 2, the expansion means (270) can be implemented as an n-in NAND gate.

[0052] The expansion means (270) receives the pixel value (binary data) of the target pixel from the flip-flop unit (250) and receives a predetermined number (e.g., 4) of pixel values ​​of surrounding pixels from surrounding pixels.

[0053] According to one embodiment, the expansion means (270) performs an area expansion operation to change (convert) the pixel value to black (0) based on the pixel values ​​of two or more surrounding pixels for each pixel where the pixel value is white (1).

[0054] Specifically, the expansion means (270) performs an area expansion operation by changing the target pixel value to black (0) when at least one pixel value among two or more (e.g., four) surrounding pixels of a target pixel (410, 420) whose pixel value is white in the pixel area is black (0). In other words, the expansion means (270) can operate as an All-1 detector that maintains the pixel value as white (1) only when the pixel values ​​(binary data) of a predetermined number of surrounding pixels based on the target pixel (410, 420) are all white (1).

[0055] For example, in the case of Example 3 of FIG. 4, since at least one of the four surrounding pixels (411, 412, 413, 414) of the target pixel (410) with a pixel value of white (1) is black (0), an area expansion operation is performed in the expansion means (270) to change the target pixel value to black (0). According to this area expansion operation, the size of the area where the pixel value is black (0) in the entire pixel area can be expanded.

[0056] In contrast, in the case of Example 4 of FIG. 4, since the surrounding 4 pixels (421, 422, 423, 424) of the target pixel (420) whose pixel value is white (1) are all white (1), this is detected (All-1 detector) and the target pixel value is kept as white (1).

[0057] However, this is an example and is not limited thereto, and as another example, the expansion means (270) may be implemented so that the area expansion operation is performed when more than a predetermined number of surrounding pixels (e.g., more than 2 pixels, which is half of the 4 surrounding pixels) are black (0).

[0058] The second multiplexer (280) uses a control signal (CTRL) to select a result value of either the expansion means (270) or the reduction means (260) and transmits it to the first multiplexer (240). For example, according to a predetermined reduction and expansion algorithm, the result value of the reduction means (260) may be selected in the area reduction step, and the result value of the expansion means (270) may be selected in the area expansion step.

[0059] The first multiplexer (240) uses a selection signal (sel) to transmit a pixel value changed according to the area reduction operation or area expansion operation transmitted from the second multiplexer (280) to the flip-flop unit (250).

[0060] The flip-flop unit (250) stores, i.e., updates, the pixel value output from the first multiplexer (240) according to the clock (CLK) operation. Accordingly, the flip-flop unit (250) can store a pixel value (new result value) that has been changed (updated) according to the area reduction operation or area expansion operation for each pixel. Each pixel value stored in this way is provided to neighboring pixels and transmitted through the column bus and row bus to be used in determining the values ​​of the column register (120) and row register (110).

[0061] In a pupil area measuring device (1) according to one embodiment, the processing unit (220) obtains a remaining high-density core area by progressively whitening a pixel area with low black density multiple times through a region reduction operation, and determines (measures) the pupil area of ​​the eye by progressively blackening a white pixel area adjacent to the core area multiple times through a region expansion operation. The pupil area measuring device (1) uses the pupil size to determine (measure) the pupil area corresponding to the pupil area thus determined.

[0062] FIG. 5 is a flowchart of the operation for determining the pupil area in a pupil area measuring device according to one embodiment of the present invention. FIG. 6 is a diagram illustrating an example of determining the pupil area in an eye image according to one embodiment of the present invention.

[0063] Referring to Fig. 5, in operation 510, the user's eye is captured to obtain multiple pixel values.

[0064] For example, referring to FIG. 6, the pupil area measuring device (1) can obtain multiple pixel values ​​by capturing pixel-by-pixel image data (610) of an eye image through a pixel array (100).

[0065] In operation 520, the pupil area measuring device (1) binarizes the acquired pixel values ​​into black and white.

[0066] According to one embodiment, the image data (610) captured in operation 510 is converted into binary data (620) through the binarization unit (230) of the processing unit (220) in operation 520. The binary data (620) thus converted is represented by a pixel value of black (0) or white (1) for each pixel.

[0067] The converted binary data (620) has multiple pixel regions in which the pixel value is black (0). For example, referring to FIG. 6, multiple pixel regions in which pixels with a pixel value of black (0) exist corresponding to locations such as eyebrows or pupils can be identified.

[0068] In operation 530, the pupil area measuring device (1) repeatedly performs the area reduction operation to determine the center of the pupil area.

[0069] According to one embodiment, the processing unit (220) repeatedly performs an area reduction operation to whiten pixel areas with a relatively low density of black (0) in binary data (620) in multiple steps. Through this area reduction operation, the processing unit (220) obtains a single remaining high-density core area (621) with a high density of black in the binary data (620), and determines this core area (621) as the center of the pupil area.

[0070] As described in relation to FIGS. 2 and 3, the processing unit (220) repeatedly performs an area reduction operation to reduce (shrink) the area where the pixel value is black (1) by changing the corresponding pixel value of the target pixel (310) from black (0) to white (1) when at least one of the two or more (e.g., four) surrounding pixels (311, 312, 313, 314) for the target pixel (310) where the pixel value is black (1) has a pixel value of white (1) (e.g., the pixel value of the surrounding pixels (311, 312) is white (1).

[0071] Referring to FIG. 6, by repeating the area reduction operation, the pixel values ​​of the areas with a relatively low density of black (0) among the pixel areas (pupil candidate areas) of the binary data (620), such as the eyebrow area (622), are turned white (1) in multiple steps, and finally, only a single core area (631) with a high density of black (0) pixels remains in the binary data (630), and this core area (631) is determined to be the center of the pupil area.

[0072] According to one embodiment, in addition to the area reduction operation, one or more area expansion operations may be performed in operation 530. For example, the processing unit (220) may perform an area expansion operation that changes the binarized pixel value to black once, and then repeat the area reduction operation. As another example, the processing unit (220) may operate in the order of a first area reduction operation, an area expansion operation, and a second area reduction operation.

[0073] By performing the area expansion operation in this way in operation 530, the pixel value of the reflection position (623) within the pupil (621) of the eye image in the pixel area of ​​the binary data (620) is changed from white (1) to black (0).

[0074] For example, when an external light (e.g., infrared) is irradiated in operation 510 to capture the user's eye, the reflection location (613) where the external light is reflected (glint) within the pupil (611) in the captured image data (610) may be brightly displayed, and according to the binarization of operation 520, the pixel value of the reflection location (623) within the pupil (621) in the binary data (620) is represented as white (1). In this case, if the pixel value within the pupil (621) is white (1), an error may occur when finding the center point of the pupil area through the area reduction operation; therefore, the pixel value is changed to black (0) through the pupil expansion operation, and the area reduction operation is repeated.

[0075] According to one embodiment, in operation 530, the processing unit (220) determines that the area (631) is the center of the pupil area when the number of pixel columns or rows of one remaining core area (631) is less than or equal to a predetermined value according to the area reduction operation that is performed repeatedly in stages.

[0076] Here, when the number of pixel columns or rows becomes less than a predetermined value, it is when the remaining core area (631) resulting from the area reduction operation converges to a center point consisting of fewer than a predetermined number of pixels (e.g., 5), thereby reaching a reduction limit point (shrinkage limit point) where no further area reduction operation is required, and when such a reduction limit point is reached, it becomes the end point of the area reduction operation.

[0077] FIG. 7 is a diagram relating to a region reduction operation and the determination of the end point of the region reduction operation according to one embodiment of the present invention.

[0078] According to one embodiment, the pixel values ​​(binary data) resulting from the performance of the area reduction operation are summed by pixel column and row, and the area reduction operation is terminated according to the summing result. Then, when the area reduction operation is terminated, the remaining high-density single-core area (631) can be determined as the center of the pupil area.

[0079] In each column and row of the pixel array (100), a representative value of the pixel value (binary data) is stored for each pixel column or row.

[0080] Referring to FIG. 1, a pupil area measuring device (1) according to one embodiment includes a row register (110) in which representative values ​​of pixel values ​​for each column of a pixel array (100) are stored in a row form, and a column register (120) in which representative values ​​of pixel values ​​for each row of a pixel array (100) are stored in a column form.

[0081] In one embodiment, the row register (110) includes a plurality of cells arranged in a column, and the column register (120) includes a plurality of cells arranged in a row. Each cell may be implemented as a capacitor, and a representative value of a pixel value (binary data) may be stored for each corresponding column or row.

[0082] According to one embodiment, in each cell of the row register (110) and the column register (120), a representative value 1 is stored when at least one of the pixel values ​​of the pixels of the corresponding column or row is black (0), and a representative value 0 is stored only when all the pixel values ​​of the pixels of the corresponding column or row are white (1). For example, referring to FIG. 7 (a), it can be seen that a representative value (0,1,1,1,1,1,1,0) is stored in each cell of the column-row register (110), and (0,1,1,1,1,1,1,0) is stored in each cell of the register (120).

[0083] According to one embodiment, the pupil area measuring device (1) includes a termination determination unit (700).

[0084] The termination decision unit (700) includes a row adder (710), a column adder (720), a full adder (730), and a comparator (740).

[0085] The row summing unit (710) counts the number of rows in which the representative value of each cell of the row register (110), i.e., the black (0) pixel, exists, i.e., calculates the row summing value. The column summing unit (720) counts the columns in which the representative value of each cell of the column register (120), i.e., the black (0) pixel, exists, i.e., calculates the column summing value. The matrix summing unit (730) sums the column summing value and the row summing value to calculate the total summing value.

[0086] The comparison unit (740) compares the total sum value calculated by the matrix sum unit (730) with a predetermined reference value (e.g., 5), and the termination of the area reduction operation can be determined according to the comparison result (output value) of the comparison unit (740).

[0087] According to one embodiment, the time when the total sum of the comparison results of the comparison unit (740) becomes smaller than the reference value is determined as the end point of the area reduction operation.

[0088] For example, referring to FIG. 7(a), since six representative values ​​having a value of 1 are stored in the row register (110), six row sum values ​​are calculated in the row sum unit (710). Since six representative values ​​having a value of 1 are stored in the column register (120), six column sum values ​​are calculated in the column sum unit (720). In the matrix sum unit (730), the column sum values ​​and row sum values ​​are added together to calculate a total sum value of 12ea. The comparison unit (740) compares the total sum value of 12ea with a predetermined reference value (e.g., 5ea) and outputs a signal corresponding to the comparison result. For example, if the total sum value is greater than or equal to the reference value (12ea > 5ea), the comparison unit (740) outputs a logic high (1), and in response to this, the area reduction operation of operation 530 can be continuously repeated (go on).

[0089] As the area reduction operation is continuously performed step by step in this way, the binary data of each pixel of the pixel array (100) can be converted as shown in (b) of FIG. 7. For example, since two representative values ​​with a value of 1 are stored in the row register (110), two row sum values ​​are calculated in the row sum unit (710). Since two representative values ​​with a value of 1 are stored in the column register (120), two column sum values ​​are calculated in the column sum unit (720). In the matrix sum unit (730), the column sum values ​​and row sum values ​​are added to calculate two total sum values. The comparison unit (740) compares the four total sum values ​​with a predetermined reference value (e.g., five) and outputs a signal corresponding to the comparison result. For example, if the total sum value is smaller than the reference value (4ea < 5ea), the comparison unit (740) outputs a logic row (0), and in response, the area reduction operation can be terminated (stop). Here, the total sum value (e.g., 4ea) calculated at the time the area reduction operation is terminated corresponds to the number of pixels in the pixel array (100) where the pixel value is black (0). That is, the termination decision unit (700) can terminate the area reduction operation if the number of pixels where the pixel value is black (0) becomes smaller than a predetermined reference number (e.g., 5ea) according to the area reduction operation.

[0090] In addition, at this time, as the number of columns or rows constituting one remaining core area (631) with a pixel value of black (0) according to the area reduction operation becomes less than a predetermined value (e.g., 2), the remaining high-density core area (631) at the end of the area reduction operation can be determined as the center of the pupil area.

[0091] According to one embodiment, when the area reduction operation is terminated, the number of times the area reduction operation is repeated in operation 530 is counted. Since the area reduction, expansion, and flip-flop operations are all performed in synchronization with the clock, the number of times the area reduction operation is performed can be counted by counting the clock. The counted number can be utilized later when performing an area expansion operation to determine the pupil area.

[0092] In one embodiment, if one or more area expansion operations are performed in addition to the area reduction operation in operation 530, only the number of repeated operations is counted by subtracting the area expansion operation from the total number of times the area reduction operation is performed.

[0093] Referring again to FIG. 5, in operation 540, the pupil area measuring device (1) determines the pupil area by repeatedly performing the area expansion operation in correspondence with the degree of performance of the area reduction operation in operation 530.

[0094] FIG. 8 is a diagram relating to a region expansion operation and the determination of a pupil region according to a region expansion operation according to an embodiment of the present invention.

[0095] According to one embodiment, the processing unit (220) repeatedly performs an area expansion operation to progressively blacken a pixel area of ​​white (1) adjacent to the core area (631) obtained in operation 530. Through this area expansion operation, the core area (631) can be gradually expanded, and the expanded core area is determined as the pupil area of ​​the eye.

[0096] As described in relation to FIGS. 2 and FIGS. 4, the processing unit (220) repeatedly performs an area expansion operation to expand (increase) the area where the pixel value is black (0) by changing the corresponding pixel value of the target pixel (410) from white (1) to black (0) when at least one of the two or more (e.g., four) surrounding pixels (411, 412, 413, 414) for the target pixel (410) where the pixel value is white (1) is black (e.g., the pixel value of the surrounding pixels (413, 414) is black).

[0097] At this time, the area expansion operation is repeated in correspondence with the degree of the area reduction operation performed in operation 530. For example, the processing unit (220) may repeat the area expansion operation as many times as the counted number of repetitions of the area reduction operation. Referring to FIG. 8, as a result of repeating the area expansion operation, the size of the area where the pixel value is black (0) is gradually expanded radially from the core area, that is, the center (810) of the pupil area, thereby determining the pupil area (820).

[0098] Referring to FIG. 6, by repeating the area expansion operation in this way, one remaining core area (631) among the pixel areas of the binary data (630), that is, the pupil area (641) expanded from the center of the pupil area, is determined.

[0099] In operation 540, the processing unit (220) performs an area expansion operation corresponding to the degree of the area reduction operation performed in operation 530. Accordingly, the pixel values ​​of other pixel areas, such as the eyebrow area (632) with low density of black (0) and other pixel areas excluding the core area (631) remaining from the area reduction operation of operation 530, are all changed to white (1) to exclude (eliminate) the pupil area candidates, and then the pupil area (641) is determined based on the center of the pupil area, which is the only remaining core area (631), so that the pupil area (641) can be restored at an accurate location without error in the eye image (640).

[0100] In operation 550, the size of the pupil area is determined for the recovered area. Referring to FIG. 6, in one embodiment, the size of the pupil area (641) can be determined by coordinate information (x1, x2, y1, y2).

[0101] FIG. 9 is a diagram illustrating a configuration for determining the size of a pupil area according to an embodiment of the present invention. FIG. 10 is a diagram regarding the acquisition of location information of a pupil area according to an embodiment of the present invention. FIG. 11 is a diagram regarding the determination of the size of a pupil area according to an embodiment of the present invention.

[0102] Referring to FIG. 9, a pupil area measuring device (1) according to one embodiment further includes a row selector (910), a column selector (920), a row flag detection unit (930), a column flag detection unit (940), a row address generation unit (950), a column address generation unit (960), a matrix summation unit (970), and an operation unit (980). Here, the row flag detection unit (930) and the column flag detection unit (960) each include a row register (110) and a column register (120).

[0103] Referring to FIG. 10, the row selection unit (910) sets the direction for acquiring location information by bit control (1-bit control) of the row flag detection unit (930) to find the cell where flag 1 is stored from the leftmost cell (column) or to acquire the cell where flag 1 is stored from the rightmost cell (column). By setting sequentially for both directions, location information i=2 and i=7 can be obtained. The location information thus acquired can be generated as row address information by the row address generation unit (930).

[0104] In the same way, by selecting a direction in the column selection unit (920) and bit control for the column flag detection unit (940), location information j=2 of the cell where flag 1 is stored from the topmost cell (row) and location information j=7 of the cell where flag 1 is stored from the bottommost cell (row) are obtained. The external information thus obtained can be generated as column address information by the column address generation unit (940).

[0105] Based on the location information (matrix information) of these pupil regions, the size of the pupil region is determined as 6 pixel rows X 6 pixel columns.

[0106] According to another embodiment, the size of the pupil area is determined based on the size of the center of the pupil area determined in operation 530 and the number of pixel columns or rows corresponding to the number of times the area reduction operation or area expansion operation is performed in operation 530 and operation 540.

[0107] For example, in the embodiments of FIGS. 7 and FIGS. 8, the size of the center (760, 810) of the pupil area determined in operation 530 is 2 pixels row X 2 pixels column = 4 pixels. Referring to FIG. 7, the number of times the area reduction operation is performed in operation 530 is 2. As shown in FIG. 8, since the original pupil size can be expanded by the number of times the reduction operation is performed, the pupil size becomes 2 + (2+2) = 6 pixels row when the reduction counts of 2 are added to the left and right respectively. Likewise, the column becomes 2 + (2+2) = 6. Therefore, the size of the pupil area can be determined as 6 pixels row X 6 pixels column.

[0108] According to another embodiment, the size of the pupil area can be obtained using row register information and column register information of the pupil area.

[0109] Referring to FIG. 11, when the pupil area is determined, flag 1 is stored in the row register (110) in a cell corresponding to the horizontal direction of the pupil, so that the result of the row summation unit (710) represents the row size (Column size = 6), and flag 1 is stored in the column register (120) in a cell corresponding to the vertical direction of the pupil, and the result of the column summation unit (720) generates information corresponding to the column size (Row size = 6). By outputting this information, namely Row / width 6 and Column / height 6, the size of the pupil area can be determined as 6 pixel rows X 6 pixel columns.

[0110] In the pupil area measuring device (1) according to one embodiment of the present invention as described above, the pupil area can be effectively measured with a simpler configuration. That is, the pupil area measuring device (1) determines the center of a single pupil area through an area reduction operation that gradually whitens a pixel area with low black density, and determines the pupil area and its size through an area expansion operation that gradually blackens a white pixel area adjacent to the center of the pupil area. This makes it easy to remove noise, which is a black area other than the pupil, from an eye image and to restore the pupil area at an accurate location. Furthermore, by implementing the pupil area measuring device (1), which can measure the pupil area including not only the center of the pupil but also the pupil size, in a single-chip form, it is possible to reduce manufacturing costs through minimizing the number of parts and consequently miniaturizing the product.

[0111] In addition, according to the above embodiment, by implementing all circuit configurations, including the processing unit (220), as digital circuits, the pupil area measuring device (1) operates stably and has the advantage of being easier to miniaturize the chip.

[0112] Meanwhile, according to one embodiment of the present invention, the pupil area measuring device (1) may further be provided with a separate capacitor that stores the original pixel value when image data is captured for each pixel of the pixel array (100).

[0113] Since the original pixel value of the initially captured image data is continuously stored in this separate capacitor, even if the pixel value of a specific pixel changes due to the area reduction or area expansion operation, the pupil area measuring device (1) can perform a function or operation using the information (original pixel value) when necessary. In other words, the image shooting operation and the image data processing operation are separated to enable parallel processing.

[0114] Although the present invention has been described in detail through preferred embodiments above, the invention is not limited thereto and can be implemented in various ways within the scope of the claims.

Claims

1. In a CMOS image sensor-based pupil area measurement device, A pixel array having a pixel region arranged in a matrix form and having a plurality of interconnected pixels, and acquiring a plurality of pixel values ​​by capturing the user's eye; and A pupil area measuring device comprising a pupil area processing unit that determines the pupil area of ​​the eye by performing an area reduction operation to obtain a remaining high-density core area by binarizing the pixel values ​​obtained for each pixel into black and white and whitening a pixel area with low black density multiple times in stages, and an area expansion operation to expand the core area by blackening a white pixel area adjacent to the core area multiple times in stages.

2. In Paragraph 1, The pupil region processing unit above is a pupil region measuring device that binarizes a plurality of pixel values ​​for each of the above pixels.

3. In Paragraph 1, The pupil area processing unit above is a pupil area measuring device that performs the area expansion operation by changing the pixel value to black when, for each pixel, at least one of two or more surrounding pixels has a pixel value of black.

4. In Paragraph 1, The pupil area processing unit above is a pupil area measuring device that performs the area reduction operation by changing the pixel value to white when, for each pixel, at least one of two or more surrounding pixels has a pixel value of white.

5. In Paragraph 1, A pupil area measuring device that determines the core area as the center of the pupil area when the number of pixel columns or rows of the core area remaining after the above area reduction operation is less than or equal to a predetermined value.

6. In Paragraph 5, A pupil area measuring device that determines the size of the pupil area based on the size of the center of the pupil area and the number of pixel columns or rows corresponding to the number of times the area reduction operation or the area expansion operation is performed.

7. In Paragraph 1, A pupil area measuring device that determines the size of the pupil area based on the number of pixel columns or rows of a pixel group having black pixel values ​​among the pixels of the determined pupil area.

Images