Electronic device comprising vision sensor, and operation method thereof

Abstract

An electronic device according to one embodiment of the present invention may comprise: a vision sensor for outputting event data by detecting a change in the intensity of light incident from a light source; an AER buffer for storing the event data; a noise analysis unit for detecting noise by checking a relationship between the event data stored in the AER buffer; and an AER filter unit for receiving, from the AER buffer, event data corresponding to the noise detected by the noise analysis unit, and deleting same.

Description

Electronic device including a vision sensor and method of operation thereof

[0001] The present invention relates to an electronic device including a vision sensor and a method of operating the same. Specifically, the invention relates to an electronic device including a vision sensor that performs noise filtering while maintaining generated event data in the form of an Address Event Representation (AER), and a method of operating the same.

[0002] When an event (e.g., a change in light intensity) occurs, the vision sensor generates information about the event, namely an event signal, and transmits the event signal to the processor.

[0003] Meanwhile, while an event may be triggered by a valid signal indicating a change in light intensity, it may also be triggered by noise. For example, a vision sensor determines whether an event occurs based on changes in light intensity (brightness); although an event should not occur when there is no change in light intensity, an event may still occur due to electronic noise in the circuit.

[0004] It is necessary to determine whether an event was caused by noise or a valid signal, and to conduct research not only on vision sensors for accurately sensing events caused by valid signals but also on configurations capable of accurately determining events caused by valid signals.

[0005] The present invention aims to provide an electronic device including a vision sensor to which a filtering method is applied to improve the inefficient process of decoding event data in AER form, constructing an event frame, removing noise, and then encoding it back into AER form, and a method of operation thereof.

[0006] The technical problems that this embodiment aims to solve are not limited to those described above, and other technical problems can be inferred from the following embodiments.

[0007] An electronic device according to one embodiment of the present invention may include: a vision sensor that detects a change in the intensity of light incident from a light source and outputs event data; an AER buffer that stores the event data; a noise analysis unit that detects noise by verifying the relationship between the event data stored in the AER buffer; and an AER filter unit that receives and deletes event data corresponding to the noise detected by the noise analysis unit from the AER buffer.

[0008] In one embodiment, the AER buffer may be characterized by being configured as a FIFO data structure.

[0009] In one embodiment, the event data may be characterized as being in the form of an AER.

[0010] In one embodiment, the AER filter unit can delete event data stored in the AER buffer used to detect event data corresponding to the noise.

[0011] A method of operation of an electronic device according to an embodiment of the present invention may include: a step of detecting a change in the intensity of light incident from a light source and outputting event data; a step of storing the event data in an AER buffer; a step of detecting noise by checking the relationship between the event data stored in the AER buffer; and a step of deleting the event data corresponding to the detected noise.

[0012] According to the electronic device including a vision sensor and the method of operation thereof according to the present invention, noise can be removed more directly and quickly than in the past, and thereby the accuracy of frame recognition of the electronic device can be improved.

[0013] FIG. 1 is a block diagram illustrating an image processing system according to an embodiment of the present invention.

[0014] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present invention.

[0015] FIG. 3 is a block diagram showing the configuration of a vision sensor according to one embodiment of the present invention.

[0016] FIG. 4 is a conceptual diagram showing the schematic configuration of a pixel array according to one embodiment.

[0017] FIG. 5 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present invention.

[0018] FIG. 6 is a conceptual diagram showing the operation of an electronic device according to one embodiment of the present invention.

[0019] Hereinafter, various embodiments of the present invention are described with reference to the accompanying drawings. The present invention is not limited to specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present invention. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0020] In this document, expressions such as "have," "can have," "include," or "can include" refer to the existence of the relevant feature (e.g., numerical values, functions, actions, or components, etc.) and do not exclude the existence of additional features.

[0021] In this document, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0022] Expressions such as "first," "second," "first," or "second" used in this document may modify various components regardless of order and / or importance, and are used merely to distinguish one component from another without limiting such components. For example, without departing from the scope of rights set forth in this document, the first component may be named the second component, and similarly, the second component may be renamed the first component.

[0023] As used in this document, the expression "configured to" may be replaced, depending on the context, with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean "specifically designed to."

[0024] In this document, terms transmitted or received between the first electronic device(s) and the second electronic device(s), such as “command,” “instruction,” “control information,” “message,” “information,” “data,” “packet,” “data packet,” “intent,” and / or “signal,” may include or refer to humanly perceptible ideas or specific electrical representations (e.g., digital codes / analog physical quantities) without being limited by their expression. It will be obvious to a person skilled in the art to which the invention disclosed in this document pertains that the exemplary expressions listed above may be interpreted in various ways depending on the context in which they are used. In this document, “a is greater than B” means not only that “a is greater than B” but also includes the meaning that “a is equal to or greater than B”.

[0025] The terms used in this document are used merely to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art described in this document. Terms used in this document that are defined in general dictionaries may be interpreted as having the same or similar meaning as they have in the context of the relevant technology, and are not to be interpreted in an ideal or overly formal sense unless explicitly defined in this document. In some cases, even terms defined in this document may not be interpreted to exclude the embodiments of this document.

[0026] FIG. 1 is a block diagram for explaining an image processing system (1) according to an embodiment of the present invention.

[0027] The image processing system (1) may include a light source (10) and an electronic device (20).

[0028] According to an embodiment of the present invention, the light source (10) may include at least one light-emitting element. For example, the light source (10) may include an LED, a laser diode, and a VCSEL (Vertical Cavity Surface Emitting Laser).

[0029] According to an embodiment of the present invention, the electronic device (20) may include a vision sensor (21) and a processor (25).

[0030] The vision sensor (21) can collect light incident from the light source (10). The vision sensor (21) can detect a change in the intensity of the incident light and output an event signal. For example, if an event occurs in which the intensity of the light increases, the vision sensor (21) can output a corresponding positive event. Conversely, if an event occurs in which the intensity of the light decreases, the vision sensor (21) can output a negative event.

[0031] According to an embodiment of the present invention, the vision sensor (21) may be a dynamic vision sensor. For example, the vision sensor (21) may access a pixel where a change in light intensity is detected and output an event signal. For example, a change in light intensity may be generated from the movement of a subject being captured by the vision sensor (21) or from the movement of the vision sensor (21) itself. The event signal may be generated mainly from the outline of the subject.

[0032] Since the vision sensor (21) outputs only the value corresponding to the light whose intensity changes, the amount of processing data can be reduced compared to other sensors (e.g., CMOS image sensor, etc.).

[0033] The vision sensor (21) can detect changes in the intensity of the collected light and provide an event signal to the processor (25).

[0034] The processor (25) can receive an event signal from the vision sensor (21). The processor (25) can respond according to the situation based on the event signal. For example, the processor (25) can execute or change the operation of the electronic device (20) based on the event signal.

[0035] According to an embodiment of the present invention, the processor (25) can convert raw data captured by the sensor (21) into a high-quality image. The processor (25) can improve image quality by removing noise occurring in a low-light environment. The processor (25) can adjust color imbalances caused by lighting conditions or sensor limitations. The processor (25) can perform noise removal, scene recognition, real-time object recognition, etc. by utilizing a pre-established artificial intelligence model.

[0036] FIG. 2 is a block diagram showing the configuration of a vision sensor (21) according to one embodiment of the present invention.

[0037] According to one embodiment of the present invention, the vision sensor (21) may include an event detection circuit (210) and an output buffer (330). The components (210, 230) of the vision sensor (210) shown in FIG. 2 are merely exemplary components for explaining the method of operation of the electronic device (20) according to one embodiment of the present invention. That is, it is evident that the electronic device (20) according to one embodiment of the present invention may additionally include other components other than those shown.

[0038] According to an embodiment of the present invention, the event detection circuit (210) may include a pixel array (211), a column AER (Address Event Representation) circuit (213), and a row AER circuit (215).

[0039] The pixel array (211) may include at least one pixel that detects an event based on a change in incident light luminance. The pixel array (211) may include at least one pixel in which at least one row and at least one column are arranged in a matrix form.

[0040] According to an embodiment of the present invention, a pixel may be provided with a photoelectric conversion element that generates a charge according to the luminance of incident light. When the pixel detects a change in the luminance of incident light based on the photocurrent flowing from the photoelectric conversion element, it may provide a request for reading from the pixel to a column AER circuit (213) and a row AER circuit (215). The pixel may output an event signal indicating that an event has been detected by the column AER circuit (213) and the row AER circuit (215).

[0041] Specifically, a signal indicating that an event has occurred in which the light intensity increases or decreases in a pixel included in the pixel array (211) may be provided from the pixel to the column AER circuit (213). For example, the pixel array (211) may detect a change in light from each pixel and output an output voltage, and determine whether an event has occurred by comparing the output voltage with a preset threshold value. When an event occurs, the pixel array (211) may provide an event signal to the column AER circuit (213).

[0042] The column AER circuit (213) can provide an acknowledgment signal (ACK) to the pixel in response to a signal received from the pixel that detected the event. The pixel that received the acknowledgment signal (ACK) can provide polarity information of the event that occurred to the row AER circuit (215).

[0043] The column AER circuit (215) can generate the column address of the pixel that detected the event based on the signal received from the pixel that detected the event.

[0044] The low AER circuit (215) can receive polarity information from a pixel that detected an event. Based on the polarity information, the low AER circuit (215) can generate a timestamp containing information about the time at which the event occurred.

[0045] The low AER circuit (215) can provide a reset signal to the pixel where the event occurred in response to polarity information. The reset signal can reset the pixel where the event occurred. Furthermore, the low AER circuit (215) can generate a low address of the pixel where the event occurred.

[0046] According to an embodiment of the present invention, a pixel can detect the presence or absence of an event by comparing a photocurrent corresponding to the luminance of incident light with a predetermined threshold value. For example, if the amount of change in luminance is greater than a preset threshold value, the pixel can detect that change as a positive event. As another example, if the amount of change in luminance is smaller than a preset threshold value, the pixel can detect that change as a negative event.

[0047] The event detection circuit (210) can provide information about the generated event, column address, row address, polarity information and timestamp to the output buffer (230).

[0048] The output buffer (230) can generate a packet based on a column address, a row address, polarity information, and a timestamp. The output buffer (230) can add a header indicating the start of the packet to the front of the packet and a tail indicating the end of the packet to the back. The output buffer (230) can output event data in the form of an AER. According to an embodiment of the present invention, the event data in the form of an AER refers to the generated packet.

[0049] FIG. 3 is a conceptual diagram showing the schematic configuration of a pixel array (211) according to one embodiment.

[0050] Multiple unit pixels (PX) arranged in an MxN array form can be connected to multiple row lines (ROW 0 to ROW M-1) and multiple column lines (COL 0 to COL N-1).

[0051] A plurality of unit pixels (PX) included in the pixel array (210) can be scanned in row (ROW) units or column (COL) units. In one embodiment, while sequentially scanning a plurality of columns (COL), pixel signals can be detected from the unit pixels (PX), and event signals can be generated using the pixel signals.

[0052] FIG. 4 is a block diagram showing the configuration of an electronic device (20) according to an embodiment of the present invention.

[0053] An electronic device (20) according to one embodiment of the present invention may include a vision sensor (410), an AER buffer (430), a noise analysis unit (450), and an AER correction unit (470). The vision sensor (410) may correspond to the vision sensor (21) shown in FIG. 2. For reference, the components (410, 430, 450, 470) of the electronic device (20) shown in FIG. 4 are merely exemplary components for explaining the operation method of the electronic device (20) according to one embodiment of the present invention. That is, it is evident that the electronic device (20) according to one embodiment of the present invention may additionally include other components other than those shown.

[0054] The vision sensor (410) can detect changes in the intensity of collected light and generate event data. The generated event data may be in the form of an AER. According to an embodiment of the present invention, the event data in the form of an AER may include respective information regarding a column address, a row address, polarity information, and a timestamp.

[0055] The vision sensor (410) can provide the generated event data to the AER buffer (430).

[0056] The AER buffer (430) may be configured as a data structure that sequentially inputs and sequentially outputs (FIFO) the provided event data. According to an embodiment of the present invention, the AER buffer (430) may store event data in the form of AER input in 2D Raster Scan order according to a FIFO structure.

[0057] The AER buffer (430) can temporarily store event data while operations necessary for noise removal are being performed. For example, the AER buffer (430) can temporarily store the event data while the noise analysis unit (450) performs operations to determine the noise.

[0058] The AER buffer (430) can store event data based on the size of the kernel used for filtering. For example, if the kernel used for filtering is 3x3, 3 event data can be stored in the AER buffer (430).

[0059] The noise analysis unit (450) can detect event data that is properly recognized (hereinafter referred to as normal data) and event data that is incorrectly recognized (hereinafter referred to as noise) by checking the relationship between event data stored in the AER buffer (430). The noise analysis unit (450) can provide information about the detected noise to the AER filter unit (470).

[0060] The AER filter unit (470) can receive event data corresponding to the noise detected by the noise analysis unit (450) from the AER buffer (430) and delete it in its original AER form.

[0061] According to various embodiments of the present invention, the AER filter unit (470) can modify event data corresponding to noise detected by the noise analysis unit (450) into normal data in the AER buffer (430).

[0062] Meanwhile, the AER filter unit (470) can delete normal data left in the AER buffer (430) which is unnecessary for additional noise detection operations when noise is detected.

[0063] FIG. 5 is a flowchart showing the operation of an electronic device (20) according to one embodiment of the present invention.

[0064] In step S501, the vision sensor (410) can detect event data. The vision sensor (410) can detect a change in the intensity of the collected light and provide the event data to the AER buffer (430).

[0065] In step S503, the AER buffer (430) can store the provided event data in a FIFO structure.

[0066] In step S505, the noise analysis unit (450) can detect normal data and noise by checking the relationship between event data stored in the AER buffer (430). The noise analysis unit (450) can provide information about the detected noise to the AER correction unit (470).

[0067] In step S507, the AER filter unit (470) can delete event data corresponding to the noise detected by the noise analysis unit (450) from the AER buffer (430) in its original AER form. Meanwhile, the AER filter unit (470) can delete normal data remaining in the AER buffer (430) that is unnecessary for additional noise detection operations because noise has been detected.

[0068] FIG. 6 is a conceptual diagram illustrating the operation of an electronic device (20) according to an embodiment of the present invention. FIG. 6 illustrates the process of storing and deleting event data in an AER buffer (430) over time. The first to fifth event data (A to E) are all event data in the form of AER. For convenience of explanation, the third event data (C) is assumed to be noise.

[0069] The AER buffer (430) can receive the first event data (A), the second event data (B), and the third event data (C) sequentially according to the scanning order of the vision sensor (410) and store them in a FIFO structure.

[0070] The AER buffer (430) can receive and store the first event data (A) from the vision sensor (410).

[0071] The noise analysis unit (450) does not determine whether there is noise for the first event data (A) because there is no data adjacent to the first event data (A) in the AER buffer (430).

[0072] The AER buffer (430) can receive and store the second event data (B) from the vision sensor (410).

[0073] The noise analysis unit (450) can determine the second event data (B) as normal data based on the first event data (A) adjacent to the second event data (B). The noise analysis unit (450) can provide determination information regarding the second event data (B) to the AER filter unit (470).

[0074] The AER buffer (430) can receive and store the third event data (C) from the vision sensor (410).

[0075] The noise analysis unit (450) can determine the third event data (C) as noise based on the first event data (A) and the second event data (B) adjacent to the third event data (C). The noise analysis unit (450) can provide determination information regarding the third event data (C) to the AER filter unit (470).

[0076] The AER filter unit (470) can receive the third event data (C) determined to be noise from the AER buffer (430) and remove it or modify it into normal data.

[0077] The AER filter section (470) can delete the first event data (A) and the second event data (B) that are stored in the AER buffer (430) from the AER buffer (430) because they have already been used to detect noise in the third event data (C).

[0078] The AER buffer (430) can receive and store the fourth event data (D) from the vision sensor (410).

[0079] The noise analysis unit (450) does not determine whether there is noise for the fourth event data because there is no data adjacent to the fourth event data (D) in the AER buffer (430).

[0080] The AER buffer (430) can receive and store the fifth event data (E) from the vision sensor (410).

[0081] The noise analysis unit (450) can determine the fifth event data (E) as normal data based on the fourth event data (D) adjacent to the fifth event data (E). The noise analysis unit (450) can provide determination information regarding the fifth event data (E) to the AER filter unit (470).

[0082]

[0083] Although all components constituting an embodiment of the present invention have been described above as being combined or operating in combination, the present invention is not necessarily limited to such embodiments. That is, within the scope of the purpose of the present invention, all components may be selectively combined in one or more ways to operate.

[0084] Meanwhile, the various embodiments described herein may be implemented by hardware, middleware, microcode, software and / or combinations thereof. For example, the various embodiments may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions presented herein, or combinations thereof.

[0085] Additionally, for example, various embodiments may be stored or encoded on a computer-readable medium containing instructions. Instructions stored or encoded on a computer-readable medium may enable a programmable processor or other processor to perform a method, for example, when the instructions are executed. A computer-readable medium includes a computer storage medium, and the computer storage medium may be any available medium accessible by a computer. For example, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage media, magnetic disk storage media or other magnetic storage devices.

[0086] Such hardware, software, firmware, etc., may be implemented within the same device or in individual devices to support the various operations and functions described in this specification. Additionally, components, units, modules, components, etc., described as "parts" in this invention may be implemented together or individually as separate but interoperable logic devices. Descriptions of different features of modules, units, etc., are intended to highlight different functional embodiments and do not necessarily imply that they must be realized by individual hardware or software components. Rather, functions associated with one or more modules or units may be performed by individual hardware or software components or integrated within common or individual hardware or software components.

[0087] Although operations are depicted in a specific order in the drawings, it should not be understood that these operations must be performed in the specific order depicted or in a sequential order to achieve the desired result, or that all depicted operations must be performed. In any environment, multitasking and parallel processing may be advantageous. Furthermore, the distinction of various components in the above-described embodiments should not be understood as requiring such distinction in all embodiments, and it should be understood that the described components may generally be integrated together into a single software product or packaged into multiple software products.

[0088] The electronic device, server, or external device according to the various embodiments of the present document described above may include, for example, at least one of a smartphone, tablet PC, mobile phone, video phone, desktop PC, laptop PC, PDA (personal digital assistant), PMP (portable multimedia player), MP3 player, mobile medical device, camera, or wearable device.

[0089] According to various embodiments, the wearable device may include at least one of an accessory type (e.g., a watch, ring, bracelet, anklet, necklace, glasses, contact lens, or head-mounted device (HMD)), a fabric or clothing integrated type (e.g., electronic clothing), a body-attached type (e.g., a skin pad or tattoo), or a bio-implantable type (e.g., an implantable circuit).

[0090] In some embodiments, the electronic device or external device may be a home appliance. The home appliance may include, for example, at least one of a television, a DVD player (Digital Video Disk player), audio, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box, a game console, an electronic dictionary, an electronic key, a camcorder, or a digital photo frame.

[0091] In another embodiment, the electronic device, external device, and wearable device may include at least one of various medical devices (e.g., various portable medical measuring devices (blood glucose meter, heart rate monitor, blood pressure monitor, or body temperature monitor, etc.), MRA (magnetic resonance angiography), MRI (magnetic resonance imaging), CT (computed tomography), imaging device, or ultrasound device, etc.), navigation device, satellite navigation system (GNSS (Global Navigation Satellite System)), EDR (event data recorder), FDR (flight data recorder), automotive infotainment device, home robot, or Internet of Things device (e.g., light bulb, various sensor, electric or gas meter, sprinkler device, fire alarm, thermostat, street light, exercise equipment, hot water tank, heater, boiler, etc.).

[0092] As described above, the best embodiments have been disclosed in the drawings and specification. Specific terms have been used herein, but they are used only for the purpose of describing the invention and are not intended to limit the meaning or the scope of the invention as described in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the invention should be determined by the technical spirit of the appended claims.

Claims

1. A vision sensor that detects changes in the intensity of light incident from a light source and outputs event data; An AER buffer that stores the above event data; A noise analysis unit that detects noise by verifying the relationship between event data stored in the above AER buffer; and An AER filter unit that receives and deletes event data corresponding to the noise detected by the noise analysis unit from the AER buffer. An electronic device including 2. In Claim 1, The above AER buffer is Characterized by being composed of a FIFO data structure Electronic device.

3. In Claim 1, The above event data is characterized by being in the AER format. Electronic device.

4. In Claim 1, The above AER filter section Deleting event data stored in the AER buffer used to detect event data corresponding to the above noise Electronic device.

5. In the method of operating an electronic device, A step of detecting a change in the intensity of light incident from a light source and outputting event data; A step of storing the above event data in the AER buffer; A step of detecting noise by verifying the relationship between event data stored in the above AER buffer; and Step of deleting event data corresponding to the detected noise above A method of operation of an electronic device including

Images