Single-event architecture for in-sensor visual localization

Abstract

A single-event architecture is presented for in-sensor visual localization. The architecture includes: a pixel array comprised of a plurality of photodetectors; a plurality of analog-to-time converters interfaced with the pixel array, and a temporal processor interfaced with the plurality of analog-to-time converters. Each analog-to-time converter in the plurality of analog-to-time-converters is configured to receive a pixel value for an image from one of the photodetectors in the plurality of photodetectors and operates to encode the pixel value from the photodetector into a delay in an output signal. The temporal processor performs edge extraction on pixel data using race logic, thereby outputting edge data for edges extracted from the pixel data, where the pixel data is comprised of the output signals received from the plurality of analog-to-time converters.

Description

Attorney Docket No. 2115-008446-WO-POASINGLE-EVENT ARCHITECTURE FOR IN-SENSOR VISUAL LOCALIZATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 751 ,346, filed on January 30, 2025. The entire disclosure of the above application is incorporated herein by reference.FIELD

[0002] The present disclosure relates to single-event architecture for in-sensor visual localization.BACKGROUND

[0003] Image sensors, ubiquitous in modern devices from AR / VR headsets to robots and vehicles, produce data streams that can exceed ten gigabytes per minute. While processing this volume of data presents significant computational challenges, the communication overhead emerges as an equally critical bottleneck — particularly in terms of latency and energy efficiency. This is evident in the Mobile Industry Processor Interface (MIPI), where delays can exceed tens of milliseconds for high-resolution images, constituting a significant portion of the time budget in latency-critical applications. Additionally, the energy required to transmit a single byte (-100 pJ) surpasses that for a multiply-accumulate operation on the same byte by two orders of magnitude.

[0004] These constraints, coupled with downstream algorithms typically operating on extracted features rather than raw data, have sparked significant interest in in-sensor computing. In-sensor extraction of valuable information has been investigated using both analog and digital methods. Analog designs leverage efficient multiply-accumulate operations to accelerate neural network execution but face known challenges in scalability (e.g., due to large and noisy analog components) and generalizability. Digital solutions, on the other hand, either require costly analog-to-digital converters (ADCs) and complex memory stacks, again hindering scalability, or rely on application-specific sampling techniques, limiting broader applicability.

[0005] This disclosure opens a new space for exploration between conventional analog and digital methods in this domain by introducing a scalable digital architecture (SEAL) with analog-like efficiency for in-sensor visual localization as seen in Figure 1. The key to blending the strengths of digital and analog lies in a shift in encoding at theAttorney Docket No. 2115-008446-WO-POAsensor output from binary to race logic’s delay-based representation as shown in Figure 2. Race logic, a hybrid of analog and digital traits, encodes values as delays from a reference signal - an analog feature - while ensuring signals are strictly 0 or 1 - digital characteristic. Information is embedded in the timing of the 0 — > 1 transition.

[0006] This section provides background information related to the present disclosure which is not necessarily prior art.SUMMARY

[0007] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0008] A single-event architecture is presented for in-sensor visual localization. The architecture includes: a pixel array comprised of a plurality of photodetectors; a plurality of analog-to-time converters interfaced with the pixel array, and a temporal processor interfaced with the plurality of analog-to-time converters. Each analog-to-time converter in the plurality of analog-to-time-converters is configured to receive a pixel value for an image from one of the photodetectors in the plurality of photodetectors and operates to encode the pixel value from the photodetector into a delay in an output signal. The temporal processor performs edge extraction on pixel data using race logic, thereby outputting edge data for edges extracted from the pixel data, where the pixel data is comprised of the output signals received from the plurality of analog-to-time converters.

[0009] In one embodiment, each analog-to-time converter includes a comparator circuit interconnected by a switch to a supply voltage, where the switch disconnects the comparator circuit from the supply voltage when power gated.

[0010] Prior to edge extraction, the temporal processor performs median filtering on the pixel data using race logic. More specifically, the temporal processor implements median filtering using a first arrival logic circuit to determine a minimum value and a last arrival circuit to determine a maximum value.

[0011] In some embodiments, the pixel array, the plurality of analog-to-time converters and the temporal processor are integrated together on a microchip.

[0012] The architecture may further include a frontend processor integrated on the microchip. The frontend processor is configured to receive edge data from the temporal processor and performs keypoint detection and tracking in the image using the edge data from the temporal processor.Attorney Docket No. 2115-008446-WO-POA

[0013] In another aspect, a method is presented for performing keypoint detection in an image. The method includes: receiving a binary image; for each pixel of the binary image, computing spatial gradients in region around a given pixel using a ternary derivative filter; for each pixel of the binary image, forming a covariance matrix for the given pixel from the spatial gradients of the given pixel; for each pixel of the binary image, determining a minimum eigenvalue of the covariance matrix; and for each pixel of the binary image, identifying the given pixel as a keypoint in response to the minimum eigenvalue exceeding a predefined threshold.

[0014] Spatial gradients may be computed by convolving pixels adjacent to the given pixel with the ternary derivative filter, where the ternary filter includes vectors [-1 0 1 ] and [-1 0 1 ]Tsuch that resulting gradients are in range of [-1 , 1 ].

[0015] In one embodiment, the covariance matrix is formed according towhere lx is the horizontal gradient for the given pixel and ly is the vertical gradient for the given pixel.

[0016] In another aspect, a method is presented for performing keypoint tracking in an image. The method includes: receiving a binary image of a previous frame; receiving a set of keypoints for a previous frame; constructing a first series of pyramid levels from the binary image of the previous frame, where each pyramid level in the first series of pyramid levels has a lower resolution than other pyramid levels in the first series of pyramid levels; and for each pixel in each pyramid level in the first series of pyramid levels, computing spatial gradient for a given pixel. The method further includes: receiving a binary image of a current frame; constructing a second series of pyramid levels from the binary image of the current frame, where each pyramid level in the second series of pyramid levels has a lower resolution than other pyramid levels in the second series of pyramid levels; for each pixel in each pyramid level in the second series of pyramid levels, computing spatial gradient for a given pixel; for each pixel in each pyramid level in the second series of pyramid levels, computing a temporal gradient in the relation to corresponding pixel in the first series of pyramid levels; and for each keypoint in the set of keypoints, determining a displacement of a given keypoint in the current frame.

[0017] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended forAttorney Docket No. 2115-008446-WO-POApurposes of illustration only and are not intended to limit the scope of the present disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0019] Figure 1 is a graph showing a data volume comparison between off / near-sensor visual localization (top line) and the proposed single event architecture approach (bottom line). The single event architecture achieves data volume reduction in two phases: first, temporal processing driven by race logic generates denoised, binarized edge images from raw sensor data, achieving a 90% reduction; second, the quantized binary implementation of keypoint detection and optical flow (frontend processing) further compresses this data into keypoint positions and displacements, resulting in over 99% reduction from the raw data volume.

[0020] Figure 2 is a diagram showing values encoded as delays, with information represented by the timing of the 0 1 transition relative to a reference point. The 0 - 1 transition for a occurs after 2 time units, At , so a represents a value of 2. Similarly, the transition for b occurs after 3 time units, meaning b represents a value of 3. Regardless of the variable’s value range and specific value, only a single wire per variable and a single event per wire per computation cycle are required.

[0021] Figure 3 depicts the proposed single-event architecture.

[0022] Figure 4 is a block diagram showing the components of the single-event architecture.

[0023] Figure 5 are diagrams showing the fundamental operations in race logic are First Arrival (FA), Last Arrival (LA), Delay (D), and Inhibit (I). Values are encoded as delays, with each unit delay, At , corresponding to a clock period in synchronous implementations. In this encoding scheme, smaller values correspond to shorter delays and larger values to longer delays, making FA act as the min operator, LA as max, D as increment and I as threshold. Implementing these operations in hardware requires only basic components: OR and AND gates, flip-flops, and augmented SR latches.

[0024] Figure 6A depicts a conventional single-slope analog-to-digital converter.

[0025] Figure 6B is a schematic of a proposed comparator design for use in an analog-to-time converter.Attorney Docket No. 2115-008446-WO-POA

[0026] Figure 6C is a graphing showing the voltage bias delay tradeoff for the proposed comparator.

[0027] Figure 7 is a timeline comparing operation of an analog-to-time converter with a conventional single-slope ADC.

[0028] Figures 8A and 8B show circuit implementations for a three pixel and five pixel median filter, respectively.

[0029] Figure 9A are examples of an edge filter operation.

[0030] Figure 9B is a schematic of an example implementation of edge extraction using race logic.

[0031] Figure 9C are diagrams illustrating waveforms for an edge extraction operation. Both horizontal and vertical edges are detected. At the end of the race logic operation, the computed horizontal and vertical edge values are merged and latched into a 1 -bit per-pixel SRAM cell.

[0032] Figure 10 is a diagram depicting an interface between the temporal processor and the frontend processor.

[0033] Figures 11A-11 D illustrate aspects of keypoint detection. In Fig. 11 A, standard GFTT spatial derivative with 24 bits per element in the covariance matrix M. In Fig. 11 B, GFTT is applied to 1 -bit edge data; using 1 -bit edges instead of 8-bit inputs reduces the bit width required to store the Z / term in M to 8 bits. In Fig. 11C, GFTT is applied to 1 -bit edge data with a ternary 2-bit derivative, replacing the 3-bit Sobel filter; this results in a reduced representation of 4 bits, with Z / xand Z / yas integers ranging from 0 to 9. In Fig. 11 D, reducing the resolution of lx and ly from 11 bits to 2 bits limits the number of possible values for the product terms; this restricted range (10 possible values) enables the use of a 100-entry lookup table (LUT) for square root calculations.

[0034] Figures 12A-12E illustrate aspects of keypoint tracking. In Fig. 12A, a three-level LK scale pyramid is shown. Data is read from patches centered on detected keypoints and processed in a vector-parallel pipeline. Downsampling filters operate with a stride of 2. In Fig. 12B, standard 5 X 5 Gaussian blur filter is applied to 1 -bit edge data for downsampling, increasing the bit width per pixel from 1 to 8. In Fig. 120, an adopted 2 X 2 Box filter is applied to 1 -bit edge data for downsampling, resulting in 3 bits per pixel. In Fig. 12D, aggressive direct sampling that eliminates storage overhead by selecting a single pixel value, without blurring during downsampling. Fig. 12E shows aAttorney Docket No. 2115-008446-WO-POAdemonstration of the three downsampling techniques. Among these techniques, box filtering provides the best balance between accuracy and resource efficiency.

[0035] Figure 13 is chart showing the energy-per-frame comparison for HybVIO on RPi4 with and without the single-event architecture.

[0036] Figure 14 is a flowchart depicting a technique for performing keypoint detection.

[0037] Figure 15 is a flowchart depicting a technique for performing keypoint tracking.

[0038] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.DETAILED DESCRIPTION

[0039] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0040] With reference to Figure 3, a scalable digital architecture 30 use of time within the race logic paradigm enables highspeed pixel-parallel computation in a stacked digital pixel sensor (DPS) architecture. This is accomplished through several key advances. In summary, the scalable digital architecture 30 introduces a new analog-to-time converter (ATC) that replaces conventional ADCs. This analog-to-time converter converts pixel array 31 outputs into digital delay-coded signals, rather than the standard weighted binary codes. The ATC is co-optimized with a temporal processor 32 to minimize area usage and energy consumption through timescale manipulation, adjustments to the unit delay length At, and the implementation of power gating techniques. For the temporal processor design, race logic’s minimal wire count is exploited, requiring only one wire per variable regardless of its value range (single-event-per-wire), and low switching activity, with precisely one event per wire for any value (single-event-per-wire). This enables data preprocessing, denoising and edge extraction, at maximum parallelism, again with minimal resource and energy costs. By extracting edges and collapsing the time dimension of delay-coded signals - converting them to binary without additional overhead - the architecture decreases data storage requirements from 10 / 14-bit SRAM per pixel to 1 bit, resulting in over a 90% reduction in storage needs and associated costs compared to raw images.

[0041] After extracting the edge images from the temporal processor 32 and storing them in a pixel SRAM, a frontend processor 33 reads them column-wise forAttorney Docket No. 2115-008446-WO-POAvector-parallel pipelined processing. For keypoint detection and tracking, the Good Features to Track (GFTT) and pyramidal Lucas-Kanade (LK) algorithms may be used, both standard methods for visual localization. The binarized edges naturally condense the range of numerical representation in these algorithms, compared to raw data. This results in a heavily quantized implementation that enables intra-column vector parallelism and minimal SRAM needs, achieving high throughput while minimizing latency. Data volume reduction exceeds 99% compared to raw images because only keypoints and tracks are sent to the host processor.

[0042] The proposed single-event architecture enables pixel-parallel digital computation for high-speed processing, suitable for environments with rapid motion. It consists of three main components: analog-to-time converters, which replace conventional analog-to-digital converters, a temporal median filter for direct denoising, and a temporal edge extractor. This design does not affect data quality, while reducing data volume by over 90% compared to raw images by transmitting only edge data. Synthesis results reveal that a Boolean implementation would have resulted in significant resource and energy overheads across all three components. Meanwhile, a race logic implementation takes 23.4x fewer gates and 22.7x less energy than its Boolean counterpart. The single-event architecture is integrated into a digital pixel sensor (DPS) that inherently supports a global shutter, where all pixels are exposed simultaneously to eliminate motion distortions. No changes are required for this integration to the top 4-transistor active pixel sensor (4T APS) array, which is standard in current image sensors.

[0043] Figure 4 depicts the core components of the single-event architecture 40. The single-event architecture is comprised of a pixel array 41 , a plurality of analog-to-time converters 42, a temporal processor 43, and a frontend processor 44. Each of these components is further described below.

[0044] The pixel array 41 is comprised of a plurality of photodetectors, such as photodiodes. The pixel array 41 converts photons captured by the photodiodes into voltage signals. Within a traditional DPS, these signals are passed through a hybrid bond to single-slope ADCs (SS-ADCs). The SS-ADCs perform the conversion from analog to digital binary in two steps: analog-to-time conversion (ATC) and time-to-digital conversion (TDC). The analog-to-time converters do not need to be placed in the bottom layer but could be placed in other layers of the chip as well.

[0045] With reference to Figure 6A, an analog-to-time converter 61 relies on a ramp generator 62, which produces a monotonically decreasing voltage ramp, and aAttorney Docket No. 2115-008446-WO-POAcomparator 63, which compares this ramp with the sampled pixel voltage. The beginning of the voltage ramp serves as an initial reference point, and the time at which the pixel voltage matches the ramp value marks an event. This process encodes the pixel voltage, corresponding to the captured light intensity, into a delay. The TDC processes this delay using a counter and a latch. The counter starts counting from the initial reference point at the beginning of the conversion process. When the comparator flips, the counter value is stored in the latch, converting the delay into a digital value.

[0046] Following this breakdown of the ADC, the idea of this disclosure is to eliminate the TDC component, unnecessary for temporal processing with race logic. Unlike Boolean logic, which requires full digital conversion, race logic operates directly on delay-coded events - the same type of output produced by an analog-to-time converter. Removing the TDC (global counter and latch) simplifies the analog-to-digital interface and creates additional opportunities for co-optimization. These benefits are evident in asynchronous race logic, where inverter chains replace flip-flops for implementing the Delay operator, reducing clocking needs.

[0047] Referring to Figure 5, the core principle of race logic lies in encoding values as temporal delays relative to a reference point; a value is represented by the time at which a signal transitions from 0 to 1. Computations are realized through the temporal interactions of signals propagating within the race logic circuit. The fundamental temporal operations - First Arrival (FA), Last Arrival (LA), Delay (D), and Inhibit (I) — implement the min, max, increment, and threshold operators, respectively, under the assumption that shorter delays correspond to smaller values and longer delays to larger values. These operators form a complete set for realizing any invariant and causal function over the domain of positive integers and infinity and can be implemented using CMOS standard cells.

[0048] The output type of a standard ATC design matches the input type of race logic circuits. However, it may not yield optimal designs. The number of inverters in an inverter chain is determined by the delay of each inverter and the length of the delay unit, At . To reduce the number of inverters required, either the inverters must operate more slowly, or At must be reduced. Slowing down the inverters can be achieved by lowering their bias voltage, a well-established case in low-voltage design. Alternatively, shortening At requires both the ramp generator and the comparator to operate faster. Here, the focus is on the comparator (Fig. 6B), as ramp generators in pixel parallel implementationsAttorney Docket No. 2115-008446-WO-POAare shared across ADC / ATC arrays, making their cost less critical. The relationship between voltage bias and delay for the comparator is shown in Fig. 6C.

[0049] Increasing the comparator’s bias voltage (slowing down the circuit, as it includes a PMOS bias transistor) reduces power consumption, whereas decreasing the bias voltage (as is the case here to speed up the comparator) increases power consumption. However, even with higher power consumption in the latter case, the energy consumption can still be significantly lower compared to that of a typical ADC design. The key insight is that a faster, more power intensive comparator is paired with a faster ramp generator. This results in a steeper ramp slope, effectively compressing the timeline. Consequently, the comparator needs to remain active for a much shorter duration - on the order of tens to hundreds of ns in this case, compared to tens to hundreds of / zs in traditional designs.

[0050] This insight is leveraged to enhance the design of an analog-to-time converter with a common power-gating mechanism: a switch that disconnects the supply voltage to the comparator when it is not needed. A timeline illustrating the comparator’s on / off states, along with its wake-up and shut-down times, is shown in Fig. 7.

[0051] Returning to Figure 4, a plurality of analog-to-time converters 42 are interfaced with the pixel array 41. Each analog-to-time converter in the plurality of analog-to-time-converters is configured to receive a pixel value for an image from one of the photodetectors in the pixel array 41 and operates to encode the pixel value from the photodetector into a delay in an output signal.

[0052] The temporal processor 43 is in turn interfaced with the plurality of analog-to-time converters 42. During operation, the temporal processor 43 performs edge extraction on pixel data using race logic, where the pixel data is comprised of the output signals received from the plurality of analog-to-time converters. In this way, edge data for edges is extracted from the pixel data.

[0053] Prior to edge extraction, the temporal processor 43 performs median filtering on the pixel data using race logic. Data denoising is a standard preprocessing step aimed at reducing the impact of sensor noise on image quality and subsequent analysis. This noise can obscure true image features, such as edges (particularly in the case of salt-and-pepper and Gaussian noise), making it difficult to distinguish between real image details and noise artifacts, especially in low-light conditions.Attorney Docket No. 2115-008446-WO-POA

[0054] Median filtering smooths out random fluctuations in an image by replacing each pixel value with the median of its neighboring pixels within a specified window size, using a sliding window approach. The median is computed through a sorting process. In one example implementation, a truncated bitonic sorter is employed, which is particularly suitable for race logic as it relies solely on min and max functions.

[0055] Figures 8A and 8B illustrate example designs for a filter window spanning three pixels and five pixels, respectively. Increasing the filter window size from a baseline of three pixels to five pixels (placed in a cross pattern) improves denoising effectiveness and can enhance the overall accuracy of the system. However, this improvement comes at a cost. The larger window requires four additional gates per pixel, resulting in gate counts of 4 and 8 for the three-pixel and five-pixel temporal implementations, respectively. The pixel-level energy consumption is 2.5 fJ and 5.4 fJ per frame for these implementations.

[0056] For comparison, a Boolean implementation, assuming a standard 10-bit ADC resolution, demands significantly higher resources, with 250 gates (or 25.4 / zm2) required for three-pixel windows and 790 gates (or 220.4 / zm2) for five-pixel windows, based on synthesis results obtained using the Synopsys Design Compiler in the TSMC 22 nm process. These area figures come close to or exceed the common pixel sizes of image sensors from recent literature, ranging from 2 / zm2to 50 / zm2. Energy consumption is estimated at 0.2 pJ and 1.1 pJ per pixel per frame, respectively.

[0057] The next stage in the pipeline is edge extraction, which further reduces data volume without losing useful information. Edges provide sufficient structural detail for keypoint detectors and trackers to identify corners — the primary targets of algorithms like Features from accelerated segment test (FAST), Good Features to Track (GFTT), and LK optical flow.

[0058] Consider a pixel part of an edge if it experiences an intensity change with a magnitude greater than a threshold N. Given pixel values a and b, an edge is present if |a -b\ > N. In an example implementation, where pixel intensity is represented as a delay, the threshold N is similarly mapped to a delay, enabling computation through race logic. To avoid direct subtraction between a and b, the edge condition is reformulated as max a, b) > min(a, b) +N, which aligns with race logic’s native min, max, and increment operations.

[0059] For the conditional part of the expression, the Inhibit (threshold) operator is used, with max a, b) as the inhibiting signal and min(a, b) + N as the data signal.Attorney Docket No. 2115-008446-WO-POAAccording to the definition of Inhibit, its output is a temporal signal that toggles to indicate an edge when min(a, b) +N arrives before max a, b). Accuracy results show that the temporal part of this information adds minimal value, thus one can simplify by collapsing the time dimension (translating °° to 1 ’bO andto 1 ’b 1 ) and storing only 1 -bit per pixel.

[0060] Figures 9A-9C illustrate the circuitry of the proposed race logic and the resulting image after edge extraction on the denoised output of Fig. 8A with N = Q. The hardware implementation demands 19 gates for an asynchronous operation, with the delay unit At set equal to a 4-inverter delay (~1 ns), and consumes 15.1 fJ of energy per pixel per frame. In comparison, a Boolean implementation requires 289 logic gates (or 93.8 / ini2), exceeding (similarly to a median filter Boolean implementation) the pixel size of modern image sensors and consuming 0.2 pJ of energy per frame.

[0061] For simplicity, N is fixed in the designs. Introducing a variable edge threshold, for example by adjusting the voltage ramp duration through modification of the ramp generator’s voltage bias could potentially improve accuracy.

[0062] Next, a frontend processor consumes the edge data produced by the temporal processor and performs keypoint detection and tracking in the image. In an example embodiment, the keypoint detection is performed using GFTT and tracking is performed with pyramidal LK optical flow. The temporal processor already reduces data volume by 90% compared to raw data, lowering MIPI communication costs. Bringing this computation into the sensor, rather than using an out-of-sensor accelerator, boosts data reduction to >99% and minimizes DRAM requirements (and associated latency) for the host processor.

[0063] Referring to Figure 10, the proposed frontend processor architecture streams data from the pixel SRAMs in a column-wise fashion, performing keypoint detection in parallel within each column and in a pipelined manner across columns. Synthesis results show that such parallelism would not fit within the sensor area without optimizations. To satisfy area constraints, 1 -bit edge data is used instead of the typical 8-bit inputs to GFTT and smaller, lower-precision filters are chosen.

[0064] The keypoints of the current frame are detected using GFTT and stored in a keypoint SRAM. In the next frame, the stored keypoints are retrieved to calculate the temporal gradient between the current and previous frames, along with the spatial gradients of the previous frame, as part of the LK optical flow algorithm. Although both spatial and temporal gradients present opportunities for reuse by storing precomputed values in another SRAM, one finds that performing the computation twice is more efficientAttorney Docket No. 2115-008446-WO-POAin terms of both energy consumption and area. To track larger displacements, the proposed single-event architecture also supports optical flow pyramids, again deploying aggressive quantization techniques and smaller-sized filters to decrease SRAM usage and increase parallelism.

[0065] Figure 14 depicts an example embodiment for detecting keypoints in an image. Upon receiving a binary image at 141 , spatial gradients are computed at 142 for each pixel of the binary image using a ternary derivative filter. For each pixel of the binary image, a covariance matrix is formed at 143 from the spatial gradients of a given pixel, and a minimum eigenvalue of the covariance matrix is determined at 144. Pixels in the binary image are identified as keypoints at 145 in response to the minimum eigenvalue exceeding a predefined threshold. Each of these steps are described in more detail below.

[0066] In the example embodiment, a GFTT algorithm detects keypoints in images, particularly corners, by calculating the spatial gradients in local regions around pixels. These gradients measure the change in intensity in the horizontal ( / x) and vertical ( / y) directions. While edge detection primarily focuses on the magnitude of the gradient, corner detection considers both the magnitude and direction of the gradient.

[0067] This is reflected in the formulation of the GFTT covariance matrix M, which encapsulates the distribution of intensity change in a 3x3 window W centered around each pixel. The matrix M is calculated as follows:

[0068] The eigenvalues of M characterize the types of image regions. In flat regions, both eigenvalues are small; in edge regions, one eigenvalue is large while the other remains small; and in corner regions, both eigenvalues are large. To detect corners, the GFTT algorithm identifies regions where the minimum eigenvalue ( min) is large, indicating significant intensity variations in all directions within the window.

[0069] Calculating Amin can be computationally expensive as it involves complex operations such as the square root. This difficulty is evidenced in the keypoint detection acceleration literature, where simpler but less precise methods like FAST are commonly adopted. Instead of compromising accuracy, one can prioritize GFTT and address its computational complexity through multi-level quantization. This involves using 1 -bit edge inputs and ternary derivative filters, simplifying the hardware implementation.Attorney Docket No. 2115-008446-WO-POA

[0070] For a sensor resolution of 752x480, the proposed vector-parallel processing unit occupies 137.1 / zm2, or 0.7% of the pixel array area assuming a pixel pitch of 5 / zm. It consumes 0.4 pJ of energy per pixel per frame and completes detection for the entire frame in under 760 frontend processor cycles. In comparison, an 8-bit vector-parallel GFTT implementation would occupy 5397.3 / zm2, making it 39.4x larger than the proposed design and equal to 28% of the pixel array area. Additionally, it would consume 139.1 pJ per frame, which is over 300x higher than our solution. A non-vector-parallel version of this GFTT processing unit would reduce hardware requirements but result in significantly higher per-pixel per-cycle latency, exceeding 752 x 480 = 360, 960 cycles — roughly 480x the cycles required by SEAL’S GFTT accelerator.

[0071] In the reference GFTT implementation with 8-bit data inputs, each element of the covariance matrix M is represented using 24 bits (Fig. 11 A). In contrast, 1 -bit data inputs are used (i.e. , the binarized edge data produced by the temporal processor), which reduces the bit width of each M element to 8 bits (Fig. 11B). The resulting energy and area improvements are validated through synthesis using Synopsys Design Compiler and PrimeTime PX in TSMC 22 nm technology, with the results summarized in Table 4.Table 4

[0072] A baseline for hardware comparison is a 16-bit fixed-point (16b FXP) GFTT processing unit operating on 8-bit input data. This is achieved by normalizing and quantizing the original 24-bit representation to a 16-bit fixed-point representation, more suitable for hardware realization. Transitioning from this 16-bit implementation to the proposed implementation that operates on the 1 -bit edge data achieves a 70x reduction in energy consumption and a 5x reduction in area. The energy savings stem from two factors: reduced resource needs and the sparsity of active edge pixels. Analysis shows that only 5 to 30% (Table 3) of pixels are non-zero, resulting in low switching activity.

[0073] In terms of accuracy for the VIO task, simulation results obtained using the HybVIO framework on the EuRoC dataset indicate an average error reduction of 3 cm, from 17.5 cm (baseline, 8-bit data) to 14.5 cm (SEAL, 1 -bit edge data). When usingAttorney Docket No. 2115-008446-WO-POAVINS-Mono, the transition from raw images (baseline) to edge data (SEAL) increases the average error by 0.3 cm, from 18.4 cm to 18.7 cm.

[0074] Next, the Sobel filter, used in the original implementation, is replaced with a ternary (3VL) derivative (Fig. 11C). The Sobel filter produces values in the range [-4, 4] when convolved with a binary edge image, as its kernel values range from -2 to 2. This requires 4-bit wide gradients. The ternary derivative replaces the standard kernels with the 1 D vectors [-1 0 1] and [-1 0 1]r. The resulting gradients are within the range [-1 , 1], enabling representation with 2 bits. Consequently, each element of M is represented with 4 bits.

[0075] This quantization reduces the representation space, enabling the singleevent architecture to perform square root operations with a look-up table featuring 100 entries (Fig. 11 D). Table 4 provides accuracy and implementation results, comparing both against a baseline solution and the implementation described in this section.

[0076] Figure 15 depicts an example embodiment for tracking keypoints in an image. A binary image for a current frame and a previous frame, along with a set of keypoints for the previous frame, serve as inputs. A first series of pyramid levels are first constructed at 151 from the binary image of the previous frame, where the binary image of the previous frame serves as base of the pyramid and each subsequent pyramid level has a lower resolution than the preceding pyramid level. Specifically, the pyramid levels are constructed by downsampling the preceding level to half resolution. For each pixel in each pyramid level in the first series of pyramid levels, a spatial gradient for a given pixel is computed at 152.

[0077] Next, a second series of pyramid levels are constructed at 153 from the binary image of the current frame, where the binary image of the current frame serves as base of the pyramid and each subsequent pyramid level has a lower resolution than the preceding pyramid level. For each pixel in each pyramid level in the second series of pyramid levels, a spatial gradient for a given pixel is computed at 154. Additionally, for each pixel in each pyramid level in the second series of pyramid levels, a temporal gradient is computed at 155 in the relation to corresponding pixel in the first series of pyramid levels. For each keypoint in the set of keypoints, a displacement of a given keypoint in the current frame is determined at 156. Each of these steps are described in more detail below.

[0078] For keypoint tracking, the LK optical flow algorithm tracks keypoints across images by computing spatial gradients ( / %, ly) and temporal gradients It). Unlike GFTT,Attorney Docket No. 2115-008446-WO-POAwhich also computes spatial gradients, the operational window W in this case is significantly larger, typically ranging from 13 x 13 to 31 x 31. It also applies specifically to keypoints, rather than all pixels in the image. Considering this, one can read out only the data belonging to such windows, or patches, for processing. From an implementation perspective, this means that instead of activating all rows of the SRAM for a full-column readout, one can drive just the rows required by the patch.

[0079] While it is theoretically possible to store the spatial gradients computed by GFTT in SRAM for some reuse by the LK algorithm, recomputing these gradients proves to be advantageous. This is because: (1) processing frequency: GFTT is applied only to keyframes because detecting new keypoints is unnecessary for every frame, whereas the LK algorithm processes all frames to track keypoint movements; (2) implementation efficiency: analytical results indicate that storing spatial gradients incurs a 3x storage overhead compared to the existing pixel SRAM; and (3) temporal gradient: preserving spatial gradients does not eliminate the need to store the previous frame, as it is still required for temporal gradient computation.

[0080] Opportunities to reuse temporal gradients are also present. Between iterations, patch locations often shift minimally, leading to significant overlap with patches from previous iterations. However, as with spatial gradient reuse, the storage cost outweighs the benefits of recomputation. This is especially true for edge images, where storing temporal gradients requires 2x the storage capacity of the original pixel data. For all gradient computations, one can deploy the same quantization methods as in keypoint detection.

[0081] In the example embodiment, the LK pyramidal algorithm enhances keypoint tracking through a multi-scale approach that handles large pixel displacements between frames (Fig. 12A). Starting with a three-level image pyramid, where each level is downsampled to half the resolution of the previous one, the algorithm begins estimation at the coarsest level where large motions are more easily captured. The process then iteratively refines the flow estimate by using each level’s result as an initial guess for the next finer level, selecting new image patches around these estimates until reaching the base level with full resolution.

[0082] During the pyramid construction, a 5 x 5 Gaussian blur filter is typically applied to suppress noise and prevent aliasing. This filtering operation transforms a 1 -bit edge representation into 8-bit values (Fig. 12B), increasing storage requirements. Although each pyramid level contains progressively fewer pixels (1 / 4 of the previousAttorney Docket No. 2115-008446-WO-POAlevel), the transition from 1 bit per pixel (edge image before the Gaussian filter) to 8 bits per pixel (edge image after the Gaussian filter) means that the first downsampled level, for example, requires 2x the storage of the edge frame. Further details for the pyramidal implementation may be found in an article entitled “Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the Algorithm” by Jean-Yves Bouguet which is incorporated herein by reference.

[0083] This implementation adopts a 2x2 Box filter (Fig. 120) as an alternative to the 5x5 Gaussian filter. This reduces SRAM requirements from 8 to 3 bits per pixel while maintaining comparable accuracy (~1 cm variation for both HybVIO and VINS-Mono). Although similar SRAM savings could be achieved by quantizing the outputs of the Gaussian blur, the Box filter requires fewer operations. Additionally, the single-event architecture’s vector-parallel processing within patches enables SRAM reuse, allowing a single SRAM to serve all pyramid layers instead of requiring three separate ones. This parallelism supports on-the-fly recalculation of pyramid levels, eliminating the need for storing and accessing precomputed values. In contrast, sequential pixel processing would introduce a latency bottleneck during frame-wide downsampling without parallelization.

[0084] With reference to Figure 12D, an aggressive optimization strategy that eliminates such SRAM overheads by directly sampling a single pixel from each 2 x 2 Box window was also investigated. This method preserves the original 1 -bit per pixel representation, but it degrades the localization accuracy by 2.2x compared to previous approaches. Table 5 below summarizes the comparative results.&Table 5

[0085] For each keypoint in the set of keypoints, the displacement of the keypoint between the previous frame and the current frame is computed iteratively across the pyramid levels. At the coarsest pyramid level, an initial displacement estimate is initializedAttorney Docket No. 2115-008446-WO-POAto zero. For a given keypoint, a window of pixels centered at the keypoint is defined at the current pyramid level. Within this window, spatial gradient values and temporal gradient values are accumulated to form a system of equations representing the motion in the image around the keypoint. The system is solved to compute a displacement update that minimizes the image intensity mismatch between the previous and current frame within the window. The computed displacement is added to the current displacement estimate and the process is repeated a predetermined number of times. The accumulated displacement estimate is then scaled for the next finer pyramid level, where the process is then repeated to refine the final estimate. After processing the finest pyramid level, the result represents the final estimated displacement of the keypoint from the previous frame to the current frame.

[0086] For evaluation purposes, the proposed ATC implementation was developed in Cadence Virtuoso and simulated with Cadence Spectre X in the TSMC 28 nm process. This design was analyzed both in terms of energy consumption and latency, but not area. Obtaining the ATC area would require a layout under specific pixel size and sensor requirements, which is beyond the scope of this disclosure. Additionally, no extra analog components are introduced to the standard CMOS image sensor ATC.

[0087] The temporal and frontend processors were synthesized in TSMC 22 nm using the Synopsys Design Compiler and verified through VCS cycle-accurate simulation and FPGA prototyping (Intel / Altera 10 GX development kit). All results were scaled to 28 nm using DeepScaleTool to match the ATC process. Energy consumption was estimated using Synopsys PrimeTime PX with fully annotated switching activity data. For the MIPI interface, an energy model, commonly used in recent in-sensor computing literature, was employed.

[0088] To perform system-level analysis, covering the entire VIO pipeline, the non-SEAL tasks were run on three host platforms with different performance capabilities and energy requirements: a Raspberry Pi 4 (RPi4) with an ARM Cortex-A72 CPU, a mobile Intel Core i7-8750H CPU, and an AMD EPYC 7552 CPU. The use of RPi4 aligns with related studies targeting VIO applications in AR and UAV settings. The use of more powerful CPUs provides a broader perspective, particularly for backend computations such as optimization in pose estimation.

[0089] Accuracy was evaluated using the root mean square absolute trajectory error (RMS ATE) on the state-of-the-art EuRoC dataset and tested with the HybVIO and VINS-Mono frameworks under monocular VIO settings. All components of theAttorney Docket No. 2115-008446-WO-POAalgorithms, excluding SEAL, were left unchanged, and the edge threshold for edge extraction was held constant. HybVIO uses a backend based on the Multi-State Constraint Kalman Filter, while VINS-Mono uses a backend based on non-linear optimization; demonstrating SEAL’S broad compatibility. Although SEAL supports operation at higher frame rates, the evaluation was conducted using EuRoC’s default 20 fps images. A higher-fps dataset, matching SEAL’S capabilities, could potentially improve accuracy further by mitigating the impact of large displacements, currently addressed by the pyramidal LK optical flow accelerator.

[0090] Replacing ADCs with ATCs saves resources by eliminating the TDC component (global counter and TDC latches). This modification reduces the per-pixel SRAM area from 16.5 / zm2(obtained from the layout of an SRAM cell in 65 nm process and scaled to 28 nm using DeepScaleTool) to 1.65 / zm2, assuming linear scaling with bit count (going from 10 bits per pixel to 1). SEAL utilizes this freed area for embedded processing functions. Post-synthesis results of our temporal processor show that it occupies 14.6 / zm2of area, which, when combined with the pixel SRAM, totals 16.3 / zm2(similar to the baseline DPS design). As discussed above, this would not have been possible with a Boolean implementation, which demands orders of magnitude more hardware resources for equivalent functionality.

[0091] The SEAL frontend processor occupies 0.03mm2, with additional SRAMs requiring 0.3 mm2. These area requirements are in line with recent similar approaches, such as BlissCam, which takes 0.3 mm2in a 22 nm process node. In perspective, this represents a small fraction of the total pixel array (752 x 480), which spans 9.0 mm2with a 5 / zm pixel pitch.

[0092] Table 6 provides a comparative analysis between SEAL and recent related work for keypoint detection and tracking. The reported total energy consumption values include the costs associated with the DPS, as detailed in Table 7. SEAL achieves 7.9x and 35.7x energy savings compared to Navion and Robo- Visio, respectively. When considering only processing energy, SEAL delivers 1 ,5x and 28.8x savings. The smaller improvements in the latter case are due to the base DPS, which dominates energy consumption in the two off-sensor accelerator approaches (e.g., 132 pJ for ADC, SRAM readout, and MIPI communication vs. 18.2 pJ for keypoint detection and tracking for Navion).

[0093] Figure 13 shows the energy consumption analysis of the complete localization system, including RANSAC and backend optimizations. An RPi4 runningAttorney Docket No. 2115-OQ8446-WO-POAHybVIO is used as the baseline configuration. While this baseline setup requires the transmission of all raw data the RPi4 for processing, the SEAL-based configuration reduces the data volume by over 99%. This results in a 1.5x reduction in overall energy consumption for SEAL. The analysis accounts for both active and idle energy consumption of the host RPi4, with idle energy — consumed when the RPi4 is not actively processing workloads — comprising over 55% of its energy consumption.

[0094] Table 6 presents a latency comparison between Navion, RoboVisio, and SEAL. Starting from the moment the image is captured by the pixel array, SEAL completes keypoint detection and tracking for a frame in 0.8 ms - a detailed timing breakdown is available in Table 7. In contrast, Navion requires 10.1 ms (12x longer) and RoboVisio takes 26 ms (32x longer) for the same operations. Unlike energy consumption, where the base DPS dominates, the largest contributor to Navion’s and RoboVisio’s latency is the processing stage. For instance, in Navion, ADC, SRAM readout, and MIPI communication account for only 0.7 ms, while keypoint detection and tracking need 9.4 ms.Table 6Attorney Docket No. 2115-008446-WO-POATable 7

[0095] A breakdown of end-to-end frame localization latency across different platforms, all running the HybVIO framework, is provided in Table 8. The distribution and absolute values of processing times vary significantly across host platforms. For instance, backend processing consumes 63% of total processing time on the RPi4, but only 32% on the i7 processor. These variations affect SEAL’S relative performance improvements: we observe 1.5x, 2.2x, and 2. Ox latency reductions on the RPi4, i7, and EPYC platforms, respectively. Notably, SEAL shifts the performance bottleneck, on the i7 and EPYC configurations, from keypoint detection and tracking (SEAL’S targets) to backend processing, suggesting opportunities for further optimization in this area.Table 8

[0096] Table 9 presents a comparative analysis of accuracy performance for HybVIO and VINS-Mono on 11 EuRoC sequences. The baseline case assumes full execution on the host processor, independent of the specific processor choice. Integrating SEAL with HybVIO results in an average accuracy improvement of 1.0 cm, while its addition to VINS-Mono leads to an average error increase of 0.3 cm. A more detailed sequence-level analysis reveals no consistent trend. The effects of SEAL onAttorney Docket No. 2115-008446-WO-POAaccuracy vary across frameworks and sequences, ranging from an 10 cm reduction in error (e.g., on the MH_05 sequence with HybVIO) to a 4 cm increase in error (e.g., on the MH_01 sequence with with HybVIO and on the V1_01 sequence with VINS-Mono). The parameters used for execution were identical to those in the original implementation. No modifications were made to the algorithm or the HybVIO and VINS-Mono frameworks. Observed accuracy differences could potentially be addressed through parameter tuning, however, such an optimization falls outside our scope.Table 9

[0097] The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.

Claims

Attorney Docket No. 2115-008446-WO-POACLAIMSWhat is claimed is:

1. An image sensor system, comprising:a pixel array comprised of a plurality of photodetectors;a plurality of analog-to-time converters interfaced with the pixel array, each analog-to-time converter in the plurality of analog-to-time-converters is configured to receive a pixel value for an image from one of the photodetectors in the plurality of photodetectors and operates to encode the pixel value from the photodetector into a delay in an output signal; anda temporal processor interfaced with the plurality of analog-to-time converters and performs edge extraction on pixel data using race logic, thereby outputting edge data for edges extracted from the pixel data, where the pixel data is comprised of the output signals received from the plurality of analog-to-time converters.

2. The image sensor system of claim 1 wherein each analog-to-time converter includes a comparator circuit interconnected by a switch to a supply voltage, where the switch disconnects the comparator circuit from the supply voltage when power gated.

3. The image sensor system of claim 1 wherein, for each pair of pixels, the temporal processor detects an edge condition according tomax(a,b) > min(a,b) +Nwhere max is a maximum of pixel values a and b, min is a minimum of pixel values a and b, and N is a predetermined threshold value.

4. The image sensor system of claim 1 wherein, prior to edge extraction, the temporal processor performs median filtering on the pixel data using race logic.

5. The image sensor system of claim 4 wherein the temporal processor implements median filtering using a first arrival logic circuit to determine a minimum value and a last arrival circuit to determine a maximum value.Attorney Docket No. 2115-008446-WO-POA6. The image sensor system of claim 1 wherein the pixel array, the plurality of analog-to-time converters and the temporal processor are integrated together on a microchip.

7. The image sensor system of claim 6 further comprises a frontend processor integrated on the microchip.

8. The image sensor system of claim 7 wherein the frontend processor is configured to receive edge data from the temporal processor and performs keypoint detection and tracking in the image using the edge data from the temporal processor.

9. The image sensor system of claim 8 wherein the frontend processor detects keypoints in the image by calculating spatial gradients in local regions around pixels.

10. The image sensor system of claim 9 wherein the frontend processor tracks keypoints across images using a Lucas-Kanade method.

11. An image sensor system, comprising:a pixel array comprised of a plurality of photodetectors;a plurality of analog-to-time converters interfaced with the pixel array, each analog-to-time converter in the plurality of analog-to-time-converters is configured to receive a pixel value for an image from one of the photodetectors in the plurality of photodetectors and operates to encode the pixel value from the photodetector into a delay in an output signal;a temporal processor interfaced with the plurality of analog-to-time converters and performs edge extraction on pixel data using race logic, thereby outputting edge data for edges extracted from the pixel data, where the pixel data is comprised of the output signals received from the plurality of analog-to-time converters; anda frontend processor is configured to receive edge data from the temporal processor and performs keypoint detection and tracking in the image using the edge data from the temporal processor.Attorney Docket No. 2115-008446-WO-POA12. The image sensor system of claim 11 wherein each analog-to-time converter includes a comparator circuit interconnected by a switch to a supply voltage, where the switch disconnects the comparator circuit from the supply voltage when power gated.

13. The image sensor system of claim 11 wherein, for each pair of pixels, the temporal processor detects an edge condition according tomax(a,b) > min(a,b) +Nwhere max is a maximum of pixel values a and b, min is a minimum of pixel values a and b, and N is a predetermined threshold value.

14. The image sensor system of claim 11 wherein, prior to edge extraction, the temporal processor performs median filtering on the pixel data using race logic.

15. The image sensor system of claim 14 wherein the temporal processor implements median filtering using a first arrival logic circuit to determine a minimum value and a last arrival circuit to determine a maximum value.

16. The image sensor system of claim 11 wherein the pixel array, the plurality of analog-to-time converters and the temporal processor are integrated together on a microchip.

17. The image sensor system of claim 11 wherein the frontend processor detects keypoints in the image by calculating spatial gradients in local regions around pixels.

18. The image sensor system of claim 17 wherein the frontend processor tracks keypoints across images using a Lucas-Kanade method.

19. A method for performing keypoint detection in an image, comprising: receiving a binary image;for each pixel of the binary image, computing spatial gradients in region around a given pixel using a ternary derivative filter;Attorney Docket No. 2115-008446-WO-POAfor each pixel of the binary image, forming a covariance matrix for the given pixel from the spatial gradients of the given pixel;for each pixel of the binary image, determining a minimum eigenvalue of the covariance matrix; andfor each pixel of the binary image, identifying the given pixel as a keypoint in response to the minimum eigenvalue exceeding a predefined threshold.

20. The method of claim 19 wherein computing spatial gradients includes computing a horizontal gradient for the given pixel and a vertical gradient for the given pixel.

21. The method of claim 20 wherein the covariance matrix is formed according towhere lx is the horizontal gradient for the given pixel and ly is the vertical gradient for the given pixel.

22. The method of claim 19 further comprises computing spatial gradients by convolving pixels adjacent to the given pixel with the ternary derivative filter, where the ternary filter includes vectors [-1 0 1] and [-1 0 1]Tsuch that resulting gradients are in range of [-1 , 1].

23. The method of claim 19 wherein determining a minimum eigenvalue of the covariance matrix includes determining a square root using a lookup table.

24. A method for performing keypoint tracking in a binary image, comprising: receiving a binary image of a previous frame;receiving a set of keypoints for a previous frame;constructing a first series of pyramid levels from the binary image of the previous frame, where each pyramid level in the first series of pyramid levels has a lower resolution than other pyramid levels in the first series of pyramid levels;for each pixel in each pyramid level in the first series of pyramid levels, computing spatial gradient for a given pixel;Attorney Docket No. 2115-008446-WO-POAreceiving a binary image of a current frame;constructing a second series of pyramid levels from the binary image of the current frame, where each pyramid level in the second series of pyramid levels has a lower resolution than other pyramid levels in the second series of pyramid levels;for each pixel in each pyramid level in the second series of pyramid levels, computing spatial gradient for a given pixel;for each pixel in each pyramid level in the second series of pyramid levels, computing a temporal gradient in the relation to corresponding pixel in the first series of pyramid levels; andfor each keypoint in the set of keypoints, determining a displacement of a given keypoint in the current frame.

25. The method of claim 24 further comprises constructing a first series of pyramid levels by downsampling a pyramid level to half resolution of current pyramid.

26. The method of claim 25 further comprises downsampling a pyramid level using a 2 x 2 box filter.

27. The method of claim 25 further comprises computing spatial gradient by convolving pixels adjacent to the given pixel with the ternary derivative filter, where the ternary filter includes vectors [-1 0 1] and [-1 0 1]Tsuch that resulting gradients are in range of [-1 , 1].

Images